I usually function well under pressure, but crafting a year-long Web Development course in two months was daunting. I had no experience teaching in a classroom, no experience teaching teenagers, and no material to work with. I was up the proverbial creek without a paddle, and my canoe was taking on water. The only relevant experience I had came from my time in the Army, where I learned the art of back-planning.

The process of back-planning involves (1) determining your end goal, (2) determining the more granular goals that enable you to achieve that goal, and (3) repeating the process for each of those goals. You are done when your goals are small enough to be actionable tasks.

The final consideration is feasibility. Is it feasible to deliver the curriculum, assign the tasks, perform the assessments, and expect the students to grasp the planned material in the available time?

Teaching Strategies

I have read a number of books on teaching, but the most helpful was Your First Year As a High School Teacher: Making the Transition from Total Novice to Successful Professional by Lynne Rominger, Suzanne Packard Laughrea, and Natalie Elkin (I provide a link to this book on Amazon at the end.) It didn’t delve into pedagogy, learning styles, psychology, or any of those high-browed topics. Instead, it is a book that teaches the mechanics of being a teacher. It starts at the very beginning: landing the job and signing a contract. Then, it walks you through every aspect of the profession, including human resources, payroll, preparing your classroom, creating materials and assessments, dealing with the students, talking with parents, and more. Much of this was not pertinent to my specific situation (working part-time), but it is a great book.

My first revelation from this book was the importance of decorating your classroom. I am an engineer by trade and, generally speaking, consider decorations to be frivolous. Every chapter in the book includes a number of asides called “Teacher’s Rules,” and one that particularly caught my attention was:

Whatever you do, avoid not decorating and leaving your walls and bulletin boards completely devoid of any personality. The sterile environment will turn kids off. Your goal is to create a warm environment that welcomes kids and makes them feel at home in your room and yet doesn’t distract them from their work. (Rominger et. al., ch. 4)

Chapter Seven’s explanation of state curriculum frameworks was another helpful lesson from this book. I’ll delve into this later, but suffice it to say that each state has a library of course curricula to guide lesson planning. Most (maybe all?) publish these online. This chapter also emphasizes the value of asking other teachers to share their curricula.

Don’t reinvent the wheel! Good teachers are willing to share their curriculum work with others so that no one has to duplicate existing work. Why spend two hours creating a 100-question multiple-choice test when three usable tests are already in existence? (Rominger et. al., ch. 7)

While this was not particularly helpful in my case (because web development had never been taught at my school), borrowing could save you a tremendous amount of time. The authors do note that you should take the time to adapt whatever you borrow to your own teaching style.

Chapter Eight explores lesson planning. The book does a commendable job of explaining back-planning and how learning objectives, outcomes, and assessments factor in. It also includes several examples that exemplify project plans and lesson plans. As a final note, they provide advice on adapting your plans to students having difficulty grasping the concept. Personally, I found that leaving a couple of “floating” weeks is helpful. This allowed me to spend some additional time on topics the students found particularly difficult. Next year, I can adjust my overall schedule based on my experiences and leave only a single floating week.

Finally, I found Chapter 11 particularly helpful because, while I felt confident in my course plans and mastery of the material, I was completely inexperienced with teaching high schoolers. The social aspects of teaching (i.e., classroom management) weighed most heavily on my mind. The authors really shined here. They began the chapter by describing the common teacher personalities, moved on to approaches for developing a good rapport with your students, and ended with helpful advice on dealing with trying situations. If you are new to teaching and nothing else I have said convinced you to buy this book, I hope this does. Chapter 11 is invaluable.

Developing the Curriculum

Public schools will provide teachers with a curriculum and accompanying texts. Many private schools also have preselected textbooks and courses. My situation was quite different, however, because I had to teach the first-ever offering of an elective at a private school and had no budget for purchasing course material. Obviously, my first stop was Google.

I found many paid Web Development courses and curricula but few freely available ones. Of the free ones I found, these were the most promising:

• Harvard University’s CS50 Web: https://cs50.harvard.edu/web/2020/

• IBM’s Web Development Course: https://skillsbuild.org/students/course-catalog/web-development

• Mozilla Developer Network’s: https://developer.mozilla.org/en-US/curriculum/

Harvard’s offering is tailored to highly motivated, self-taught individuals. It can be taught by high school teachers, and Harvard has created online communities where you can find support, but I did not find it particularly well-suited to these purposes.

IBM’s offering is quite impressive. It is tightly integrated with IBM’s learning management system (SkillsBuild), offers rubrics, PowerPoints, prebuilt activities, and much more, and includes courses covering a wide range of IT and Computer Science subjects. IBM offers all of this free of charge to individual students and verified schools.

Finally, Mozilla has a very extensive curriculum for web development that covers HTML5, CSS3, and JavaScript. However, it lacks PowerPoints, videos, rubrics, and other such material. Despite these shortcomings, I chose to use its material to familiarize my students with the Mozilla Developer Network website. MDN is the authoritative source for technical data on any of the web standards, and every developer should be proficient in navigating the site. Furthermore, their curriculum is relatively high-level and permits a lot of freedom in how one delivers it.

Having a list of things to teach wasn’t enough; I also wanted to ensure my course met or exceeded established standards. Most states publish their curriculum standards online. Georgia’s can be found at the Georgia Department of Education’s (GaDoe) SuitCase website: https://case.georgiastandards.org/. To verify my course’s compliance with these standards, I created a matrix that mapped each Georgia standard to items from MDN’s curriculum (fig. 1), thereby ensuring my class would meet the state standards.

Some of the state curriculum requirements did not make sense, either for my course or for the school. For example, Course Standard 1 focuses on employability and communication skills. These will be met either indirectly during the course via presentation assignments or have been thoroughly satisfied in other classes because the school follows a classical curriculum that emphasizes such skills.

Another instance where I trimmed out some of the state curriculum is Course Standard 6: Use a server-side language to build a multi-page website incorporating a web form, at least two templates with shared portions, and a data-driven home page (Server Side Languages). There is simply no way to fit this into a year-long course while thoroughly teaching the rest of the curriculum. After reading Course Standard 6 and choosing to omit it, I renamed my course to Front-end Web Development. Course Standard 6 could itself be an entire year-long course titled Backend Web Development.

Creating Materials to Present the Course

I won’t spend much time on this because I don’t find building PowerPoints and handouts particularly interesting. However, I will point out one extremely effective decision I made: cheat sheets. During my time in the Army, I attended a lot of training, most of which was unbelievably compressed because of time constraints. Nearly every course culminated in a final field exercise where Soldiers applied all of the classroom training they had received. Because of the compressed nature of the course, we were always encouraged to keep “cheat sheets” in our hip pockets for reference during the exercises. Often, these would be similar to an aviation checklist where steps for certain activities (maybe programming a SINCGARS crypto radio or establishing a satellite connection on a Command Post Node) were recorded. Another classic example is keeping a blank 9-line MEDIVAC template in your pocket so you don’t forget to transmit critical information when under stress.

I only saw my students twice a week for one hour each session, hardly enough to personally deliver the entire body of material. So, I followed the Army’s example and created “Weekly Companions” for the students that summarized the least they needed to know. Here are a few pages from one of these weekly summaries:

I also permitted students to use the MDN website when taking weekly exams. The tech industry changes so rapidly that I find it unreasonable and pointless to expect students to memorize all of the content. Instead, I expect them to be able to find the answers they need and intelligently apply that information.

Creating Practical Assignments

My next task was creating formative assessments of their skills. Because it is a coding class, they need to use coding tools. In this case, GitHub was the perfect solution. Not only should future web developers be proficient with git, but GitHub also offers a relatively obscure educational platform: GitHub Classroom. GitHub Classroom permits creating assignments (essentially a repository template), scoring those assignments, and even automating the grading process using automated tests and GitHub Actions. While I never got the integration to work just right, it can also connect directly to your Learning Management System for importing grades.

I want to call out one very important practice I follow and recommend: rubrics. If possible, always provide rubrics with assignments and always use those rubrics when grading. They give students clear guidance on what is expected and enable you to grade submissions objectively.

This is a rubric for an assignment that required students to use HTML and CSS to create a student ID:

Creating Summative Assessments

I did not place a lot of importance on exams in the class; in fact, I weighted them far less than the performance assignments. Instead, I created the exams to serve as a final review of the material covered each week. I created the exams in Google Forms, allowed the students to complete them outside of class, and encouraged them to use the MDN website and any material I provided in class (e.g., presentations, hand-outs, etc.) This may seem unorthodox, but knowing how to find and apply references is more important than rote memorization. I have found this to be true during my 14 years as a professional developer.

Note that the tests may have been “open book” and “take home,” but they were not easy. I made most of the questions scenario-based such that they required critical thinking and problem-solving to come to the right answer. I also made an attempt to inject a bit of humor where possible.

Here are a few examples:

Organizing and Planning

Having a course’s lessons, content, assignments, and exams created is a great start, but it is all for naught if you cannot properly schedule it into the school year. I decided to follow the pedagogical standard of lesson plans and unit plans. Fortunately, the MDN curriculum offered a great outline for the course that I used as my starting point.

Somewhat counterintuitively, I created my lesson plans first. I organized them as a week-by-week schedule for the entire class. I was only able to plan in this order because MDN provided such a great foundation.

Below is one page from my schedule/lesson plans. Each week outlines the material covered, assignments due, and links to supplementary material like videos. I also mapped each week to the Georgia Standards it satisfies.

When I first started this document, I quickly noticed a pattern where several weeks would revolve around a common theme. For example, in the image above, you can see that Week 2 is “Part 2” of the “Web Development” theme. Weeks 1 and 2 provide a basic introduction to the concepts covered by the course and the tools the students will be using. I drew three immediate conclusions from this:

1. They share a common theme (obviously)

2. A single summative assessment would be sufficient

3. While I estimate it will take two weeks to cover the material, it would not confuse the students to finish lesson 1 in week 2 or start lesson 2 in week 1, if necessary

It follows that Weeks 1 and 2 should logically comprise a unit of instruction. Elise Phillips at The Will to Teach wrote two excellent blog posts, one on Unit Plans and one on Lesson Plans. She emphasized the importance of unit plans in creating coherent instruction so students acquire a holistic (and unfragmented) understanding of a topic or concept.

Conclusion

I do not claim to be a professional educator or even an experienced layman. I did, however, learn a lot about course development. My goal for this post was to share some of the techniques, resources, and tools I used to accomplish a seemingly impossible goal of building a quality, year-long course in two months.

In summary:

• Back-plan: figure out what you want students to walk away knowing and plan what it takes to get there

• Re-use: don’t reinvent the wheel — use the web, other teachers’ materials, etc.

• Group like things: organize lessons that share a common theme into holistic units

• Avoid dogmatism: don’t do things a certain way just because they’ve always been done that way (e.g., maybe open book, open notes, take-home tests are okay)

• Rubrics: provide them with every assignment and use them to grade — they keep things fair and avoid many arguments with students over the justness of a grade

Read Part 1, "Into the Classroom"

Phillips, E. (n.d.). Creating Effective Unit Plans: A Guide for Teachers. The Will to Teach. Retrieved February 4, 2025, from https://willtoteach.com/creating-effective-unit-plans-a-guide-for-teachers/

Rominger, L., Packard, S., & Elkin, N. (2009). Your first year as a high school teacher : making the transition from a total novice to a successful professional. Three Rivers Press. https://www.amazon.com/Your-First-Year-School-Teacher-ebook/dp/B002361KXU

Becoming a career educator was never in my plans, but I always had a desire to teach others about my passions: web and software development. Twenty-one years later, I accepted my first job as an educator—a part-time, online teaching position at a local community college. This only served to whet my appetite for teaching because most online adjunct positions require “facilitators,” not teachers.

I spent nearly two years at that college. They graciously allowed professors to add content and assignments to the course; only you could not remove anything. I made the most of that policy with my “Introduction to Computers” class.

Being located in a small South Georgia town meant most of my students were working adults, many of whom were seniors. While the provided content covered basic Microsoft Office applications, it had many gaps, including subjects like navigating the web, compressing files using ZIP, working with PDFs, identifying trustworthy sources of information, and other essential skills. I added these to the course as small additional assignments and extra credit assignments.

Still, I wanted to play a more active role in crafting my courses. I did not want to facilitate; I wanted to teach! I left that college and went to work for Southern New Hampshire University, teaching online computer science courses. I was very excited about working in a university setting and instructing computer science courses. Sadly, my excitement was short-lived. I found SNHU even more restrictive than the college. Their course content was totally immutable. I could not even correct assignments or course content that was blatantly wrong (which created a headache for the students and me).

My fondest moments at SNHU were those rare occasions when a student requested office hours over Microsoft Teams—for a few short minutes, I actually taught. SNHU’s format is excellent for self-driven, distance learners, and the school pays quite well. However, it is utterly prohibitive to any teacher-student relationship beyond the weekly discussions and assignment feedback.

After three years at SNHU, it was time for me to find something else.

During the Summer of 2023, my wife and I enrolled our 10-year-old son in a local Christian school that followed a classical curriculum. We had grown tired of public education’s growing ratio of students to teachers, the increasingly progressive nature of its course material, and the never-ending exchange of one unproven educational approach for another. Coming out of the COVID era, like private schools across the nation, this school had seen a boon in enrollment numbers and a growing demand for a greater variety of subjects offered.

A few months later, a close friend of mine, John, told me he was in talks with the headmaster about taking a full-time position teaching marketing, digital graphics, and other related subjects. Like me, he had always been drawn to education. John ran a successful marketing agency for 15 years, often with more business than one man could handle. Sadly, the COVID economy drastically reduced the demand for tailored marketing strategies in our area, and his company never fully recovered. Around May of 2024, John accepted a full-time job as both an electives teacher and the Dean of Students. As you can imagine, this piqued my interest!

That August, I decided to take a chance and try my hand at teaching high school. I will not lie and tell you I was 100% comfortable with the idea. In fact, the thought of managing a classroom of teenagers terrified me. I had spent 13 years in the Army Reserve, spoken to hundreds of Soldiers many times, pitched multi-million dollar Department of Defense contracts, and done my fair share of public speaking. But teenagers terrified me! Regardless, I wanted to do it.

First, I asked my full-time employer if I could shape my schedule to allow two hours away, two afternoons per week. Graciously, they told me I was free to follow my dreams as long as I gave them 40 hours per week. Riding that high, I immediately fired my résumé off to the headmaster and asked if there might be a part-time position. He responded by saying that the school was very excited about the prospect of offering more vocational classes. Interestingly, he was also excited to add a few additional male teachers to the faculty. With the addition of John, their number of male teachers had risen to a staggering total of two.

By this time, the school had already begun its Fall semester, but they were so excited about adding another elective that they told me I would start in October, the beginning of their second quarter.

Needless to say, I had a lot of work to do over the next two months!

In the next part of this series, I will tell you how I organized and crafted a year-long web development course that prepares students as well as any boot camp and is aligned with Georgia’s state curriculum!

Read Part 2, "Crafting a Course"

Splitting monolithic applications into microservices is likely the most common source of microservices, a natural and appropriate state of affairs. In fact, I would argue that starting a system as a monolith is more appropriate than jumping straight to microservices. There is nothing inherently wrong with monolithic architectures. They are often simpler to maintain, easier to reason about, easier to deploy, and require far less infrastructure. Therefore, make the most of your monolith before going down this road.

Squeeze all the Goodness out of Your Monolith

Teams can make meaningful improvements without going to microservices. Some examples include (Richardson, 2020):

• Optimize your development process.

  • Are teams actually agile or just “doing” agile?

  • Do developers have the right tools?

  • Are business processes getting in the way?

• Optimize your CI/CD processes. Is everything automated?

• Improve team autonomy.

  • Is the system modularized?

  • Can Maven/Gradle subprojects be used to allow independent development?

  • Are teams cross-functional? Are there silos that need to be torn down?

• Can obsolete system components be replaced with modern alternatives?

• Can the monolith be refactored to allow horizontal scaling?

  • Can a caching database (e.g., Redis) be used for session sharing?

  • Can the system migrate to a stateless API?

  • Is an API gateway being used?

  • Is the front end separated from the back end?

Will Microservices Solve Your Problems?

Before splitting your monolith, ensure doing so is the best available option. As I wrote at the end of the last article, determine the problems that need to be solved, know the issues that microservices can solve, and ensure sufficient overlap exists to justify moving to microservices. If the answer to this question isn’t immediately apparent, take a few minutes to read this article from Segment.com: Goodbye Microservices: From 100s of Problem Children to 1 Superstar. They started with microservices and decided a monolith worked better.

An Overview of the Process

Assuming you made the most of your monolith, found microservices offer some solutions to your problems, and decided to move forward, it’s time to form a plan. Broadly, I propose you employ the Strangler Fig Pattern, described by Chris Richardson (n.d.), while mixing in a healthy dose of Domain-Driven Design. This process comprises six steps:

1. Model the business processes of the monolith

2. Identify bounded contexts and prioritize them for extraction

3. Extract the first context’s domain model and logic into a new module

4. Separate the New Module with an Anti-corruption Layer

5. Extract the module into a new service and run that service in parallel with the monolith to confirm it functions correctly

6. Remove the module from the monolith, leaving behind only an Anti-corruption Layer that leverages the new service’s APIs

Step 1: Model the Business Processes

It is impossible to make an intelligent decision about where to start decomposing a monolith without first determining what it is supposed to be doing. Simply dividing the monolith’s code up as is will almost invariably lead to a buggy distributed monolith. Instead, do the homework of modeling the processes the monolith supports. As discussed in the last article, Event Storming is a powerful and simple method of (re)discovering the system’s requirements while simultaneously modeling the business processes driving those requirements.

Step 2: Identify Bounded Contexts and Prioritize them for Extraction

Event Storming’s format for the business process model (the sticky note wall chart) makes identifying bounded contexts (and candidate services) simple. The previous article mentioned several wholistic approaches to discovering bounded contexts.

Once you’ve drawn those boundaries, prioritize them. The Strangler Fig Pattern is a methodical, slow, and piecemeal approach to decomposing a monolithic application into small services. It emphasizes low risk over expediency. Thus, it would be best if you prioritized your work to deliver the most significant business value upfront.

How you determine that value depends on your organization. I suggest reflecting on why you are considering microservices and going from there. For example, are you trying to increase the rate of software delivery, increase horizontal scalability, reduce operations costs, or improve organizational agility? Whatever the answer is, prioritize the decomposition process to emphasize those goals.

Step 3: Extract the First Bounded Context into a New Module

Now that you have a target, it’s time to start hacking at the code. I suggest performing this step in two phases:

1. Extract domain models of the bounded context to the new module

2. Extract the actual business logic to the new module

Extracting the domain models will likely include renaming entities to match the ideal system’s ubiquitous language. This step may also include changes to the database. Your goal is to fully extract the domain model (both in source code and database). Later, the tables (or collections) will be moved into their own schema or database.

Step 4: Separate the New Module with an Anti-corruption Layer

Step 3 involves the surgical transplant of domain models and code out of their current home and into a new home (the new module). Step 4 builds a wall around the new module by creating an anti-corruption layer and forcing all communications with the new module to go through that layer. In practice, this often looks like a service class(es) through which all calls to the underlying logic are funneled.

Additionally, rather than using the domain models (now exclusively owned by the new module), the remainder of the monolith should exchange DTOs with the anti-corruption layer. This will initially look like duplicate code but will make extracting the module into a separate service much easier.

Step 5: Extract the Module into a New Service

Step 3 isolated a section of the monolith into a new module, effectively realizing the bounded context identified by Step 2. Step 4 created a single interface through which the context may be accessed. Step 5 will create a new service (a separate application) using the new module.

Importantly, do not remove the module from the monolith yet. Instead, both the new module and the new service should run in parallel (see Figure 4). This provides an opportunity to compare the results of the new service to those of the module, ensuring it functions as expected.

Step 6: Remove the Module from the Monolith

Having verified the new service is fully functional, we can now remove the module from the monolith. This step should be as simple. In an ideal scenario, it only entails eliminating remnants of the module except for the Anti-corruption Layer and refactoring the Anti-corruption Layer to direct all calls to the new service. Figure 5 shows this graphically.

Note that an API gateway is shown as the interface between the monolith and the new module, but this could be any enterprise interface. Alternatives include event buses, message queues, message channels, gRPC, or polling solutions.

A Case Study

In the last article of this series, I used a fictitious Patient Records Portal application as a running example. I will continue that narrative by examining a monolithic application that contains the Billing System bounded context. The Billing System is a sizable, monolithic application that performs many functions. In fact, it is large enough to justify its own domain analysis.

The team chose the Billing System because, although it is a monolith, it is already neatly divided into modules. They hope to decompose it into microservices to improve scalability and availability (i.e., they want to scale the busier parts of the app horizontally).

Steps 1 and 2: Model the Business Processes and Identify Bounded Contexts

Because this article is not about Event Storming, I will not go into the actual modeling process. Instead, assume the team already performed an Event Storming session and discovered these bounded contexts:

Figure 6 shows the team discovered five bounded contexts:

• Service Coding: services provided to patients are coded and charged by the Debit Entry context

• Insurance: negotiating reimbursement for coded services and processing those payment transactions; the Credit Entry context handles successful transactions

• Credit Entry: processed payments are posted to accounts

• Debit Entry: processed charges are posted to accounts

• Statement Generation: produces account statements showing all processed credits and debits

After careful consideration, the team first extracted the Statement Generation context. They identified several reasons for this, including:

• The Statement Generation context is the most loosely coupled portion of the system.

• Statements are generated in massive batches each month to be mailed out, and horizontal scaling would significantly speed up this activity.

• They felt this context offers a low-risk learning opportunity before moving on to more complex parts of the system.

After digging into the Statement Generation code, the team was able to draw up a crude diagram of the overall Billing System and, specifically, the Statement Generation module:

Step 3: Extract the Bounded Context into a New Module

The team studied the diagram they created and decided the Statement Generation model would best be decomposed into two microservices:

1. Account Ledger Service:

   • Flattens credits and debits into a simplified model tailored to the needs of the Statement Generation Service

   • Eliminates coupling with the monolith that Shared Persistence would create

2. Statement Generation Service:

   • Uses the Account Ledger Service to fetch data for generating account statements

   • Prints account statements tagged for printing

First, the team extracts the Statement model into the statement-generation module.

Next, they create a new AccountLineItem model in the new account-ledger module that allows them to decouple the statement-generation context from other parts of the monolith like Credit, Debit, and Account.

Once the domain model is extracted, the team extracts actual business logic. First, they move the StatementQueryService into the new account-ledger module and rename it to AccountLedgerService. To further decouple it, they have it return AccountLineItem objects rather than account IDs. Because this new service is also responsible for maintaining a context-specific representation of the state of accounts, the team adds methods for crediting and debiting amounts. The credit-entry and debit-entry modules now call these methods to keep the AccountLineItems in sync with the rest of the system.

Step 4: Separate the New Module with an Anti-corruption Layer

Thankfully, the code is already organized, so the beginning of an Anti-corruption Layer exists in the StatementController. The team decides to further isolate the new statement-generation module from the rest of the monolith by injecting an asynchronous interface between the StatementController and the prospective services — the Statement Generation Service and the Account Ledger Service. This will make Step 5 decidedly easier and will further enable horizontal scaling.

The team does not want to modify the monolith more than necessary, so adding logic to handle retries for failed messages is out of the question. Additionally, they are dealing with financial data and must guarantee that the StatementController’s outgoing messages are delivered. Therefore, the team employs the Guaranteed Delivery pattern (Hohpe & Woolf, 2003, p. 122) with the “Generate/Print Statement Queue.” They will code the StatementController to push a message on the queue. The message will direct the StatementGenerator to generate a new account statement and, optionally, print that statement.

The StatementGenerator service will monitor this queue and process the commands to generate statements. After a statement is generated and stored in the statements table, the StatementGenerator will send a “Statement Generated” message onto the “Statement Generated Message Channel” (Hohpe & Woolf, 2003, p. 60). A message header is added for those messages that need to be printed. The “Print Message Filter” listens to this channel and only delivers messages with the “print” header to StatementPrinter. This is an implementation of Hohpe & Woolf’s (2003) Message Filter (p. 236).

Finally, the team must replace the direct calls from the credit-entry and debit-entry modules to AccountLedgerService with an asynchronous interface. As with the StatementController, they use a message queue, backed by disk storage for guaranteed delivery, named “Ledger Entry Queue.”

Step 5: Extract the Modules into New Services

The statement-generation and account-ledger modules are ready for extraction. The team creates two new projects and copies each module into a project. They also create two new database schemas (one per service) and create the appropriate tables there. They also copy the original Statements table’s data into the new schema so its history is carried over.

The AccountLedgerService’s findOutstandingAccountsByDates(...) method will be called synchronously by an HTTP GET request. To accommodate this, the team establishes an API Gateway in front of the AccountLedgerService to proxy those requests.

Next, the team duplicates their messaging infrastructure so the modules still in the monolith can run alongside the new services. They also comment out the code that prints statements in the new Statement Generation Service, but they add a logging command so they can verify that the statements would have been printed. This allows for testing and verification before moving production operations to the new services.

Finally, the team containerizes the new services and deploys them and the updated monolith’s code. They work with the testers to confirm the new services are functioning correctly.

Step 6: Remove the Modules from the Monolith

After testing the new services for two weeks, the team decided it was safe to switch to them entirely and remove their corresponding modules from the monolith. This was a simple matter of removing the modules from the monolith. They also removed the lines of code from the Anti-corruption Layer that were calling the account-ledger module.

Next, the team built and deployed new versions of the monolith and service containers. After confirming the successful deployment, they destroyed the duplicate messaging infrastructure and removed the AccountLedger and Statements tables from the monolith’s database schema.

Post-Migration

The past six steps safely extracted the Statement Generation bounded context into two services that can be horizontally scaled to meet increased demand. They may now be updated and modified without impacting the original monolith as long as the interfaces do not change.

Ultimately, the Billing System’s context map looks like Figure 12.

Conclusion

As I mentioned, this six-step process is a variation of Richardson’s (n.d.) Strangler Fig Pattern. It is a low-risk, disciplined approach to decomposing a monolith into smaller services. Some key takeaways include:

• Ideally, outside the Anti-corruption Layer, the monolith remains unaware of any changes.

• If your monolith originally received responses as entities from the extracted code and your services now return DTOs, you will need to convert those entities to adapters (i.e., wrappers) of the DTOs. You must then modify your Anti-corruption Layer to return the adapters to the rest of the monolith.

• Although the size of your microservices may vary, the guiding principle is to ensure they neatly encapsulate their bounded contexts while remaining loosely coupled and highly cohesive.

• There is no rule that a monolith must be fully decomposed. Extract microservices as necessary, but do not be dogmatic about it.

• Hohpe, G., & Woolf, B. (2003). Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley Longman Publishing Co., Inc. https://dl.acm.org/doi/10.5555/940308

• Richardson, C. (2020). Decompose your Monolith: Six Principles for Refactoring a Monolith to Microservices. Microservice Architecture. https://microservices.io/post/refactoring/2020/07/28/six-principles-for-refactoring-to-microservices.html

• Richardson, C. (n.d.). The Strangler Fig application pattern: incremental modernization to microservices. Microservice Architecture. Retrieved December 1, 2023, from https://microservices.io/post/refactoring/2023/06/21/strangler-fig-application-pattern-incremental-modernization-to-services.md.html

Domain-Driven Design to the Rescue!

The Importance of Naming Things

One of the most challenging tasks developers face is effectively naming things. By effective, I mean in a way that lasts the tests of time and remains meaningful, even to your successors who will someday maintain your code. This task is so infamous that it has been immortalized in the annals of the software industry by witty quotes, jokes, and comics.

One of the most famous quotes (which has, in turn, inspired many others) was spoken by Phil Karlton while leading a team of developers at Netscape:

There are only two hard things in Computer Science: cache invalidation and naming things.

If you are familiar with the complexities of cache invalidation, you will understand why comparing these two things is both paradoxical and somewhat shocking. In other cases, developers name things in ways that make sense to them but are incoherent to others (see Figure 1).

Domain-Driven Design seeks to reduce the incongruence between a software system’s problem space (the customer’s domain) and its solution space (the system’s architecture and code). DDD starts with two essential tasks: identifying things and naming those things.

Bridging the Gap

DDD bridges the gap between the problem and solution spaces by systematically studying the problem space’s real-world systems and mapping those ideas into the solution domain. Eric Evans’ Domain-Driven Design: Tackling Complexity in the Heart of Software (2004) is the “bible” of DDD. It lists many model elements, design patterns, strategies, and techniques of DDD; however, we will only discuss three that help solve our immediate problems with microservices: ubiquitous language, domains, and bounded contexts.

For the rest of this section, I will use a fictitious Patient Records Portal system as a case study. This system should serve patients and providers by providing access to their medical records.

Ubiquitous Language

The first step in Domain-Driven Design is adopting a Ubiquitous Language. The client and development team share this vocabulary (see Figure 2).

Creating and using a ubiquitous language offers many benefits, including:

• Unambiguous requirements

• Self-documenting code that explains (through class, function, and variable names) what part of the client’s problem(s) it solves

• Components of code (classes, methods, objects, packages, etc.) map to real-world concepts

• Developers form a better intuition for the client’s problems and optimal solutions

• The system architecture reflects the structures of the problem domain

• Eliminates confusion in person-to-person communication

• Enables loosely coupled and highly cohesive code

In the context of the Patient Records Portal, the ubiquitous language should provide the development team with a vocabulary they both comprehend and use. It must clear up confusion and ensure that words hold the same meanings for the client(s) and the development team.

To demonstrate the value of a ubiquitous language, study the list of words below. Each word is either specific to the medical field or overloaded by meanings in the medical field and other fields:

• Assays

• Provider

• Classification

• Topography

These are only four trivial examples, but imagine the confusion they would cause without a ubiquitous language

Consider assays; according to The American Heritage® Dictionary, it could mean assessing a substance, the substance being assessed, or the assessment results. Imagine if the Patient Records Portal clients understood assay to mean “an assessment” while developers understood it to mean “the results of an assessment.” This mistake is not only easy to make, but it could go undetected for a long while.

Provider is overloaded with different meanings from the medical, software, and other fields. In the medical field, it often refers to a physician. However, provider could also mean a nurse practitioner, physician’s assistant, dentist, or other professional. Even more confusing, physicians may or may not be providers, and providers may or may not be physicians. While medical professionals can probably deduce the intended meaning from the context in which the word was used, a software team is unlikely to do the same.

Topography is usually associated with maps that include elevation markings, relief features, and shading. In the medical field, however, it means a specific location on the human body. Classification is another broadly used term. In the medical field, it refers to coding diagnoses, treatments, medications, and other things so they can be precisely recorded and billed. It may also refer to data classified as protected health information (PHI) by the Health Insurance Portability and Accountability Act (HIPAA). In the government domain, it refers to the sensitivity of information and the potential harm its unintended release could cause.

We have already anticipated a lot of confusion resulting from just four words. What happens when we begin throwing software terms into the mix? For example:

• ProviderRegistry

• AbstractAssayFactory

• TopographyDiagnosisBuilder

• ClassificationConversionService

These are only four trivial examples, but imagine the confusion they would cause without a ubiquitous language.

Domains

The next step in applying DDD to a microservices project is identifying the subdomains the project will touch. The domain is what a business does and the world it does it in (Vernon, 2013, p. 43). An organization’s domain includes multiple subdomains that house different categories of activity and knowledge required for a business to function. There are three types of subdomains: core, supporting, and generic.

💡 Subdomains model the problem space of a project — real-world business activities and ideas.

Figure 3 (above) depicts the Patient Records Portal project’s overall domain comprising six subdomains. Each subdomain can be either a core, supporting, or generic subdomain (Vernon, 2013, p. 51-52):

• Supporting subdomains model those aspects of the project that are essential but not core. For example, imagery (e.g., X-rays, CT scans, sonograms) is a valuable part of patient records, but it is not the core of this project.

• A generic subdomain is a part of the project that is not special, particular, or unique to the project but still essential to its success. Figure 3 shows Identity & Access Management (IAM) as a generic subdomain. Not only is making the application secure a good idea, but it is also legally required when dealing with health information. At the same time, IAM is not a concept unique or special to our project. Therefore, we include it in our domain but label it as generic.

Bounded Contexts

Even with a ubiquitous language developed for a project, it is possible to encounter conflicting models and terms within that project. It is also possible for different terms to refer to very similar concepts. For example:

• The term schedule is used when classifying drugs (e.g., schedule 2 drugs—like narcotics—have medical uses but are also highly prone to abuse). This conflicts with using schedule in the Scheduling subdomain, which refers to appointments.

• The idea of “customer” may be labeled differently in each subdomain. This is especially likely if any subdomains are applications maintained by external entities (e.g., SaaS products). The Pharmacological subdomain may call them “customer,” the Patient Records subdomain probably calls them “patient,” the Imaging subdomain may call them “subject,” and the Billing subdomain may refer to them as an “account.” More importantly, that person is probably represented by a different data model with different fields in each subdomain.

DDD calls these boundaries between models and meaning bounded contexts. As the cell walls shown in Figure 4 prevent the spillage of organelles, so do bounded contexts prevent the spillage of one domain’s models and meanings into another’s. That spillage results in buggy software that is tightly coupled and incoherent.

💡 Bounded Contexts model a project’s solution space — the software systems that solve a problem.

Evans (2004, p. 336) suggests that these boundaries should be realized logically (e.g., separate database schemas and code repositories) and physically (e.g., team organization). This proposition flips Conway’s Law on its head by forcing team structure to match the system rather than the inverse.

Alberto Brandolini (2015, p. 116) describes the significance of bounded contexts like this:

Now I consider “getting the boundaries right” the single design decision with the most significant impact over the entire life of a software project.

Figure 5 depicts one possibility for how bounded contexts might be drawn in the Patient Records Portal project. Notice that the bounded contexts often, but not always, align one-for-one with a system’s subdomains. There are two exceptions in Figure 5: the Scheduling System and the Pharmacy System.

The Scheduling System overlaps three subdomains apart from the Scheduling subdomain. The scheduling system might be used for patient appointments, imaging scans, and laboratory procedures. In that case, it must include concepts and models from those subdomains.

Similarly, the Pharmacy System may manage prescriptions, prescription fills, medication regimens, patients’ adverse reactions to medications, and control over pharmacological substances. Some medication regimens require periodic blood tests to monitor things like liver health. Therefore, the Pharmacy System would include some ideas and models from the Lab Orders subdomain.

From Context Maps to Microservices

Now that the Patient Records Portal has been modeled as a context map, it is relatively simple to identify prospective microservices — look at the bounded contexts. A domain’s bounded contexts translate cleanly to microservices for two simple reasons:

1. They are highly cohesive:

   • Focused on solving a specific problem

   • Contain a single, specific model

   • Contain a single subset of the domain’s ubiquitous language

2. They are loosely coupled:

   • Models are not shared with other bounded contexts

   • Overlapping multiple subdomains means they will have their representations of models, not shared models

   • Boundaries require interfaces for bounded contexts to interact with one another

Interestingly, this solves three of the four anti-patterns discussed in Microservices Anti-patterns, Part 1: What are they?:

1. Distributed Monolith: A distributed monolith is a collection of microservices that know too much about each other and are overly dependent on each other. Bounded contexts do neither of these things.

2. Shared Persistence: There can be no shared persistence without shared models.

3. Shared Libraries: There is no reason to share libraries if every microservice is appropriately bound to be highly cohesive and loosely coupled.

Figure 5 clearly shows the need for at least seven microservices:

1. Patient Records Service

2. Lab Service

3. Scheduling Service

4. Imaging Service

5. Pharmacy Service

6. IAM Service

7. Billing Service

Notice my use of the phrase “at least.” I find these seven microservices to be large. No rule prohibits a bounded context from containing more than one microservice. For example, the Billing Service might be split into:

• Account Statements Service

• Outstanding Accounts Report Service

• Statement Mailing Service

• Transaction Summary Report Service

• Past Due Report Service

• Collections Service

The key to successfully subdividing bounded contexts into multiple microservices is to treat each microservice as its own bounded context. They must not share models, persistence, kernels, or libraries. They should remain highly cohesive (focused on a single problem) and loosely coupled (unaware of the inner workings of other microservices).

Storming to Discovery

I have explained what ubiquitous language, domains, subdomains, and bounded contexts are. You know the domain and its subdomains form the problem space — the customer’s business functions. I also explained how a subdomain houses a specific category of activity and knowledge that falls within the overall domain. Therefore, it is possible to discern these details by studying the customer’s business or asking them directly.

Bounded contexts, however, are not so easily identified. They model the solution space — the system that will solve the project’s problem. Discovering bounded contexts and overlaying them atop the business’s subdomains is the first step in designing a new distributed system composed of microservices. How, then, does one go about making this discovery? There is no shortage of opinions on requirements gathering and design practices, but the most effective one I have experienced (particularly for Domain-Driven Design) is EventStorming.

There is no singular definition for EventStorming. Even its progenitor, Alberto Brandolini, does not explicitly define the term in his book about EventStorming, Introducing EventStorming: An Act of Deliberate Collective Learning. However, by borrowing some concepts from his book and mixing them with Wikipedia’s explanation, I came up with this:

EventStorming: A workshop where stakeholders, domain experts, and a software team collaborate to model business processes, identify problems worth solving, share knowledge, and begin planning solutions. The workshop is lightweight, using only markers and sticky notes, and highly collaborative, requiring everyone’s active participation.

I won’t explain how to perform EventStorming, as that would take another series of articles. I will, however, provide some tips on identifying bounded contexts once you have completed a storming session.

First, Brandolini (2015) suggests adding two relatively obscure (not commonly mentioned when discussing EventStorming) markers to the wall when storming: pivotal events and swimlanes.

He defines them as:

• Pivotal Events: “specific events which are particularly relevant for the business and mark a transition between different business phases” (p. 126)

• Horizontal Swimlanes: “a common way to structure portions of the whole flow” to be used when “…the flow is not linear. There are branches, loops, and things that happen in parallel.” (p. 128)

Figure 6 (above) is a simple example borrowed from Brandolini’s book that depicts both pivotal events and swimlanes. Pivotal events are highlighted by placing a yellow line (marker or tape) beneath the orange event sticky note. Horizontal swimlanes are drawn as a red line (marker or tape) that indicates the extent of a nested flow within the overall business flow.

Before moving on, it is crucial to have the participants of the EventStorming agree on the pivotal events and swimlanes. Once you have their approval, you can use both the pivotal events and horizontal swimlanes as guides for where to discover bounded contexts (see Figure 7 below). Brandolini emphasizes that this is a heuristic, not a guarantee or rule.

Before leaving this section, I implore you to read Chapter 6: Discovering Bounded Contexts with EventStorming in Brandolini’s book. My explanation above is an extreme distillation of what the book covers. Please purchase his book if you plan to use DDD or EventStorming and can spare $10. It is worth it.

Microservices are a Magic Pixie Dust

This pattern describes when an organization adopts microservices to solve problems that are not solvable with microservices. Microservices are a highly cohesive, loosely coupled architecture that offers easy deployability, testability, and scalability. They will not solve tangential and unrelated issues like code quality, poor development and deployment pipelines, or existing architectural issues (an overgrown monolith). Microservices may be a part of the actual solution. As Richardson notes, you must first understand the cause of your problems, the problems microservices can solve, and determine if the two overlap.

Microservices are the Goal

Having experienced this anti-pattern, it is especially poignant to me. Whether it is an executive who jumps on every buzzword bandwagon or an overzealous architect who wants to pad his résumé with a long list of new technologies, the result is the same: using microservices for the sake of using microservices. Many issues can come from this strategy, not the least of which is that microservices may be a poor choice for a given project.

Scattershot Adoption

This is another anti-pattern I have experienced firsthand. This describes the situation where multiple teams in an organization decide to start building microservices without coordinating their efforts. Several issues arise from this practice, including duplicated effort, myriad approaches to infrastructure, and blurred bounded contexts.

Trying to Fly Before You Can Walk

Richardson describes this as occurring when organizations skip past getting basic software development practices right and go straight to building microservices. It would be best to master the fundamentals before progressing to more complicated exercises. Those fundamentals include clean code, automated testing, CI/CD pipelines, and solid design/architecture skills. While he didn’t mention this, I would add a functioning orchestrated hosting environment and continuous container delivery platform. Amazon Web Services (AWS) offers excellent options. If an organization prefers a self-managed solution, Kubernetes combined with ArgoCD is a good choice.

Focusing on Technology

An organization that spends large sums of money on top-notch cloud and orchestration platforms before they understand their needs or the problems they are trying to solve has fallen victim to this anti-pattern. Instead, they should focus on the benefits of microservices, apply them as appropriate, invest in relevant technologies, and evolve their systems as needed.

More the Merrier

This happens when organizations take “micro” to the extreme and create many overly fine-grained microservices. This creates unnecessary complexity, resource spending, and, possibly, latency because of the additional network traffic. Additionally, unnaturally dividing a system into microservices is an easy way to blur bounded contexts and introduce anti-patterns like shared persistence, shared libraries, and distributed monoliths.

Red Flag Law

This anti-pattern describes organizations that keep the same operating practices and team structures used for monolithic development when building microservices. Richardson explains, “If you don’t change your organization’s software development process and structure, it’s likely that you will not benefit from the microservice architecture.”

A Little Advice

Using DDD and EventStorming to get bounded contexts right is an excellent first step to avoiding the Microservices Anti-patterns I discussed in the previous article. Avoiding Richardson’s Microservices Adoption Anti-patterns will also guide you to use microservices where they make sense and get the maximum benefit from them. Below, I list a few final suggestions to help you be successful.

Be Flexibly DRY

Developers tout the “Don’t Repeat Yourself” dogma, but it can cause problems for you in microservices. This is especially true at the edges of bounded contexts. Microservices that use the same data should not share the same models. Their models may be very similar, but they should still be tailored to each service’s needs. This is another way of saying don’t share kernels, don’t share persistence, and don’t share libraries.

A Small Monolith may be a Better Option

Microservices are not always the right solution. If you cannot cleanly separate the concerns of a monolith, creating microservices will likely cause you more difficulties than you already have.

Parting Thoughts

When deploying new microservices systems, be deliberate about enforcing the principles of high cohesion and low coupling. Ensure each microservice has a specific purpose and “stays in its lane.” Leverage Domain-Driven Design when discovering the system’s requirements and architecting a solution. Finally, if you are newly adopting microservices, take a conservative approach to that journey and avoid Richardson’s adoption anti-patterns.

In the following article in this series, I write about avoiding microservices anti-patterns when migrating a monolithic application to microservices architecture.

• Brandolini, A. (2015). Introducing EventStorming: An Act of Deliberate Collective Learning. Leanpub. http://leanpub.com/introducingeventstorming

• Evans, E. (2004). Domain-Driven Design: Tackling Complexity in the Heart of Software. Pearson Education, Inc.

• Richardson, C. (n.d.). Microservices adoption anti-patterns. Microservice Architecture. Retrieved November 22, 2023, from https://microservices.io/microservices/antipatterns/-/the/series/2019/06/18/microservices-adoption-antipatterns.html

• Vernon, V. (2013). Implementing Domain-Driven Design. Pearson Education, Inc.

1. Technology Heterogeneity

2. Resilience

3. Scaling

4. Ease of Deployment

5. Organizational Alignment

6. Composability

7. Optimizing for Replaceability

Of these seven traits, five depend on the services being loosely coupled. This principle is the most frequent victim of microservice anti-patterns. It is, therefore, unsurprising that organizations often fail to realize their goals when they migrate their systems from monoliths to microservices. Their microservices may be highly resilient and easily scalable, thanks to tools like Kubernetes, but maintaining those services quickly becomes unmanageable.

Their microservices may be highly resilient and easily scalable, thanks to tools like Kubernetes, but maintaining those services quickly becomes unmanageable.

The five anti-patterns we will discuss all create dependencies amongst microservices, making them tightly coupled, brittle, and difficult to maintain. They are:

1. Distributed Monolith

2. Entangled Data, Data Taffy, or Shared Persistence

3. Piggy Bank

4. Dependency Disorder

Distributed Monolith

While this anti-pattern is most commonly found in microservices that result from splitting a monolithic application, I have also seen it in greenfield projects. The Distributed Monolith is the result of a system’s fall to the other three anti-patterns.

Indicators of Distributed Monoliths (see Figure 1 for an example) include:

• Deploying a new version of one microservice requires new versions of other microservices to be deployed

• Multiple services share common resources (e.g., databases, message channels, etc.)

• Standard libraries of code are shared between the microservices

• Any of the microservices cannot perform their tasks without other microservices being present

Let’s dissect Figure 1 before moving on. First and most apparent, you will notice that three of the four services rely on the same database. This likely indicates that these three services must all be updated if their data model changes.

Next, notice that microservices 1 and 2 communicate with each other. The diagram doesn’t indicate what form that communication takes, but we can assume it is some form of API or RPC. If one of the services is responsible for triggering the other, they should probably be merged.

Microservices 3 and 4 are coupled via a message channel. This could be good or bad, depending on who owns and maintains those services. The microservices should be merged if they are from the same development team, as it appears that microservice 4 waits for a message from 3 and then writes something to the database. In that case, the message channel is an unnecessary complication.

If different teams own microservices 3 and 4, those teams should carefully assess their arrangement. Maybe both services should be owned by the same team. Maybe the organizational structure of the teams is wrong. Perhaps the teams fell victim to Richardson’s (n.d.) Scattershot Adoption anti-pattern, and they should collaborate on a better solution. It is also possible that using the message channel is the best solution.

Entangled Data, Data Taffy, or Shared Persistence

Shared Persistence most often refers to multiple microservices using the same relational database. Taibi, Lenarduzzi, and Pahl (2020) found that the worst scenario is when the afflicted microservices share the same database entities (p. 11). While it is also bad when other types of databases are shared amongst services (e.g., key-value stores, document databases, etc.), the effects aren’t as profound due to the flexibility of the schemas and the horizontal scalability of NoSQL databases.

Indicators of Shared Persistence include:

• Altering a database table schema requires modifying the code of multiple microservices

• Shared code libraries of object-relational mapping entities used by multiple microservices

• Difficulty determining which microservice project should contain database migration scripts

• Microservices are suffering due to database performance, row locks, and other forms of resource contention

Figure 2 depicts a classic case of Entangled Data. A cursory glance reveals the following issues:

• Because Java Microservice 1 and Java Microservice 2 depend on the same Java ORM Entities Library, they are tightly coupled via that dependency. This dependency is, in turn, tightly coupled to the database schema.

• All three microservices depend on the same database table. In turn, they are tightly coupled to each other via that table. Any changes made to that table will require:

  • The Java ORM Entities Library be updated

  • Java Microservice 1 and Java Microservice 2 must be updated to use the new version of the Java ORM Entities Library. They may require code changes to be compatible with the new version of the library.

  • The Golang Microservice must be updated so its representation of the users table matches the database schema.

• The graphic doesn’t tell us which, if any, of the microservices write to the database. However, if more than one writes to it, they may encounter resource contention issues.

• If the owning team employs automated database migrations, how should they choose which service is responsible?

Piggy Bank

The Piggy Bank anti-pattern describes what happens when teams take a naive approach to migrating a monolithic application to microservices. The team begins with an application aptly described as a “pig.” It is large, ugly, and odorous with code smells.

Monson (2019) writes, “When a team is first introduced to this…the first inclination is to smash the whole thing up into little pieces, like a piggy bank with a hammer. Because, of course, 1000 services will be way better than one bloated one, right?” However, without proper planning and design, these microservices will almost certainly suffer from Shared Persistence, Entangled Data, and Shared Libraries. Guess what you have, then? A Distributed Monolith.

When a team is first introduced to this…the first inclination is to smash the whole thing up into little pieces, like a piggy bank with a hammer.

Look at Figure 3 and imagine how easily a migration could transform the original Patient Records Portal monolith (on the far left) into the massive Distributed Monolith on the right. What was once a monolith is now a hodgepodge of microservices — some are anemic, some are bloated, many share databases, many depend on each other, and, presumably, many share code libraries. In short, the result of breaking that Piggy Bank is a mess.

Dependency Disorder

This anti-pattern comprises two issues under a single label: inter-service dependencies via shared code and inter-service dependencies via API calls. In short, the services suffer from an unhealthy relationship such that they cannot be independently modified and deployed in any order.

The services suffer from an unhealthy relationship such that they cannot be independently modified and deployed in any order.

While this list is assuredly not exhaustive, here are some ways Dependency Disorder can occur:

1. As a side-effect of Entangled Data:

In this situation, one microservice is treated as the “owner” of the data and contains a code module of ORM entities. Another microservice references that code module as a dependency so that it may use those ORM entities in its code base.

2. As a side-effect of Shared Persistence:

Like the first example, one microservice is treated as the “owner” of the data, but in this case, it is responsible for executing database migration scripts. Other microservices that use those same database tables are dependent on the “owner” and must be modified and simultaneously deployed with it to avoid breaking.

3. As a failure to properly version the API:

When microservices depend on each other, they implicitly enter into a contract guaranteeing that the provider will not change the API being used by the consumer. Breaching this contract would break the consumer. Despite this contract, the providing service must also be able to evolve by adding new features. The most common solution to this backward compatibility problem is to version the API. This modifies the contract such that the provider guarantees it will not alter a previous API version but remains free to add new versions.

4. As a failure to separate concerns:

This form of Dependency Disorder may arise as a result of the Piggy Bank anti-pattern. It can be distilled down to poor refactoring, creating a timing issue. For example, service A requires that B is available before A starts. If B is unavailable when A starts, then A fails to start. In the best case, your orchestrator respawns A until it successfully starts. In the worst case, B has a circular dependency on A, and a vicious cycle begins where the orchestrator continually restarts both A and B.

A couple of solutions to this problem are immediately apparent. First, the two services could be refactored to eliminate this dependency or gracefully handle failure. Second, an API gateway could be deployed to mediate interactions with and between the services.

Conclusion

To end this article on a positive note, take solace in knowing that all of these issues can be avoided by following the principles of Domain-Driven Design. In this series’ following articles, I will discuss how one might:

1. Avoid these anti-patterns for new microservice projects

2. Avoid these anti-patterns when migrating from a monolith to microservices

3. Convert a distributed monolith into proper microservices

• Monson, D. (2019, March 25). Microservices Anti-Patterns - DZone. DZone. https://dzone.com/articles/microservices-anti-patterns

• Newman, S. (2015). Building Microservices: Designing Fine-grained Systems. O’Reilly Media, Inc.

• Richardson, C. (n.d.). Microservices adoption antipatterns. Microservice Architecture. Retrieved November 22, 2023, from https://microservices.io/microservices/antipatterns/-/the/series/2019/06/18/microservices-adoption-antipatterns.html

• Taibi, D., Lenarduzzi, V., & Pahl, C. (2020). Microservices Anti-patterns: A Taxonomy. In Microservices (pp. 111–128). Springer International Publishing. https://doi.org/10.1007/978-3-030-31646-45

First things First, some Definitions…

Some definitions are in order before exploring the finer points of architectural decision-making. It is difficult to discuss terms before agreeing on their meaning…

Data-Transfer Objects (DTOs)

Martin Fowler (2002) defines a DTO as “an object that carries data between processes in order to reduce the number of method calls.” This definition deftly captures the crucial difference between DTOs and the other object types we will discuss:

DTOs are defined by their intended use, while Aggregates and Entities are defined by their contents and structure.

It follows that:

Entities and Aggregates may also be DTOs if used as DTOs.

Another common use of DTOs is simplifying the presentation of information by projecting only data fields required for a particular interface into a DTO. The MVVM (Model-View-ViewModel) pattern is one excellent example. In MVVM, the ViewModel holds only fields required for its corresponding view. The view may be a desktop application window, web application page, terminal screen, or some other output shown to the user. Views typically require only a subset of data fields from one or several models that map to data storage structures (e.g., database tables). It follows that it would be inefficient and cumbersome to map parts of a view onto multiple large and complex models when the required data would be neatly packed into a single DTO.

DTOs do not require an identity while entities must have an identity. In his pivotal work Domain-Driven Design (2003), Eric Evans provides an interesting explanation, “Many objects are not fundamentally defined by their attributes, but rather by a thread of continuity and identity” (p. 89).

Consider the United States Constitution: Any copy of the Constitution will do if you simply want to read it; however, you mean something quite different if you refer to The Constitution stored in the National Archives. In the former, you do not care what copy you get so long as it is accurate. In the latter, you mean a specific copy of the document. The latter has an identity, while the former does not.

As Evans describes below, the latter also has history:

Some objects are not defined primarily by their attributes. They represent a thread of identity that runs through time and often across distinct representations. Sometimes such an object must be matched with another object even though attributes differ. An object must be distinguished from other objects even though they might have the same attributes. Mistaken identity can lead to data corruption. (p. 91)

Notice that entities can maintain their identity across distinct representations. This is a nod to projections of entities, serializations of entities, DTOs that carry entities, or entities used as DTOs. No matter the form, the data represents the same entity.

As a final note, DTOs and entities are not mutually exclusive — an entity may simultaneously be a DTO if used as one.

Value Objects

Evans (p. 99) defines Value Objects as:

When you care only about the attributes of an element of the model, classify it as a VALUE OBJECT. Make it express the meaning of the attributes it conveys and give it related functionality. Treat the VALUE OBJECT as immutable. Don’t give it an identity and avoid the design complexities necessary to maintain ENTITIES.

All common programming languages include many value objects in their native APIs. Java, for example, has myriad representations of date and time information. These types are immutable, making them easily transferrable, easily serializable, simple to copy, and safe to share. Dispensing with the notion of identity frees the developer of many concerns.

Some ORMs offer built-in features for treating their ORM entities as value objects. Java’s Hibernate ORM has an @Immutable annotation that prevents any modifications from being persisted to an entity in the database. A developer can create a read-only ORM entity by carefully using language features or other libraries.

A common pattern I use is applying Hibernate’s @Immutable annotation to the @Entity class and only providing getters. This effectively creates a read-only entity that Hibernate will not persist changes to, and that cannot be modified after creation. Vlad Mihalcea (2019) (the Hibernate guru, in my opinion) provides a great explanation of this approach. The image below, borrowed from his article, shows the UML for such a class.

Java 17 brought another great solution with the Java Record type. Java Records are natively read-only. Using a Domain-Driven Design service that queries the ORM and maps ORM entities into Java Records will achieve the same result. Finally, Java Records are also an excellent solution for DTOs.

Aggregates

Turning again to Evans (p. 125), we find the concept of an aggregate. Aggregates and Entities are closely related concepts because aggregates must always contain an entity that acts as its root. Aggregates simplify the management of complex object graphs by (1) enforcing invariants amongst the relations and (2) ensuring the atomicity of the related objects in transactions.

Developers familiar with modern enterprise applications have likely used C#’s Entity Framework or Java’s Jakarta Persistence API (JPA). These Object-Relational Mapping (ORM) frameworks center on their concept of entity classes that map directly onto a persistence source.

Developers might assume that registering a class as an entity in one of these frameworks makes it an entity in the DDD sense. This is not always the case. Consider a value lookup table of states or territories. To perform CRUD (create, read, update, delete) operations via an ORM framework, you would need to register that class as an entity. You must assign one or more of the class’s fields as the entity ID. Such an object does not fit the definition of a Domain-Driven Design entity. They are defined primarily by their attributes (values) rather than their identity; they do not represent a thread of identity that runs through time and across representations, and they will never need to be matched to another object when their attributes differ. In short, they are Value Objects treated as entities by the ORM.

As a concrete example, one would not care which copy of a State object representing Colorado was received as long as its attributes hold the correct values. Unfortunately, if the developer wishes to manage the list of available states in a database, the ORM library would require that states be treated as ORM entities. I am not denigrating ORMs, as every record in a database must be treated as an entity because records do represent a thread of identity that runs through time and records are defined primarily by their identity (not their attributes). This is necessary because ORMs must track changes to each record, synchronize the state between the database and its first-level cache, and perform other similar tasks.

The bottom line is there is a certain amount of impedance mismatch that ORMs cannot abstract away. They must track each record individually between the middle-tier application and the persistence layer — the specific record holding Colorado’s information. On the other hand, developers view those records in the context of their domain — Colorado is a state, and any copy of Colorado will do.

How DTOs fit into the Mix

Common Arguments for DTOs

Personal experience and an Internet search yielded several common arguments for using DTOs with APIs rather than exposing entities. Before examining three frequently encountered situations where developers must choose between DTOs and entities, a study of the arguments below will provide insight into the qualities of DTOs. Then, we can apply those insights to make better-informed decisions when one of those situations arises.

Argument 1: DTOs hide implementation details of your persistence layer

This is a compelling argument at first glance. However, most serdes (Serialization-Deserialization) utilities (e.g., Jackson) allow you to omit certain fields from being (de)serialized; thus, you achieve the goal without adding another class to maintain. More importantly, however, do you need to hide implementation details? DTOs are not free; they add complexity and more code to your software. If your team will be the only users of your API, does it matter if you expose implementation details? Is the juice worth the squeeze?

Argument 2: Handling object references (associations)

This argument refers to the relationships between data elements (e.g., foreign key references in a relational database). I have found these situations arise most often during the (de)serialization process and usually appear as stack overflow exceptions due to cyclic references between parent-child objects. Normally, they can be easily resolved by omitting one end of the relationship (e.g., telling your serdes utility not to serialize the reference from the child back to the parent).

Argument 3: Use-case-specific API design

This point describes designing APIs for third-party users. In such cases, the developer seeks to make the API as user-friendly as possible by tailoring each endpoint to a specific purpose. It follows that the request and response payloads should also be purpose-driven in their structures. There is really no debating this item, as DTOs are the clear winner here.

Argument 4: API versioning

This argument is really a thin veil over a deeper question of decoupling your API from your persistence layer. As with the first question (of hiding implementation details), you must ask yourself if the additional complexity and maintenance are worth the looser coupling. Even if you plan to offer your API for third-party use, I suggest you hold off on unnecessary DTOs if your only motivation is API versioning. You can always introduce DTOs as a backward compatibility solution when deploying a new version.

Argument 5: Security

This argument normally stems from a lack of understanding of one’s ORM and is expressed in terms of injection or mass assignment vulnerabilities. The essence of either concern is that attackers might modify fields that the developer intended to restrict.

First, remember that ORM entities are not inherently more dangerous than DTOs once detached from the ORM session. Second, remember that they are necessarily detached by serialization when traveling between the front and back ends. Therefore, a field’s value can only be modified if the developer reattaches and merges an ORM entity received by the API.

That being said, if you find yourself fetching the current state of an entity from the data store and copying specific fields from a received ORM entity into it, you may want to consider introducing a DTO. If, instead, you are simply storing the entire modified entity, a DTO would be no more or less safe than what you are already doing.

Use Cases Where You Must Choose Entities or DTOs

There is no shortage of opinions about whether one should expose ORM entities to an API. As I sat brainstorming for this post, these are the three most common use cases where I encountered that question:

1. When an entire object graph is required

2. When you want to allow partial modifications by API calls

3. When several data elements must be modified atomically

I realize this list is not exhaustive, but it offers a solid basis for examining the important aspects of making this decision.

Use Case 1: The Object Graph

Regarding object-relational mapping, an object graph refers to the graph formed by object references within an entity. DDD would call this graph an aggregate.

If your requirements call for returning an entire object graph from an API endpoint, using a DTO is unlikely to be worth the additional effort and complexity it introduces. I recommend using the ORM entity until you are compelled by Arguments 3 or 4 above. Similarly, if you need to allow API users to update an entire entity graph, I don’t believe DTOs would be worth the cost.

If you only require a subset of the fields contained by the object graph, keep reading…

Use Case 2: Partial Modifications by API Calls

Permitting partial updates to the object graph does not always require the introduction of DTOs. I have frequently encountered situations where I need to allow users to modify all but one or two fields of an entity (e.g., allowing users to update their profiles while preventing them from changing their roles and permissions). In trivial systems, it hardly makes sense to create an entirely separate object to simply omit one or two fields.

Instead, I code my API methods to receive the request payload as the ORM entity class. Remember the class is received as a detached entity and makes no changes to the persistence layer. Next, I fetch the current state of the entity from the persistence layer — this ORM entity is attached. Finally, I copy properties from the detached instance to the attached instance while excluding those fields I want to remain unmodified.

I have found a couple of good approaches to copying fields. The first uses either Spring Framework’s BeanUtils class or Apache Commons BeanUtils’ BeanUtilsBean class.

Spring’s option lends itself better to these needs. It has the signature:

void copyProperties(Object source, Object target, String... ignoreProperties)

An example of this approach would look like:

@PutRequest
ResponseEntity<void> updateProfile(@RequestBody User user) {
    final User attachedUser = userRepo.findById(user.getId());
    BeanUtils.copyProperties(
        user,
        attachedUser,
        "roles", "permissions"
    );
    userRepo.save(user);
    return ResponseEntity.ok().build();
}

A second approach is using a mapping library. The most notable example in Java is ModelMapper. ModelMapper enables you to handle very complex situations and is probably overkill for copying fields between two objects of the same class. I will cover it in more detail in the next part of this series, where I show you how to introduce DTOs in a manageable way.

For the sake of completeness, however, here is an example of using ModelMapper to copy from a detached entity to an attached one while omitting certain fields:

static final ModelMapper mapper = new ModelMapper();
static final TypeMap<User, User> safeUserMap = mapper.createTypeMap(User.class, User.class, "safeUserMapper");

static {
    safeUserMap.addMappings(mapper -> {
        mapper.skip(User::setRoles);
        mapper.skip(User::setPermissions);
    });
}

@PutRequest
ResponseEntity<void> updateProfile(@RequestBody User user) {
    final User attachedUser = userRepo.findById(user.getId());
    mapper.map(user, attachedUser, "safeUserMapper");
    userRepo.save(user);
    return ResponseEntity.ok().build();
}

I would be remiss if I failed to mention Spring’s @Projection feature. It is extremely handy when you only need a subset of a single entity’s data fields. Using Spring Data, you can create an interface that contains only the accessors (i.e., getters) of an entity that you need. If you code methods of a Spring Repository to return the projection type, Spring will query for only those fields and map them into an implementation of that interface (i.e., a DTO). If you also wish to receive a projection, you would need to build a DTO class that implements the interface (as Jackson cannot deserialize to an interface without an implementation).

Here’s a quick example of using the @Projection feature:

@Entity
class User {
    @Id
    Long userId;
    Collection<String> roles;
    String firstName;
    String lastName;
    String password;
}

@Projection
interface SafeUser {
    Long getUserId();
    String getFirstName();
    String getLastName();
}

@Repository
interface UserRepository extends JpaRepository<User, Long> {
    Optional<SafeUser> findByUserId(Long userId);
}

@RestController
class UserController {
    @Autowired UserRepository userRepo;
    
    @PutRequest("/user/{id}")
    SafeUser fetchUser(@PathVariable Long id) {
        return userRepo.findByUserId(id).orElseThrow(() =>
            new ResponseStatusException(HttpStatus.NOT_FOUND));
    }
}

Use Case 3: Atomic Modifications by API Calls

Special care must be taken when an API endpoint receives a collection of fields that must be applied to multiple entities atomically.

Atomic modifications must be executed as a single unit that either completely succeeds or is rolled back.

I cannot imagine a use case 3 situation where you would not want to use a DTO. Any alternative would require multiple API calls to send the various entities involved (which would make atomicity all but impossible), or it would require creating a synthetic entity that acts as a wrapper object for the others. Both options are unappealing.

Instead, you should create a DTO that carries the involved fields. You should also create a Data Mapper (Fowler, p. 165) that moves data between your ORM layer and the DTO. Finally, you should persist the updated ORM entities atomically within a single transaction.

Consider a point-of-sale system for a vehicle service center. When a customer purchases a service package, several updates must be made atomically to the center’s database:

• Money must be transacted

• Lubricant and other supply inventory must be updated

• The vehicle’s service record must be updated

• History of the driver’s visit must be recorded

The DTOs for this use case might look like this:

class LineItemDto {
    private LineItemType type; // service or material
    private Optional<BigDecimal> quantity; // amount or hours
    private Long lineItemId;
}

class ServiceTransactionDto {
    private Collection<PurchaseLineItem> lineItems;
    private String vehicleVin;
    private Long customerId;
}

These two simple classes hide a lot of complexity within the underlying persistence layer. Thankfully, the API endpoint need only know about the ServiceTransactionDto and the Data Mapper that will handle creating service records for the vehicle, billing the client, updating inventory, and more. Using the Spring Framework, the API endpoint might look like this:

@Transactional
@PutRequest
void updateProfile(@RequestBody ServiceTransactionDto serviceTx) {
    serviceTxMapper.update(serviceTx);
}

Behind the scenes, the data mapper will use the DTO’s data to apply updates to many other entities. While the Data Mapper pattern is out of this article’s scope, I encourage you to read that section of Fowler’s book.

Conclusion

Key takeaways from this article are:

• ORM entities are not necessarily DDD entities

• DTOs do not always make sense — when given the choice, do a cost-benefit analysis before introducing DTOs

• There are several common arguments for using DTOs, but only a couple really hold water

• I’ve found three use cases where I frequently have to choose between exposing entities or creating DTOs — I present several options for dealing with these

Check back for part two of this series for tips on effectively managing DTOs.

• Evans, E. (2003). Domain-Driven Design: Tackling Complexity in the Heart of Software. Pearson Education, Inc.

• Fowler, M. (2002). Patterns of Enterprise Application Architecture (1st ed.). Addison-Wesley Professional.

• Mihalcea, V. (2019, August 20). How to map an immutable entity with JPA and Hibernate - Vlad Mihalcea. Vlad Mihalcea. https://vladmihalcea.com/immutable-entity-jpa-hibernate/

What DevOps did for software development, GitOps has done for infrastructure and deployment. Virtualization and containerization were the advent of a revolution that saw organizations get the most out of their capital investments, cut costs everywhere possible, and speed up much of their operations. 

Surprisingly, IT infrastructure still lacked the structure, discipline, and agility that have been commonplace in development for over a decade. The introduction of declarative provisioning (Infrastructure as Code) technologies like Terraform, AWS CloudFormation, Docker Stack, and Kubernetes Objects brought infrastructure a step closer to rectifying these issues because their code could be stored and managed just like other source code. At first glance, Git is a surprising tool for managing infrastructure, but it makes sense when one views it as a configuration management system, not just a version control system. This arrangement was still lacking in essential ways — notably, automation (of applying changes) and reconciliation (correcting drift caused by changes made to the environment outside of IaC). 

In 2017, Weaveworks CEO Alexis Richardson coined the term GitOps(Weaveworks, 2021). It was the culmination of his organization’s slow progression from practicing a form of GitOps-lite to today’s accepted standard. Around that time, Weaveworks released its Weave Flux tool that provided the two final pieces — automation and reconciliation. Unsurprisingly, Richardson himself defined the standard for GitOps:

1. The entire system is described declaratively

2. The canonical desired system state is versioned in Git

3. Approved changes that can be automatically applied to the system

4. Software agents to ensure correctness and alert on divergence

There still needs to be a formal or official definition for GitOps. The four principles above are the closest thing available to such a definition. It is possible, however, to extrapolate a definition based on the GitOps principles and other authoritative sources. As a springboard for the remainder of this article, let us agree to define GitOps as:

The use of Git to enforce technical reviews, version control, and canonical recording of changes to Infrastructure as Code combined with the use of other tools to automate the deployment of changes to IaC, eliminating divergence of the system’s actual state from the state in Git.

How GitOps Works

1. Developers (the stick figure person)

2. Reconciler (the GitOps management platform)

3. Convergence Mechanism (the system applying IaC to an environment — Kubernetes, CloudFormation, Terraform, etc.)

These three actors work together in an elegant dance we call GitOps. Richardson (2018) choreographs that dance as four distinct steps:

1. A developer describes the desired state of the system using a declarative syntax (e.g., Kubernetes objects or Terraform) and commits that code to a Git repository. Ideally, another developer must approve a pull request before the changes are merged into the master or main branch.

2. The reconciler detects that the desired state (what is reflected by the Git repository) and the actual state are not the same. It activates the new state by applying it to the convergence mechanism.

3. The convergence mechanism adjusts the actual state to match the desired state. Thus, Git commits “cause verifiable and idempotent updates in the cluster” (idempotent means an action may be repeated any number of times but always results in the same outcome).

4. Convergence is achieved once there are no more “‘diff’ alerts during a defined time interval.” Convergence is the state where the desired state and actual state match.

What do you get from GitOps?

GitOps borrows many of its benefits from its most fundamental component, Infrastructure as Code. These benefits include:

• Consistency: all infrastructure (including both servers and containers) is defined in the same way

• Security: all configurations are codified in a single place rather than manually applied in an infinite number of ways

• Access control: all system changes must pass through a single spot and (ideally) go through a peer review; further, changes are applied by an automated software agent

• Auditing: all changes are retained in version history

• Authority: there is a single source of truth for the system’s desired state

GitOps also adds the benefits of continuous delivery, including:

• Repeatability: changes are deployed the same way, every time

• Speed: deployments are faster

• Reliability: automation eliminates human error arising from manual deployments

• Visibility: most of the popular tools include dashboards of their own and integrate with other monitoring systems

Survey of GitOps Management Platforms

The leaders of the GitOps market generally fall into one of two categories: those managing application deployment and those managing infrastructure provisioning. It is important to note how the term “infrastructure” is somewhat overloaded. While it generally refers to resources provisioned on your cloud platform (e.g., AWS, GCP, Azure), when used in “Infrastructure as Code,” it also refers to resources provisioned by a container orchestrator as part of an application deployment.

It is important to note how the term “infrastructure” is somewhat overloaded. While it generally refers to resources provisioned on your cloud platform (e.g., AWS, GCP, Azure), when used in “Infrastructure as Code,” it also refers to resources provisioned by a container orchestrator as part of an application deployment.

GitOps Managers for Application Deployment

Flux is a free and open-source Kubernetes application that reconciles the state of a Kubernetes environment to match IaC code stored in a Git repository. It is a tool managed by the Cloud Native Computing Foundation (CNCF) and was created by Weave Works. It does not include a UI and is more often used as the core of more comprehensive GitOps tools.

Weave GitOps Enterprise is a comprehensive GitOps management platform built by Weave Works. It is a paid option that uses Flux as its reconciler. It comes with a mature and user-friendly web-based GUI. This product also supports Terraform GitOps by including Weave Works’ Terraform Controller as an additional reconciler. 

ArgoCD is one of the most popular free and open source GitOps platforms. ArgoCD is a full-featured option that includes a very mature and advanced GUI. One of its most exciting features is its display of Kubernetes deployments as graphs that update in real-time. It treats Kustomize as a first-class concern and automatically recognizes Kustomize projects.

Jenkins X takes an even more comprehensive approach by automating both CI and CD. It unifies the integration of applications and deployment of their artifacts with the deployment of infrastructure changes. Jenkins X seeks to act as a “GitOps appliance” that just works. One particularly useful feature is its tight integration with cloud computing environments. Under the hood, it leverages Terraform to use cloud-native services, regardless of the provider. An important thing to note is that Jenkins X is exclusively controlled via CLI and config files in Git. It offers two GUIs, but both are read-only.

GitLab is the Swiss Army knife of CI/CD platforms. It does everything, including serving as your Git repository, continuous integration, continuous deployment, security scanning, and more. It uses Flux as its reconciler to synchronize Kubernetes with IaC files. GitLab also integrates tightly with Terraform environments, providing and managing your cloud infrastructure.

GitOps Managers for Infrastructure Provisioning

Atlantis is a barebones, battle-tested tool for automating Terraform via GitOps. Its primary user interface is a sequence of Git comments on pull requests. Atlantis lacks support for drift detection. If you only want to automate Terraform through GitOps and self-hosting is required, this is the tool for you.

Terraform Cloud is a complete solution for using the Terraform tool for GitOps. It includes collaborative features for teams, native GitOps support, drift detection, and more. Terraform Cloud is primarily a SaaS offering, but their Enterprise pricing plan does include a self-hosted option.

Spacelift is a direct competitor to Terraform Cloud. Spacelift seeks to be a unified GitOps tool by supporting all the major Infrastructure as Code offerings. These include OpenTofu, Terraform, Terragrunt, CloudFormation, Pulumi, Kubernetes, and Ansible. It goes a step further by also supporting Policy as Code by leveraging Open Policy Agent and its Rego language. Like Terraform Cloud, Spacelift is, first and foremost, a SaaS offering but also has a self-hosted option attached to its Enterprise pricing plan. Spacelift differentiates itself with its clean and intuitive user interface. It also has a unique feature that allows testing IaC modules before deploying them.

env0 is another offering that is very similar to Terraform Cloud and Spacelift. 

It supports OpenTofu, Terraform, Terragrunt, CloudFormation, Pulumi, Kubernetes, and Helm. It differentiates itself with a robust templating feature. DevOps teams can create useful templates and allow developers downstream to deploy these templates. Essentially, downstream teams can choose from a catalog of templates and deploy ready-to-use environments for their projects.

Setting up GitOps Locally

So far, this article has built a foundation of understanding for GitOps, how it works, its benefits, and the leaders in the GitOps marketplace. It is time to put this all into practice by building, configuring, and testing a GitOps stack. The tools chosen for this task include GitHub, Docker Hub, Kubernetes, Kustomize, and ArgoCD. All example CLI commands are based on macOS.

The Flow (or the Vision)

Before rolling up our sleeves, we need a vision for the outcome. The best way to define that vision is by describing how the GitOps process should work upon completion.

Here is an outline of that flow:

1. Developers commit Kubernetes/Kustomize IaC changes to a GitHub repository

2. When changes are merged to the main or master branch, the reconciler (ArgoCD) applies the changes

3. When a new version of the application is pushed to Docker Hub with the sha-##### tag, the reconciler detects this and pulls that image for the dev namespace

4. When a new version of an image is pushed to Docker Hub and tagged as v#.#.#-RC (release candidate), it is pulled for the test namespace

5. When a new version of an image is pushed to Docker Hub and tagged as v#.#.#, it is pulled for the prod namespace

One additional requirement is load-balancing across replica sets.

Installing and Configuring Kubernetes (k8s)

Install the Tools

The easiest way to set k8s up for a local environment is via Docker Desktop. 

1. On a Mac, you can install Docker Desktop via brew by running:

brew install --cask docker

1. Open Docker Desktop, navigate to Settings and select Kubernetes.

2. Check the Enable Kubernetes option and click Apply & restart.

You will also want to install a few Kubernetes utilities:

brew install kubernetes-cli kubectx k9s

These add the kubectl, kubectx, and k9s commands to your computer, respectively. kubectl is the primary CLI tool for managing k8s. kubectx allows you to view k8s contexts (clusters) and select the one you wish to interact with. You probably won't need this unless you already have multiple clusters configured. Finally, k9s is a TUI (textual user interface) for managing k8s. It is particularly useful when editing config files, scaling replica sets, and other common tasks.

Configure k8s Namespaces

Namespaces allow separating k8s objects with identical names. For our purposes, they will act as various deployment environments. We will need four namespaces: argocd to store ArgoCD’s applications, and dev, test, and prod, for our application deployments.

In a terminal, execute these commands:

kubectl create ns argocd
kubectl create ns dev
kubectl create ns test
kubectl create ns prod

MetalLB…a Load Balancer

While k8s Replica Sets allow horizontal scaling of applications, they are useless if you cannot distribute traffic across all the active instances. Normally, k8s clusters will leverage load balancers supplied by their cloud providers. When running k8s locally, however, a k8s-native load balancer like MetalLB is required.

To add it to the local k8s cluster, execute this command in the terminal:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-frr.yaml

Before it can be used, MetalLB must be told what IP addresses it can choose from when selecting an IP to announce for a k8s service. CIDR blocks are used to specify these range(s). In the case of this example, we need the addresses used by the load balancer to loop back to the host. Thus, we’ll allocate a small section of the loop back CIDR block: 127.0.0.240/28. This is a bit of a “trick” because even though MetalLB might assign a service the IP 127.0.0.241 and expose it on port 8882, you will still access it at 127.0.0.1:8882 in your browser. 

Now, configure MetalLB to use this IP CIDR block by creating an IP Address Pool object. Then, tell MetalLB to announce these new IP addresses to the local network via ARP. 

Copy the code below into a text file named lb-ip-config.yaml:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: localhost-address-pool
  namespace: metallb-system
spec:
  addresses:
    - 127.0.0.240/28
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: ipadvertisement
  namespace: metallb-system

Apply this file to your k8s cluster: kubectl apply -f lb-ip-config.yaml

What did we just do?

We need a load balancer to allocate an IP address for k8s services and distribute traffic across instances of that service’s app. Above, we configured MetalLB as our load balancer and told it to select from the IP address range represented by the CIDR block 127.0.0.240/28. 

Later, we will deploy a horizontally scalable app — meaning multiple copies of the app run simultaneously. In k8s-speak, this is known as a replica set. Although we will be running three copies of the app, we want it to appear as a single app to our users. We also want to load balance traffic across the replica set. To do this, we will create a Kubernetes Service of type LoadBalancer. MetalLB will be used as the load balancer. 

In production, a cloud provider’s load balancer would be used rather than MetalLB. For example, Amazon Web Services’ Elastic Load Balancer (ELB) might be configured as a load balancer and would expose our services.

Installing and Configuring ArgoCD

Before testing the load balancer, we need an app to deploy. Before we can deploy an app, we need a Continuous Deployment framework to automate those deployments. Thus, we need to set up our GitOps tool: ArgoCD.

Installing ArgoCD

We created the argocd namespace earlier, so all we need to do here is apply the k8s deployment:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Once that’s complete, we must expose our new ArgoCD server. Because we’ve already configured MetalLB, let us leverage that by configuring the ArgoCD Service as type LoadBalancer:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

Confirm everything is working correctly by navigating to https://localhost.

Because ArgoCD uses a self-created certificate, you must tell your browser to navigate to the site despite its misgivings.

Installing ArgoCD Image Updater

By itself, ArgoCD solves one of two automation problems — it ensures deployments match, in reality, what is described by their IaC (Infrastructure as Code). However, there is no out-of-the-box mechanism for updating the Docker image used by a deployment as new versions of that image are released. This means that ArgoCD cannot deploy application updates automatically. 

There are two obvious ways of approaching this problem. First, you could specify an exact image tag to use in your k8s code (e.g., my app:v1.1.3). As you create new versions of your app, you could change that tag and merge those changes into the repository where your k8s code files are kept. I don’t particularly like this option because it conflates responsibilities. The k8s code is responsible for defining the structure of a deployment. Using it to define the version of the app being deployed is an anti-pattern. 

This leads to the second approach: introduce a second reconciler that tracks Docker image versions and ensures ArgoCD always uses the latest version. This is what the ArgoCD Image Updater does. 

The simplest way to install Image Updater is via Kubernetes manifest. You just need to execute this code in your terminal:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

Note that I encountered an issue with the Image Updater refusing to accept Docker Hub’s x509 certificate. To resolve it, I patched the config. I recommend you do the same for these purposes (in a real-world deployment, find a way to resolve the issue that doesn’t impact your security posture).

 kubectl patch -n argocd configmap/argocd-image-updater-config --patch 'data:
  log.level: debug
  registries.conf: |
    registries:
    - name: Docker Hub
      insecure: yes
      api_url: https://registry-1.docker.io
      prefix: docker.io
      defaultns: library'

Registering a GitHub Repository

The final task before we build our app is creating a GitHub repository and registering it with ArgoCD. 

Create a GitHub Repository

Create a GitHub repository named patient-portal-argocd (this is to be consistent with examples from previous blog posts). Ensure you initialize it with a README file. If the new repository does not have content or a branch, ArgoCD will fail to connect to it.

Next, create a new SSH key pair. In your terminal, navigate to ~/.ssh. Execute ssh-keygen. At the first prompt, name the key pair argocd. Press ENTER to accept the defaults for all subsequent prompts.

In your terminal, execute cat ~/.ssh/argocd.pub. Copy the output to your clipboard.

In GitHub, navigate to your account settings and select “SSH and GPG keys” in the navigation bar. Add a new SSH key. Name it “ArgoCD” and paste the contents of your clipboard into the Key textbox.

Register the GitHub Repository with ArgoCD

Before we proceed, we need to get your ArgoCD admin password. In your terminal, execute k9s to open the k9s Kubernetes tool.

Type :secrets. Highlight argocd-initial-admin-secret and press the x key. Copy the password shown.

Open your browser and navigate to: https://localhost. Log into ArgoCD using the username admin and the password you copied.

Click Settings on the left side of the screen. Choose Repositories and then + Connect Repo. 

Name the repository GitHub patient-portal-argocd. Select default for Project. Copy and paste the GitHub repository SSH URL to the Repository URL textbox in ArgoCD. For example, it should look similar to: git@github.com:____/patient-portal-argocd.git .

In your terminal, execute: cat ~/.ssh/argocd. Copy the output and paste it into the SSH private key data textbox in ArgoCD. Click the CONNECT button.

What did we just do?

You accomplished three major tasks:

1. We installed ArgoCD on your Kubernetes cluster and configured it to be served up by the MetalLB load balancer

2. We installed the ArgoCD Image Updater that will ensure any Kubernetes deployments always have the latest version of Docker images

3. We created a new GitHub repo to store our IaC and configured ArgoCD to add that repository to ArgoCD

Later, we will actually create some IaC files, push them to the patient-portal-argocd repository, and configure ArgoCD to deploy that project.

Creating a Simple Vite/VueJS Project

We need an app that can demonstrate a successful configuration of all parts of our project. At a minimum, it should do the following:

1. Display the current Docker image version (i.e., tag name)

2. Display the pod’s hostname

3. Be easily testable and buildable by GitHub actions

4. Be packageable as a single Docker image

Vite is a tool that makes it extremely easy to scaffold, build, and package NodeJS apps based on common frameworks like VueJS, React, and Svelte. 

Building the App

Scaffold the Project

Navigate to a directory where you keep your coding projects. Execute the code below:

npm create vite-extra@latest patient-portal -- --template ssr-vue

When prompted, simply accept the defaults (respond with Yes)

Create a Git Repository for the App

Create a new GitHub repository named patient-portal. Do not initialize it with anything.

In your terminal, navigate to the new project folder (created in the step above — Scaffold the Project) and execute the following commands. Ensure you replace the blank with your repo’s SSH URL!

git init
git add .
git commit -am "Initial commit"
git branch -M master
bit remote add origin _______
git push -u origin master

Display Git and Host Information

We will use the dotenv utility to provide our app with version information. Vite already integrates this, but we must create a .env.local file and tell Vite where to find it.

Create a directory called env at patient-portal/src/env. In that directory, create a file named .env.local and paste these contents into it:

VITE_GIT_REF=v1.0.0

Next, edit patient-portal/src/vite.config.js so it matches this code:

import { defineConfig } from 'vite'
import vue from '@vitejs/plugin-vue'

export default defineConfig({
  plugins: [vue()],
  envDir: 'src/env'
})

We will inject the hostname into VueJS’ server-side rendering (SSR) context. To do this, edit patient-portal/src/entry-server.js so it matches this code:

import { renderToString } from 'vue/server-renderer'
import { createApp } from './main'
import * as os from "os";

export async function render() {
  const { app } = createApp()

  const ctx = {
    hostname: os.hostname()
  }
  const html = await renderToString(app, ctx)

  return { html }
}

Finally, let’s rework the default HelloWorld Vue component to display the two bits of information we want. Edit patient-portal/src/components/HelloWorld.vue so it matches this code:

<script setup lang="ts">
import {ref, useSSRContext} from 'vue'

defineProps({
  msg: String
})

const gitTagCommit = ref(import.meta.env.VITE_GIT_REF);
let ctx : Record<string, any> | undefined;

if(import.meta.env.SSR) {
  ctx = useSSRContext();
}
</script>

<template>
  <h1>{{ msg }}</h1>

  <div class="card">
    <span><strong>Git Tag/Commit</strong></span>
    <span class="data">{{ gitTagCommit }}</span>
    <span><strong>Host</strong></span>
    <span class="data">{{ ctx?.hostname ?? '' }}</span>
  </div>
</template>

<style scoped>
.card {
  display: grid;
  grid-template-columns: 10rem auto;
  row-gap: 0.5rem;

  & span {
    display: flex;
    flex-direction: column;
    justify-content: center;
    align-items: center;
  }
}

.data {
  background-color: #1a1a1a;
  border-radius: 8px;
  padding: 0.6em 1.2em;
  font-size: 1em;
  font-weight: 500;
  font-family: inherit;
}
</style>

Your patient-portal directory structure should look like this:

.
├── README.md
├── index.html
├── package.json
├── public
│   └── vite.svg
├── server.js
├── src
│   ├── App.vue
│   ├── assets
│   │   └── vue.svg
│   ├── components
│   │   └── HelloWorld.vue
│   ├── entry-client.js
│   ├── entry-server.js
│   ├── env
│   │   └── .env.local
│   ├── main.js
│   └── style.css
└── vite.config.js

Updating the Git Commit Hash and Tags Automatically

We want to ensure the Git ref information is updated each time the app is built for our dev, test, or prod environments. Tags will be used to trigger deployments to test and prod; therefore, we want to show a commit for dev and tag names for test and prod. The simplest way to do this is to use a build script that updates the .env.local file before building. Of special note, the .env.local file is ignored by Git, so it will be freshly created by GitHub Actions every time.

Navigate to your patient-portal project’s root directory and run the commands below:

cat > build.sh << EOF
write_properties () {
  echo "Building for \$1"
  
  mkdir -p ./src/env
  
  if [ \$1 = 'dev' ]; then
    echo "VITE_GIT_REF=\$(git rev-parse --short HEAD)"
    echo "VITE_GIT_REF=\$(git rev-parse --short HEAD)" > ./src/env/.env.local
  elif [ \$1 = 'test' ] || [ \$1 = 'prod' ]; then
    echo "VITE_GIT_REF=\$(git describe --tags)"
    echo "VITE_GIT_REF=\$(git describe --tags)" > ./src/env/.env.local
  else
    echo "Argument 1 must be dev, test, or prod. \$1 is not recognized."
    return 1
  fi
  npm run build
}

write_properties \$1
EOF

chmod +x build.sh

Finally, ensure you commit and push all your changes:

git add .
git commit -am "Modified project to show version data and added automated build script."
git push

Test the App

Navigate to the application’s root directory in your terminal and execute the following commands:

npm install
./build.sh dev
npm run dev

If all goes well, you should be able to navigate to the URL specified in the command output. Your URL may not be the same as the one in the image below.

The web page should look similar to this:

Packaging the App in a Container

We have a functioning app, but we need to package it in a Docker container so we can deploy it to a Kubernetes cluster.

In the root of your project, create a new file named Dockerfile. Use the following code for its contents:

FROM node

RUN mkdir /opt/app
COPY package.json /opt/app
COPY dist/  /opt/app/dist
COPY server.js /opt/app

WORKDIR /opt/app

RUN npm install

ENV NODE_ENV=production
ENV PORT=8080

ENTRYPOINT ["npm", "run", "serve"]

Create a Docker Hub Image Registry

If you do not already have a Docker hub account, create one. Then, create a new registry named patient-portal.

Navigate to https://hub.docker.com/settings/security and create a new Access Token. Ensure you copy it somewhere for later use.

Configuring the App’s GitHub Actions

Now, it is time to configure continuous integration for the application. It should follow a flow similar to the one outlined for our ArgoCD IaC repository at the beginning of this article:

1. When changes are merged to the main or master branch, a new Docker image should be published with the latest tag

2. When a prerelease is created, a new Docker image should be published that has the v#.#.#-RC suffix

3. When a release is created, a new Docker image should be published that has the v#.#.# suffix

In the root directory of your project, create a new directory and subdirectory with this structure: .github/workflows/

Create a file there named github-actions.yml with these contents:

name: Node.js CI

on:
  workflow_dispatch:
  release:
    types: [ published, prereleased, released ]
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]

env:
  TARGET_TEST: ${{ github.event.release != null && contains(fromJson('["created", "prereleased", "released"]'), github.event.action) && github.event.release.prerelease }}
  TARGET_PROD: ${{ github.event.release != null && contains(fromJson('["created", "prereleased", "released"]'), github.event.action) && !github.event.release.prerelease }}

jobs:

  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18.x]
    steps:
      - uses: actions/checkout@v3
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'
      - run: npm ci
      - run: |
          if [[ $TARGET_PROD = true ]]; then 
            ./build.sh prod; 
          elif [[ $TARGET_TEST = true ]]; then 
            ./build.sh test; 
          else 
            ./build.sh dev; 
          fi
      - uses: actions/upload-artifact@v3
        with:
          name: dist
          path: dist

  publish:
    needs: build
    if: ${{ github.ref_name == github.event.repository.default_branch || contains(fromJson('["created", "prereleased", "released"]'), github.event.action) }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18.x]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/download-artifact@v3
        with:
          name: dist
          path: dist
      - run: npm ci
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Log in to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: |
            docker.io/_______/patient-portal
          tags: |
            type=sha,enable=${{ github.ref_name == github.event.repository.default_branch && github.event.release == null }}
            type=semver,pattern={{raw}},suffix=-RC,enable=${{ github.event.release != null && github.event.release.prerelease == true }}
            type=semver,pattern={{raw}},enable=${{ github.event.release != null && github.event.release.prerelease == false }}
      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          file: ./Dockerfile
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

Notice that this file relies on two secrets that provide your Docker hub username and access token. 

Ensure you modify this value, docker.io/_______/patient-portal, in the YAML file to replace the blank with your Docker hub username.

Navigate to your patient-portal GitHub repository in your browser. Go to the Settings tab. Expand Secrets and variables in the left navigation bar. Choose Actions.

Create two new Repository secrets:

• DOCKERHUB_TOKEN = the access token you created in the previous section

• DOCKERHUB_USERNAME = your Docker hub username

Test Your App’s CI

Build a Dev Image (with tag latest)

Finally, we get to test the continuous integration of the app. To do this, you only need to commit and push everything to GitHub:

git add .
git commit -am "Adding GitHub Actions CI"
git push

Navigate to the Actions tab of your GitHub repository, and you should see a build running. When it is complete, you can navigate to your Docker Hub and verify a new image was added with the latest tag and a tag similar to sha-7494edf.

Build a Test Image (with tag v1.0.0-RC)

Navigate to your repository’s Releases by clicking on Create a new release in the right-hand column:

Click Choose a tag, enter v1.0.0, and click Create a new tag.

Enter “v1.0.0” for the Release title.

Select the Set as a pre-release checkbox.

Click Publish release.

When the pre-release publishes a Docker image, it will suffix the Git tag with -RC and use the result as the image tag. This is achieved by line 73 of the GitHub Actions YAML file.

Build a Prod Image (with tag v1.0.0)

Once we have a pre-release, publishing a release is as simple as unchecking the Set as a pre-release checkbox.

While still on your repository’s Releases screen, click the edit button (a pencil) for pre-release v1.0.0. 

Uncheck Set as a pre-release. 

Then, click Update release.

When the release publishes a Docker image, the Git tag will be used as the image tag. This is achieved by line 74 of the GitHub Actions YAML file.

What did we just do?

We scaffolded a new VueJS/Vite project and modified the project to show the current Git tag/hash and the hostname serving the app. This allows us to confirm that the correct Docker image is being used in each of our dev, test, and prod environments. It also allows us to test the load balancer to ensure it distributes traffic across the replica set.

We also packaged the app into a Docker container by creating a Dockerfile, configured continuous integration via GitHub Actions, and pushed the app to GitHub. Then, we confirmed the app was built correctly and that the latest tag of the Docker image was published to Docker hub.

Finally, we confirmed that a GitHub pre-release publishes a Docker image with the tag format of v#.#.#-RC and we confirmed that a GitHub release publishes a Docker image with the tag format v#.#.#.

Deploying to Kubernetes

It is finally time to see GitOps in action! We now have a Kubernetes cluster, ArgoCD, and Image Updater installed, as well as an app to deploy. Now, we need to write the IaC code that describes the app’s deployment, deploy that code to ArgoCD, and watch the magic happen!

Writing the IaC

Recall that we already have a GitHub repository named patient-portal-argocd, to store our IaC. Let’s clone that project to our local machine where we will add IaC to it. Navigate to the directory where you keep your GitHub projects and execute the code below. Ensure you replace the blank with your GitHub username.

git clone git@github.com:______/patient-portal-argocd.git 

Base Code

We are using kustomize to simplify our Kubernetes code. It allows patches to be overlaid on base code. The three files below are base code, and we will overlay a few changes for each of our dev, test, and prod environments.

Inside the patient-portal-argocd project, create a directory named base. Add the following three files to the base.

deployment.yaml (Notice the comment in the file!)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: patientportal
spec:
  replicas: 3
  selector:
    matchLabels:
      app_name: patientportal
  template:
    metadata:
      labels:
        app_name: patientportal
    spec:
      containers:
        - name: patientportal
          # Replace the blank with your Docker Hub username
          image: docker.io/____/patient-portal:latest
          imagePullPolicy: IfNotPresent

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: patientportal
spec:
  type: LoadBalancer
  selector:
    app_name: patientportal

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

Dev Overlays

In the patient-portal-argocd project, create a new directory named overlays. Inside that directory, create another directory called dev.

Create the two following files in patient-portal-argocd/overlays/dev.

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: patientportal
spec:
  ports:
  # Serve the dev environment at http://localhost:8787
  - port: 8787
    targetPort: 8080

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: dev
resources:
  - ../../base
patches:
  - path: service.yaml

Test Overlays

In patient-portal-argocd/overlays, create a new directory named test. Inside that directory, create the three following files.

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: patientportal
spec:
  ports:
  # Serve the dev environment at http://localhost:8888
  - port: 8888
    targetPort: 8080

deployment.yaml (Notice the comment in the file!)

Later, we will configure ArgoCD Image Updater to modify the image version (below) with the latest test image. Remember, test images will have tags like v#.#.#-RC.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: patientportal
spec:
  template:
    spec:
      containers:
        - name: patientportal
          # Replace the blank with your Docker Hub username
          image: docker.io/____/patientportal:v1.0.0-RC
          imagePullPolicy: IfNotPresent

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: test
resources:
  - ../../base
patches:
  - path: service.yaml
  - path: deployment.yaml

Prod Overlays

In patient-portal-argocd/overlays, create a new directory named prod. Inside that directory, create the three following files.

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: patientportal
spec:
  ports:
  # Serve the dev environment at http://localhost:8989
  - port: 8989
    targetPort: 8080

deployment.yaml (Notice the comment in the file!)

Later, we will configure ArgoCD Image Updater to modify the image version (below) with the latest prod image. Remember, prod images will have tags like v#.#.#. 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: patientportal
spec:
  template:
    spec:
      containers:
        - name: patientportal
          # Replace the blank with your Docker Hub username
          image: docker.io/____/patientportal:v1.0.0
          imagePullPolicy: IfNotPresent

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: prod
resources:
  - ../../base
patches:
  - path: service.yaml
  - path: deployment.yaml

Commit and Push Your IaC

Before we can fire up our app on ArgoCD, we need to make the IaC available to it by pushing everything to our partient-portal-argocdrepository. In your terminal, navigate to your repository’s directory and execute:

git add .
git commit -am "Added IaC for dev, test, and prod environments"
git push

Create Dev, Test, and Prod Applications on ArgoCD

This step will make our application live in three environments. You can use the ArgoCD UI to accomplish this, but executing the scripts below is less error-prone.

Ensure you replace the blanks in each code block with your Docker Hub and GitHub usernames, respectively.

Deploy to DEV

Deploy the app to the dev environment by executing this script in your terminal (after inserting your Docker Hub and GitHub usernames):

kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: patient-portal-dev
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: devpatientportalalias=docker.io/____/patient-portal
    argocd-image-updater.argoproj.io/devpatientportalalias.update-strategy: latest
    argocd-image-updater.argoproj.io/devpatientportalalias.allow-tags: regexp:^sha-.*$
spec:
  destination:
    name: ''
    namespace: dev
    server: 'https://kubernetes.default.svc'
  source:
    path: overlays/dev
    repoURL: 'git@github.com:____/patient-portal-argocd.git'
    targetRevision: HEAD
  sources: []
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

Deploy to TEST

Deploy the app to the test environment by executing this script in your terminal (after inserting your Docker Hub and GitHub usernames):

kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: patient-portal-test
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: testpatientportalalias=docker.io/____/patient-portal
    argocd-image-updater.argoproj.io/testpatientportalalias.update-strategy: latest
    argocd-image-updater.argoproj.io/testpatientportalalias.allow-tags: regexp:^v\d.\d.\d-RC$
spec:
  destination:
    name: ''
    namespace: test
    server: 'https://kubernetes.default.svc'
  source:
    path: overlays/test
    repoURL: 'git@github.com:____/patient-portal-argocd.git'
    targetRevision: HEAD
  sources: []
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

Deploy to PROD

Deploy the app to the prod environment by executing this script in your terminal (after inserting your Docker Hub and GitHub usernames):

kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: patient-portal-prod
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: testpatientportalalias=docker.io/____/patient-portal
    argocd-image-updater.argoproj.io/testpatientportalalias.update-strategy: latest
    argocd-image-updater.argoproj.io/testpatientportalalias.allow-tags: regexp:^v\d.\d.\d$
spec:
  destination:
    name: ''
    namespace: prod
    server: 'https://kubernetes.default.svc'
  source:
    path: overlays/prod
    repoURL: 'git@github.com:____/patient-portal-argocd.git'
    targetRevision: HEAD
  sources: []
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

What did we just do?

This “Deploying to Kubernetes” section has a lot of code. However, most of it is very repetitive. In a nutshell, here is what we accomplished:

• Described our Kubernetes application deployment using Kustomizations — these included base code and a custom overlay for each environment (dev, test, and prod)

• Created ArgoCD applications based on our Kustomizations by applying ArgoCD Application YAML files to our Kubernetes cluster — we applied one for each environment (dev, test, and prod)

Testing

At this point, if we merge something to the master branch, a new sha-####### image should be pushed to Docker Hub, and ArgoCD Image Updater should update our dev environment.

If we create a new pre-release in GitHub, a new v#.#.#-RC image should be pushed to Docker Hub, and the test environment should be updated.

Likewise, a new release in GitHub should trigger the push of a new v#.#.# image to Docker Hub and the updating of the prod environment.

Finally, if we change any of our Kustomization IaC in the patient-portal-argocd and merge those changes to master, ArgoCD should apply the changes to Kubernetes. For example, assume we set scale our prod environment up to 6 pods rather than 3.

Having described how things should work, let’s verify.

Verify DEV Deployments

Let’s make a simple change to our app so the home page reads “Patient Portal” instead of “Vite + Vue.”

In your IDE or text editor, open the patient-portal/src/App.vue file and change the msg attribute on line 16. Your file should look like this when you’re done:

<script setup>
// This starter template is using Vue 3 <script setup> SFCs
// Check out https://vuejs.org/api/sfc-script-setup.html#script-setup
import HelloWorld from './components/HelloWorld.vue'
</script>

<template>
  <div>
    <a href="https://vitejs.dev" target="_blank">
      <img src="/vite.svg" class="logo" alt="Vite logo" />
    </a>
    <a href="https://vuejs.org/" target="_blank">
      <img src="./assets/vue.svg" class="logo vue" alt="Vue logo" />
    </a>
  </div>
  <HelloWorld msg="Patient Portal" />
</template>

<style scoped>
.logo {
  height: 6em;
  padding: 1.5em;
  will-change: filter;
}
.logo:hover {
  filter: drop-shadow(0 0 2em #646cffaa);
}
.logo.vue:hover {
  filter: drop-shadow(0 0 2em #42b883aa);
}
</style>

Commit and push your changes to GitHub:

git commit -am "Fix message on home screen"
git push

Monitor GitHub actions. There is about a 2-minute interval from the time the finish to GitHub Image Updater’s detection of the changes.

While you wait for things to be completed, open the ArgoCD UI, navigate to the patient-portal-dev app, and watch it. When Image Updater deploys the new version of the dev app, you will see the graph begin moving and changing.

Finally, if all went well, you should see your changes in the app when you navigate to it at http://localhost:8787:

Verify TEST Deployments

Next, let us confirm that GitHub pre-releases trigger a test environment deployment. Navigate to your GitHub repository, click  Releases in the right-hand navigation bar, and click the Draft a new release button.

Click Choose a tag, type v1.0.1, and click Create a new tag: v1.0.1. Type v1.0.1 in the Release title textbox. Select Set as pre-release. Click Publish release.

Now, monitor the GitHub actions that kicked off due to the new tag and new pre-release. Once those are complete, you should again monitor your patient-portal-test application in ArgoCD. Once the “v1.0.1-RC” image is noticed by the Image Updater, the ArgoCD diagram will move and change as the new image is rolled out.

Finally, you should see your updated app deployed to http://localhost:8888:

Verify PROD Deployments

Verifying a deployment to the patient-portal-prod app is the easiest test of all. Navigate to your GitHub repository, again select Releases in the right-hand navigation column, and click the edit (pencil) icon for v1.0.1.

Unselect (remove the check) Set as pre-release. You can optionally select Set as the latest release, but it is not required. Click Update release.

Monitor the ensuing GitHub Action execution, which will deploy an image tagged as v1.0.1 to your Docker Hub registry. Once the action completes, go watch your ArgoCD patient-portal-prod app; you will notice the graph move and change once Image Updater rolls out the new image version.

Finally, you should see the updated app deployed to http://localhost:8989:

Verify IaC Changes are Automatically Applied

The final test is to confirm that any changes made to the IaC stored in the patient-portal-argocd repository are automatically applied.

I suggest scaling up the prod environment to 6 replicas (instead of 3). To do this, open the file overlays/prod/deployment.yaml and add replicas: 6 below line 5:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: patientportal
spec:
  replicas: 6
  template:
    spec:
      containers:
        - name: patientportal
          # Replace the blank with your Docker Hub username
          image: docker.io/____/patient-portal:v1.0.0
          imagePullPolicy: IfNotPresent

Save the changes. Then commit and push your updates:

git add .
git commit -am "Scaling up prod"
git push

Open the patient-portal-argocd app in ArgoCD and watch it. In a few minutes, you should notice it adding pods to the deployment until it reaches a total of 6. To hurry things along, you can manually “Sync” and “Refresh” your application in the ArgoCD UI.

In the end, your deployment should look similar to this:

Verify Load Balancing Works

The final test is very simple; we want to perform a fast series of “hard refreshes” in our browser and confirm the hostname changes.

Navigate to the app in the prod environment. Once the page loads, perform a fast series of “hard refreshes.” In Chrome-based browsers, you do this with CTRL+SHIFT+R. The test passes if you notice the hostname changing periodically.

Conclusion

GitOps is the standard for enterprise operations today. It brings myriad benefits to an organization, including a single source of truth for deployments, versioned control over infrastructure, automation, ease of rolling back changes, consistency, repeatability, and visibility. 

This article has walked you through installing, configuring, and using Kubernetes and ArgoCD to create a GitOps environment. While it used a local Docker installation, it is trivial to create a similar system in a cloud environment like AWS or Azure.

Resources

Source code referenced throughout this article can be found at:

• https://github.com/cbush06/patient-portal-argocd

• https://github.com/cbush06/patient-portal

• Richardson, A. (2018, August 21). What Is GitOps. Weaveworks Blog. https://www.weave.works/blog/what-is-gitops-really

• Weaveworks. (2021, July 13). The History of GitOps. Weaveworks Blog. https://www.weave.works/blog/the-history-of-gitops

Even the most junior of professional software developers will have been subjected to the hair-pulling, teeth-grinding, fist-clenching experience of eliciting requirements from the client.

You know the drill. Your client needs a new feature developed to support some business process, say budgeting new machinery for a construction company. Seems simple enough until they explain all the intricacies and business rules involved. Then they need you to interface with other systems. They also want portions of the navigation to match an old system to cut the learning curve. Back and forth you go, you ask questions and get answers that lead to more questions.

You continue blindly stumbling your way through this forest for several more days, taking reams of notes in lieu of a breadcrumb trail and trying to glean enough information to start a rudimentary backlog of user stories. The whole time, you're confusing all the interwoven business processes, the client is switching from describing one issue to another without warning, and you can barely keep up with the deluge of information, much less begin codifying requirements into comprehensible sentences to take back to your team. To make matters worse, your client is growing frustrated with your questions and giving you looks that say, "Are you even listening to me?" To be fair, you feel like you're just rephrasing the same questions over and over because every answer you get only adds to your confusion.

Ultimately, your requirements session leaves you with more questions, more confusion, more frustration, and no starting point for development. 

So, What Went Wrong?

Requirements development is more than a shallow conversation where the client enumerates their needs; requirements development is a major step in systems analysis and design. Effective requirements development needs four things:

1. People discussing complex ideas must constantly test their understanding of the ideas

2. When studying any nontrivial problem, a visual representation helps you clarify and better understand the problem

3. When designing systems, people need a way to test and experiment on their designs

4. Everyone must speak the same language

A Failure to Communicate

Quoting the 1967 film Cool Hand Luke, "What we have here is a failure to communicate." Specifically, a failure to communicate effectively. When people are discussing complex and interrelated ideas (i.e., systems), the person on the receiving end must be able to test their understanding as the conversation progresses. It follows that you need to be able to test your understanding of the needs your client describes. Similarly, your client must be able to test their understanding of your proposed solutions. Without testing, neither you nor your client can be sure that you fully and correctly understand what the other is communicating.

Picturing the Problem

It is extremely difficult, if not impossible, to understand a reasonably complex problem without visual aid. Kozma (2003) neatly frames the dilemma:

In science, graphics are also used to display data, organize complex information, and promote a shared understanding of scientific phenomena. These graphics are often used to present multiple relationships and processes that are difficult to describe. (Cook, 2006, p. 2)

In short, you must bring visual representations of the client's needs to the table if you are to understand those needs and fully grasp the problem. Not only will a visual help you conceptualize the requirements and see the complex relationships between various parts, it will also serve as a common reference for you and your client. It is an aide you can reference to explain your questions and an aide the client can reference as they give an answer.

Testing Your Ideas

After collaborating around a model of your client's needs to develop a set of requirements and testing your understanding of those requirements, you need to design a solution. All but the most trivial software systems are far too complex for a person to accurately and fully represent in their memory. Just as you need a model to fully understand a client's needs, you need a model to help shape your solution. As you begin designing your solution, you naturally start with the most obvious parts of the problem. It's the less obvious concerns (e.g., edge cases, failure scenarios, relationships, and interfaces) that escape your attention. To identify these, you need a way to test your solution for gaps. You need a model.

Bobek and Tversky (2016) explain the benefits of using a visual model:

There are several notable differences between visual and verbal explanations; visual explanations map thought more directly than words and provide checks for completeness and coherence as well as a platform for inference, notably from structure to process. (p. 1)

Ubiquitous Language

Domain-driven design is a highly effective methodology for designing and building quality software that is not only consistent with the client's domain but also correctly solves that problem. One of its fundamental practices is using a Ubiquitous Language throughout the entire software development lifecycle. This means the developers, clients, and other stakeholders all use a common vocabulary. This lexicon should be applied to code, documents, e-mails, meetings, and other activities.

Using a visual model in both requirements development and solution design both encourages and smoothes the process of building that Ubiquitous Language. Elements of visualizations naturally constitute parts of the Domain Model and give all stakeholders a single frame of reference. Using those visualizations, naming things becomes a team activity.

Finally, visual models make the requirements development and design processes inclusive even before the Ubiquitous Language has been developed. Without a visualization, the domain experts and technical experts do a great deal of talking past each other. The domain experts use their industry lexicon that developers don't understand and vice versa. Visual models allow everyone to participate despite these differences in experience and knowledge.

Tools of the Trade

Requirements development tools can be roughly divided into two categories:

1. Those that document current business processes and/or the status quo. Examples include:

   1. Business Process Models

   2. UML (Unified Modeling Language) Activity and Statechart Diagrams

   3. Event storming

   4. Data Flow Diagrams

2. Those who design and model the proposed solution. Examples include:

   1. UML Activity, Statechart, Sequence, Collaboration, Use Case, and Class Diagrams

   2. Wireframes

   3. Mock-ups

   4. Event storming

   5. Data Flow Diagrams

Documenting Business Processes

If you are building a system that automates a complex business process, your first step in requirements development should be documenting that process. As you can see above, there are several tools you can use to perform this process. Earlier in the article, I covered some criteria you may want to consider when choosing:

• Inclusive: Can your clients easily join in the process?

• Intuitive: Is it simple and quick to master? Does it express thoughts well?

• Collaborative: Can everyone join in?

I believe the UML Activity diagram best captures business processes to be implemented in a software system. However, I believe these criteria rule out UML in one way or another. Most obviously, UML comes with a learning curve that the non-developer will have to tackle and, therefore, will be excluded from the exercise.

Fortunately, there is a nearly equivalent alternative that anyone can rapidly learn: Event Storming. Event Storming can be used to document, design, and explore business processes. It also lends itself nicely to capturing the various interactions between components within a system and between that system and external systems.

Event Storming is extremely simple. It lacks confusing symbols, has minimal rules, and requires only a few pads of different-colored sticky notes. The facilitator of an Event Storming session can explain the activity in very little time, one to two minutes at most. 

Finally, even though Event Storming was created for large walls in conference rooms, there are now many online tools that allow teams to collaborate in real time on a diagram.

For more details on Event Storming and plenty of material from its creator, Alberto Brandolini, visit eventstorming.com.

Documenting the Interface and Flow

User-centric applications will have an interface that is tightly coupled to its data flow. This makes sense because the user is either the primary source of the data, the primary destination of the data, or the primary trigger of manipulations of that data.

Developing requirements and designs for such an application will have two goals:

1. Document the flow of data and business processes

2. Design an interface that injects the user into that flow and those processes

Depending on the size and complexity of the data flow and business processes, it may be worth taking the time to capture them explicitly. If this is the case, I suggest Event Storming for all the reasons covered in the section above.

On the other hand, I have worked on systems (e.g., a simple class scheduling system for training events, a data repository for equipment manuals, etc.) where the business flow and business processes are relatively simple. In these cases, the processes are so tightly coupled with the interface that you can capture all the relevant details while designing the user interface. 

The dogmatic engineer might disparage this shortcut, but hear me out. Most of my clients are not engineers or even technologists. In the example above of the class scheduling system, they are Soldiers in jobs that have nothing to do with systems engineering or software development. Working through more traditional requirements development events with them would garner little interest and involvement; it would likely lead to frustration and resentment.

Instead, I used Adobe XD to prototype mock-ups for all the obvious flows through the system. I then held a one-hour session where I walked a small group of stakeholders through each flow while asking probing questions and soliciting feedback. The visuals of the prototype captured their imagination and attention far better than other forms of requirements gathering. The group could identify various edge cases, scenarios, and flows I had missed. They pointed out pain points where the interface could better support their jobs and imagined ways elements they saw in the wireframes could be extended to add even more value. 

In short, if you are starting a project with a user interface tightly coupled to the data flow and processes it enables, consider using visual tools like wireframes and prototypes to capture the users’ requirements and start designing solutions. With a visual model, you can test your understanding and design for "completeness and coherence."

Agile Requirements

In closing, one of the tenets of Agile methodologies is performing the Software Development Lifecycle phases in parallel. I propose you extend this philosophy to requirements development and solution design by doing both tasks simultaneously. Using an intuitive visual model as the medium for this task allows you to test both your understanding of the client's needs and your plans for a solution. Furthermore, it makes the process accessible to stakeholders who are unfamiliar with more technical tools like UML diagrams.

The Contract that Died

I once worked on a government contract that died. It did not die for the usual reasons like its services no longer being required, a lack of quality in work, or the final product having been delivered. It died because the leaders lacked courage and refused to trust their team. They refused to trust the team because they did not understand what the team did; and, though they could have trusted us, through a leap of faith in spite of their incompetence, they lacked the courage to do so. 

Over the course of several years, the government imposed a few constraints here, made some unwise choices there, and eventually strangled our best efforts to deliver the services they had paid for. As the lead developer on the team, I protested loudly, frequently, and vehemently for a long time. Sadly, my protests fell on unknowing and untrusting ears. They fell on ears that neither understood the implications of the government's decisions nor trusted me enough to fight back regardless of their handicapped knowledge.

At the end of the contract, my job felt more like playing “Mother May I” with Bill Lumbergh (of Office Space) than developing software. I had to ask permission for everything, was watched like a hawk, was completely ignored until something broke, and was always the last to know anything of importance. If you thought to yourself, "That sounds like a run-of-the-mill micromanager to me," you would be right. However, I believe micromanagement is only a symptom of underlying problems that come from a lack of competence, a lack of trust, or both.

Put another way, I believe there are three flavors of micromanagers:

• Competent but untrusting

• Incompetent but trusting

• Incompetent and untrusting

Before I continue, know that being a micromanager does not always mean that person is "bad." Sometimes, they are "bad" people in the self-centered, ego-centric sort of way and only care about themselves. However, I believe these are the exceptions and, in most cases, micromanagers are good people who care about getting the job done right.

The Contract's Death Rattle

The death of that government contract was no surprise to the development team. Its raspy, half-choking death rattle began long before a merciful demise. Life support was administered through temporary, one-year extensions rather than the normal 3-5 year renewal; and, CPR came in the form of offloading portions of our members' time to other contracts as funds began drying up. No, there was absolutely no surprise when the end came, despite management's best efforts to keep the wraps on their failed resucitations. As a morbid sort of post-mortem, I want to walk you through the metastization of this bad management cancer.

When I first joined the contract, it possessed an environment in which anyone would enjoy working. There was freedom; creativity and experimentation were encouraged; and, there was just enough process to keep chaos at bay. However, the good times were short-lived. Corporate started tasking our project manager with duties outside our contract. These additional responsibilities and a few other factors led our PM to reframe their role to that of a program manager and abdicate their day-to-day responsibilities to several new hires who were to learn project management on-the-job.

These junior PMs came from diverse backgrounds including business process analysis, event planning, and technical writing. There was no shortage of intelligence and can-doism but there was a complete lack of experience in the field to which they had been assigned -- software development.

This incompetence manifested in the third form of micromanagement above: incompetant and untrusting. The project managers did not understand the work they were managing and, whether out of an abundance of concern for saving face or a basic lack of faith, did not trust the team to get the job done.

The day-to-day symptoms of this management style included:

• A convoluted, incomprehensible business process whose creator could not even brief correctly

• Demands for increasingly verbose and daily comments on every in-progress ticket

• Information hording

• Exponential growth of SOPs (standard operating procedures) and documented business processes

• Management through faceless and impersonal Jira tickets rather than face-to-face leadership

• Inordinate focus on irrelevant details of how things were done rather than the the end product

• Seeking to demonstrate personal value through:

• Finding mistakes in other's work

• Finding roadblocks rather than removing them

• Excessive quantities of useless documentation

• Increasing isolation between management and developers

Every one of these items is a result of either

• incompetence;

• distrust; or,

• fear.

If any of my story resonates with your experiences, stick with me while I dig into the reasons project managers fall victim these shortcomings and how these issues impact projects and developers.

The contract finally ended when the government financed another project that subsumed the responsibilities of the applications we maintained. Personally, I blame this outcome on: (1) a lack of our responsiveness to the customer's needs; (2) a failure of our management to actively advocate for our ability to meet the customer's needs; and, (3) a failure of both our management and their government counterparts to solicit more work for our contract.

I believe the first and second problems could have been avoided by courageous and trusting leaders.

Leaders Without Trust Are Not Leaders

Most thought-work (e.g., software development, mathematicians, physicists, accountants, lawyers, etc.) operates according to the principles of research and development rather than manufacturing or assembly lines. Thought-work jobs start with a problem to be solved rather than a blueprint to be followed. These jobs require investigation, stating of hypotheses, experimentation, testing, and implementation. All of these stages are subject to failure and refinement. They require independent thinkers who are free to operate in an environment where the concerns of business are abstracted away so the thought-worker can focus on thinking (Spolsky, 2006). Micromanagers break down these abstractions because they constantly inject themselves into the thought-workers' world demanding the worker perform tasks unrelated to solving the problem at hand, to either edify the manager or enable their continued incompetence and distrust.

In short, micromanagers destroy their thought-workers by distrustingthem and, as a result, erasing the development abstraction. With the development abstraction gone, workers are led to believe that their priorities and passions are misplaced, that metrics about what they do are more important than what they produce, that mountains of unused documentation appear more productive than a functioning product, that value is measured by adherence to dogmatic processes rather than customer satisfaction, and that they are so untrusted their every activity must be reported. They are told things like "We need all of your tasks documented in SOPs," "You must record everything you do in comments on your tickets," or "You must submit a daily e-mail recording all of your activities."

In the end, the workers realize management cares more about framing business in a form they are comfortable with and less about actually delivering value. The concerns of the managers are forced into the world of their developers and the developers' progress grinds to a halt.

Trust Without Courage is Useless

For a moment, let us assume the project managers discussed above do trust their team but still lack courage. The result is passively impotent leaders. In this case, they are not actively obstructing the team as in the previous section but they are still failing to advocate for the team. The outcome is the same -- the loss of the development abstration.

Good leaders who trust their teams must be courageous enough to follow through. This means they do things like:

• Advocate for their team to higher management

• Shield the team from politics

• Fight for resources

• Negotiate requirements -- both internally (e.g., product owners and marketing) and externally (e.g., customers or partner organizations)

• Defend the schedule

• Support the team's assessments and recommendations

• Eliminate barriers and impediments

My time as a Federal civil servant showed me exactly what it is like to work for people who trust you but lack the courage to follow through. As a developer, I received a laptop with barely enough resources to run Microsoft Outlook. It was impossible to run Eclipse and other developer tools. I requested developer machines for the team multiple times but my leader did not want to fight for the money to purchase the tools we needed. 

During this same period of my career, I was working on building a system used to solicit contract offers for services. If you are familiar with the Department of Defense (DoD), you know that, with very few exceptions, all of their IT systems must use Smart Card (a.k.a., Common Access Card (CAC)) authentication. This is a physical token that holds the owner's private keys and Personal Identity Verification (PIV) X.509 certificates. The issue for my project was the people and companies bidding on contracts would not receive these CAC cards until winning the contract. In other words, our target users would not be able to access the system. Obviously, username/password authentication was the best alternative but our cybersecurity manager adamantly stated this was not authorized. I did my homework and tracked down all the relavent DoD regulations, orders, and publications proving we could use username/password authentication as long as we requested and received a waiver. The cybersecurity manager refused to even look at my research and, rather than fighing on my behalf, my manager chose to bend the knee.

Following these events, my motivation and pride in my product took a serious dip. If my manager did not care enough to fight for me, then why should I care? If my manager did not care enough about building a usable product, was my job even important?

As with the stories above, project managers often suffer from a lack of courage to stand up for their team. Many others suffer from the fear of looking weak to their teams. This was a condition I encountered far more often in my military career than in my civilian one.

You often find these PMs attempting to hide behind their own subordinates by encouraging them to do the courageous things (e.g., report ethics violations, make risky investments, think outside the box) that they do not do themselves. Detert (2022) makes the case, “Yet that’s what leaders who ‘encourage courage’ are essentially doing. They’re implicitly saying that because the aren’t courageous enough to change the conditions in their organization to make it safer for people to be honest, try new things, or take other prudent risks, everyone else should be courageous enough to do them anyway.”

Other times, these PMs will do everything they can to avoid admitting they do not know something. I have witnessed PMs fabricate lies, incoherently babble until they lose their audience, and drop a ten minute string of buzzwords, all to avoid admitting they don't know something. This condition relates to the next topic, incompetence, in that it is often incompetent managers who refuse to admit their lack of knowledge for fear of exposing that incompetence. Detert (2022) simultaneously condemns the willfully incompetent and encourages the knowledgeable who are afraid of losing face: “Once people know you’re competent, it makes you look stronger (not weak) when you admit ‘I don’t know’ or say ‘Please help with this.’”

Incompetence is no Excuse

I have found two common forms of incompetence plaguing project managers:

1. A lack of knowledge -- they do not understand what their team does

2. A lack of expertise -- they do not understand their role as project manager

1. Lack of Knowledge -- PMs Who Don't Know Their Team's Jobs

It takes a tremendous amount of courage to place your faith and trust in other people. Leaders are evaluated by the performance of their team. This means that they must trust their teams with their own careers. 

A personal lack of expertise in the field a leader's team operates in only compounds this issue. In that case, the leader must trust their team to make a project successful while they do not fully understand that the team should be doing to realize that success.

Project managers should minimally understand the level of effort and complexity of their team members' various tasks. Llopis (2013) writes, "Though leaders cannot be expected to have all of the answers -- they should not play at arms length either. The 21st century leader must be more high-touch in order to effectively evaluate the business and coach-up their employees."

I believe leaders must understand what they are asking of their team members when they assign different tasks. Likewise, they must avoid over-delegating as a result of their own shortcomings. If you are a project manager, you should have enough competence that you can make educated inferences about the project schedule and its dependencies based on the requirements and designs your team is working from. In The Contract that Died (above), my project managers did little more than demand the team provide arbitrary dates and then plug those dates into Microsoft Project. Some people may find this acceptable, however, I do not. A competent project manager should know enough to ask probing questions, realize high risk areas of the project, and recognize overly pessimistic or overly optimistic estimates given by the team.

So far, I have only discussed the project manager's role in building a schedule and estimating levels of effort. What about the rest of the Software Development Lifecycle (SLDC)? Can the PM build a quality schedule with no understanding of testing tools, strategies, methodologies, and types? Would the PM realize the team overlooked load testing, failed to test all of the possible user platforms/browsers, or completely forgot to perform functional testing on external system intefaces/APIs? Some readers may believe I'm expecting too much from a project manager. I believe that the PM should not understand the implementations of interfaces or the methods of load testing, but if the PM is to lead a team of developers they need more than a high-level, academic understanding of the SDLC. They should have enough experience behind them to have developed an intuition about the activities involved in successful software projects.

A PM is certainly not a developer. Furthermore, I have found, with few exceptions, developers make poor project managers. The fact remains that PMs should understand how developers operate within the SDLC, how their decisions impact developers, and what their role is inside the SDLC.

2. Lack of Expertise -- PMs Who Don't Know Their Jobs

Project managers who do not know their roles will do everything they can to appear valuable. This frequently manifests as them inventing jobs for themselves, injecting themselves where they don't belong, placing an inordinate emphasis on nonissues, or exaggerating problems while offering no solutions. I believe this problem occurs more frequently with project managers who were highly technial workers in the past (e.g., former developers, testers, engineers) and tried (unsuccessfully) to transition to management.

Some examples I have witnessed include project managers who:

• take on the role of quality assurance (the job of testers);

• make it their life goal to act as timecard police; and,

• act as project schedule enforcers who want to beat deadlines (consequences be damned).

These project managers will either track every movement of their team members as a salve for anxiety stemming their sense of being lost or they will find other roles they are comfortable with and try to supervise those. Regardless of how their ineptitude shows itself, the end result is always a project manager who obstructs rather than facilitates their team's success.

Good project managers, those that are true leaders, work to enable their team, remove impediments, shield the team from politics, provide resources, and inspire and motivate. They work to both defend their team and advocate on their behalf. They hold their team responsible while also being its greatest cheerleader. Their is a difference between managers and leaders. Good project managers are also leaders.

Conclusion

Project managers do not have to be technical experts. They do not have to possess superhuman levels of courage and demonstrate praise-worthy heroics on a daily basis. They do not even have to be fully proficient at their own jobs. 

However, project managers must be courageous enough to take calculated risks, to go to bat for the teams, to admit when they lack knowledge, and to lead from the front. 

Project managers must also trust their teams. They must trust them eough to act based on team's input, to stay out of the way while the team works, and to vouch for the team to their external stakeholders.

It is these courageous and trusting people whom we more frequently call leaders.