Prompt engineering has emerged as a critical skill in this AI-dominated landscape. It involves fine-tuning interactions with GAIS to achieve accurate, precise and efficient results. A prime example is the Midjourney Showcase page, where users continuously refine their prompts to transform the mundane to a masterpiece. This disparity in outcomes between a novice and an expert is reminiscent of the difference in creations using tools like Photoshop and Illustrator, where the same tools can yield vastly different results based on the user’s expertise.

The burgeoning field of prompt engineering is as fascinating as it is perplexing. In the tech world, there’s a tendency to prioritize machines over human interaction. The “computer nerds” – and I include myself in this group – have enthusiastically embraced learning to work with GAIS. But what if the key skills for successful prompt engineering are the same ones that enable us to excel in teamwork and relationships?

Effectively engaging with both AI systems like ChatGPT and DALL-E, and in our personal and professional relationships, requires thoughtful interaction. This isn’t a far-fetched idea, considering GAIS are trained on human creativity. Hence, connecting with a GAIS can be quite akin to connecting with another human being.

Just as a prompt engineer meticulously considers word choice and sentence structure to elicit the desired response from an AI, similar attention to detail is crucial in human interactions. Whether you’re a friend, leader, coach, or just a likable team member, your success often hinges on your ability to connect with others. Dale Carnegie’s classic “How To Win Friends And Influence People” could very well be rebranded as “Prompt Engineering Humans for Dummies.”

This message is particularly pertinent for tech professionals who often view human elements as secondary. If you find excitement in crafting a new prompt for Bard, consider channeling similar enthusiasm into forming new human connections. Beyond coding and technical skills, fostering strong relationships is vital. The typical software engineer might be adept at prompt engineering for AI, staying abreast of the latest in database technology and cloud APIs, yet they often invest minimal effort in connecting with their peers and managers. Borrowing a phrase from sports, “it’s about the Jim’s and Joe’s, not just the 1’s and 0’s.”

Prompt engineering with humans can manifest in various ways. How we communicate with others significantly impacts the responses we get. Phrasing questions thoughtfully can lead to better and more straightforward answers. Whether it’s motivating a team during crunch time, delivering a halftime speech, or simply interacting with family members, the effectiveness of our communication is paramount. Children, for instance, know exactly how to “prompt” their parents to give in to their demands The way we approach conversations can either build bridges or create barriers. Getting poor results from ChatGPT is a signal that the prompt needs refined or discard altogether. It’s not rocket science; the user is accustomed to finding a new prompt to forge ahead. Your colleague, family member, neighbor and angry customer are all similar in that with the right prompt a connection can be created.

I urge you to recognize the parallels between prompt engineering and effective human communication. Acknowledging this connection invites us to reconsider our approach to conversations, both with AI and with each other. Just as a well-crafted prompt can yield remarkable results from an AI, the right words and approach can enhance our human connections, foster understanding, and lead to more meaningful interactions.

The analogy between AI interaction and human relationships transcends mere technical curiosity; it has real-life implications. As technology increasingly mediates our world, the skills of prompt engineering metaphorically translate to the art of conversation and connection. This perspective encourages us to reflect on our communication styles, how we listen, and how we respond, both in digital and real-life interactions. Whether you’re a tech enthusiast, a professional in the field, or simply navigating modern communication complexities, this insight serves as a powerful reminder of the impact of our words and the potential of our interactions. Let’s apply the lessons from the realm of generative AI to our most valuable asset: our relationships with others.

To put it simply, a Manager README is a document that outlines your management style, expectations, and preferences. It’s a user manual for you as a manager, and it is something that you can choose to share with your team members.

Unsurprisingly, the internet was polarized on the value of a Manager README. I read from those that felt it was contentious and narcissistic. Others felt like it was a useful tool when leveraged in the right way. My thought process was that if nothing else, if I only kept this document private, I should create it to explore my own thoughts to really discover who I wanted to be as a manager. In the end, I was very proud of what I created. I have gone on to share this with others in the management team, my direct reports and the public internet. Sharing this document is one of the better career decisions I have ever made.

First, creating my Manager README allowed me to rapidly build my identity as a manager. From the beginning, I learned who I want to be and how I would try to conduct myself.

Second, it has greased the wheels and got me moved along in job interviews. The hiring team has the best picture of what it’s going to be like to add me to their management team. Interviewers have shared only positive feedback on this document and voiced their appreciation for providing it. In one case, I was rejected for a lead role but then called back a few months later for an additional opening purely because they thought my Manager README distinguished me from other candidates.

Third, I continue to leverage the content I created in my Manager README to help me communicate ideas and create policy for my current teams. I’ve lost track of the times I’ve referred back to this document to help me fast track a current initiative or make a quick decision because I had already put in the thought-cycles to know how to act.

Should you create a Manager README? Or even an Engineer README? I sure think you should, if that’s not already obvious :) You’ll never regret putting a few hours into the project. It’s a great document to have even if your decision is to keep it to yourself. Do you have an existing README? My contact information is below, share your README with me.

I like to say “your data tells a story.” Every team and organization has data buried in the tools they use that shows how the team operates. I use this data to drive decisions to reward people, discover problems, tweak process and operate more efficiently as a team. I have managed or co-managed 200+ software engineers and dozens of teams practicing scrum or kanban. I am routinely amazed at insights we can attain simply by collecting, organizing and analyzing the data from our tools.

The systems, tools and apps we regularly use are filled with information about how we work, work effiency, when we work and general interactions within the team. Data alone is of little value similar to an unorganized bucket of magnetic plastic letters. We must find ways to organize, process and extract insights that drive action and the decision making process. The most powerful metrics are those that motivate us to take action.

Building metrics from our data gives us the power to make data-driven decisions in our teams and organizations. Using metrics, we can rely less on anecdotal experiences with people and teams which are often riddled with bias and difficult or impossible to confirm (or deny!). When we have a team member that we suspect is struggling then we should leverage the data to further our understanding of how this person performs within both absolute and relative metrics.

I’ll be completely honest: sharing some of this information makes me a tad uncomfortable because this information can be abused by bad managers. I mean this sincerely: use these metrics carefully to reward team members that are doing great work, analyze situations with underperforming employees, reward the entire team for jobs well done, project delivery dates based on project scope or build forecasts for staffing a project based on historical workloads.

Goodhart’s Law [1] states “When a measure becomes a target, it ceases to be a good measure”. The team will alter their behavior as they become more aware of how their performance is measured and incentivized. Observe the organic behavior of the team for a period of time while using this information to make specific and targeted decisions. Predicting the associated changes in behavior from incentives is difficult so another reminder to use good judgement.

Do not hide the metrics from the team nor lie about your intentions or what you have learned. The team will pickup on dishonesty and impure motives which ruins trust. Publicly highlight and reward the great work that team members are performing. Quietly and privately work with team members that are struggling to find success.

Back to Roger. We recognized there was a clear problem and initiated more direct oversight and coaching. Through more candid conversations, we learned that Roger didn’t actually like the culture of our organization nor our software engineering practices. It was a bad fit for us and him. Roger left after a few months. As a result, I learned that I should talk more directly about our teams and culture during the recruiting process to be as honest and as clear as possible to candidates.

Here’s another story. Over the course of 18 months we scaled our software engineering head count to over 100 people across North America. In hindsight, I think our oversight of people was too lax. I discovered multiple people that were doing nothing. Software engineers that were simply choosing to not do any work. I measured their contributions and it scored at zero. Dozens of months of people-time with nothing to show for it.

On a positive front, we used data to confirm our anecdotal accounts with the top performing software engineers on the team. In those cases, the data helped us make legitimate cases for promotions and compensation. In other situations, the data drove decision making to make informed choices on personnel as we reduced (contracting) teams near the end of the project.

To leave the reader with actionable information for KPI’s for Software Engineering teams I have this for you. First, ensure the teams are making changes in the codebase through pull requests. Second, review the pull requests by person per week. Pull requests are nearly an industry standard and teams should be familiar. Also, pull requests are easy to work with and track. Pull Requests by person per week is the best KPI I can recommend that is near universal across teams and organizations so start here. To grow your metrics it will depend on the organization so I recommend to start simple and grow them intentionally. Ask yourself questions to see if you can formulate an answer with the data.

Remember, your data tells a story. Use the data to gain a tactical advantage for your team and organization to keep winning.

[1] https://en.wikipedia.org/wiki/Goodhart%27s_law

Acquiring a job offer, regardless of where you are in your career, is a testament to your ability to meet the employer’s needs. Employers seek candidates who can seamlessly integrate into their teams and immediately contribute. But how can you make yourself stand out in a competitive field where everyone is vying for the same positions? The answer lies in showcasing your skills and proving that you’re not just talk but action, particularly in the realm of software development.

The journey to becoming a successful software engineer demands consistent effort and dedication. There’s no shortcut or a set 40-hour plan to land your dream job. It requires daily commitment – allocating at least an hour every day to actively code. Reading and watching content might load your brain with ideas, but the real magic happens when you translate those ideas into tangible code through your fingertips. Writing code, practicing, and continuously building your portfolio are the keys to making significant progress in this field.

Hiring managers highly value tangible proof of your capabilities – a portfolio filled with projects that exhibit your coding skills. Verbal stories about your knowledge simply won’t cut it in a field where action speaks louder than words. It’s not just about talking about your skills; it’s about proving them through real, demonstrable projects. A solid portfolio showcases your commitment and dedication, setting you apart from other candidates and demonstrating your ability to “walk-the-walk” as well as “talk-the-talk.”

In the competitive landscape of the software engineering industry, your dedication, ability to consistently produce code, and a portfolio of demonstrable projects are what will set you apart and land you that dream job. It’s time to step up, showcase your skills, and prove your worth beyond words. Your future as a software engineer starts with the projects you build today.

As a software engineering manager, I am often tasked with ensuring the team is doing things and we the team are achieving specific outcomes. Often I need to make sure specific things are happening on a regular basis. Occasionally I need to understand why the team is NOT doing something we otherwise think they should or thought they were.

For any change or behavior I need the team to act on, I lean on my golden rule to maximize our chance of success: “make the right thing (to do) the easy thing (to do) because when you make the right thing the hard thing you get the wrong thing”. Human nature is to minimize effort so every ounce of added friction for “the thing the team is to do” means the team is more likely to struggle executing.

Life is full of examples where the default behavior occurs because it’s been made so darn easy how could it happen any other way? The grocery store displays bakery items and fried food at the entrance and candy bars at the checkout lane. Social media websites siphon user data because of lax security settings. Disney World setup gift stores as the attraction exit so the parents will buy more gifts for their children. Why would you want to make it any harder than it needs to be for your team members to find success?

Keep meetings efficient by setting an agenda ahead of time. Eat less candy by removing it from your house. Complete your workouts by setting a workout plan ahead of time and pick a gym that is easy to access. Ensure the unit tests are run by including them with automation as part of the check-in process. Ensure the test environment is stable by adding automated verification tests. Agile software development practices encourage the team to break down work in to smaller pieces so they can be delivered more quickly. Reducing scope means reducing complexity which yields an increased chance of getting it right on the first attempt.

Software engineers are a predictable group of people - they easily get engrossed in their work which is similar to acute tunnel vision. They are “zoomed in on the microscope” and very easily miss things in the bigger picture. Software engineers are frequently shifting between solutioning, coding, testing, fixing, refactoring, beautifying, simplifying and optimizing. They miss things (in the moment) that seem horrifyingly obvious once sufficiently detached from the work. We have all heard “I could have sworn that button was working when I merged the code…” Drawing back to the tunnel vision analogy - the primary thing that made sense at the time was their work immediately in front of them. Use your imagination to replace “software engineer” with your field of choice and the specific ways they operate - it’s all the same at the end.

I’ve lost track of the number of times a senior team member has brought up a change for the team that I felt was overly confusing or ambiguous or perhaps even both. My role as a manager is sometimes a therapist - to absorb the negative emotions, diminish their power and then get the person to work towards a solution. If I can get them to get me to clearly understand the problem and articulate the solution then we have solid evidence that we can get the team moving on board too.

My approach of “making things easy” is completely unremarkable and probably makes a lot of sense on face value. However, I’m frequently shocked at the high levels of compliance that teammates are expected to showcase simply because they were ordered to do so. That’s just not how people work. As the leader, our objective is to ensure that the established processes for the team are simple, concise, straight forward and make sense. So simple, a caveman could do it if necessary. Your team will SHOW YOU through their actions when they are picking up what you have been putting down. The actions of the team will show you what they understand.

It is the leaders responsibility to ensure the team is given the proper runway and support to maximize their potential It is the leaders responsibility to ensure that problems are properly remediated once identified. One option for problem resolution that is quite common is to get mad, lead through negative emotions, announce the team members are stupid and lazy, make passive aggressive comments about the lack of commitment or professionalism and finally alienate everyone in your vicinity. Reaching for the phrase “what the heck is wrong with my team??” is the perfect litmus test for knowing that the team has been tasked with ambiguous orders that are too difficult to execute in a reliable and timely manner. Jocko Willink taught me “there are no bad teams, only bad leaders” to which I fully believe because I’ve seen it play out time and time again. Seeing is believing. The leader is the only person that can be identified as the root cause of a problem!

Another option, the preferred option for a strong leader, is to review the underlying root causes for the problem and then enact change where we make the desired outcome so simple to accomplish it’s impossible to NOT do it. Ensure the team knows there are real reasons for why they need to follow protocol. Chat with the team to see if they can identify why they are struggling to follow through with the requests. Keep in mind, they might not even know themselves. Listen to them if they offer meaningful improvements but if not, the leader must be maximally aware of what can be done to ensure the team members can follow through for what is being asked of them.

Remember this short phrase: make the right thing the easy thing. No one will ever say “thank you” for making it easy but they will do what you need them to when you need them to it.

Enforcing TDD on a team or organization isn’t my style. After many years of championing unit testing initiatives I’ve just come to accept that unit tests are an uphill battle for your average development team. At least for the teams for which I’ve been responsible.

Here’s a reason to practice TDD that you probably won’t read anywhere else: writing code and tests at the same time is the only way to guarantee the developer is producing testable code. Writing testable code means the code can be called from multiple “entry points”. The first entry point is the actual application be it a web application, system service or something else. The second entry point is the test runner. The only way (and the easy way!) to produce testable code is to write code and tests at the same time. Doing so allows the developer to easily see when the code can be run only from the application entry point. We are only human. I’m sure you’ve been on an application that goes through a significant initialization process to setup all of the implicit dependencies that are needed just to run the darn API call that is the focus of our attention.

I’m really not a complainer. I’ve just accepted my reality and learned how to best move forward on a tricky and nuanced topic. Below is my playbook to add unit testing to any software team on the planet. The leader can hold the team completely accountable to ensure tests are added once the below steps are complete.

1. Give your team the ability to write unit tests and make it as easy as possible. Ensure that there are test projects in source control and the developers know how to run the tests. Add this to your onboarding process. Show the developer how to load the tests, add new tests and run them. Show the developer how to see the test results from the automated builds/pipelines.
2. All unit tests must be run in the CI/check-in process. Running tests through the automated check-in is the only way to guarantee the tests continue to run in the future.
3. Developers must have the ability to see code coverage on their developer laptop. Otherwise, they are flying blind to know if they are writing too many or too few tests.
4. Hold the team accountable through code coverage minimums.

Remember, writing tests and code at the same time is the only way to guarantee that the team is writing testable code. Even if you do all of these things it’s still not a guarantee the team will even bother to write tests. You can make it available. You can make it easy. Now the team must be held accountable. I’ve seen exactly one team in my career establish long-term and continued success adding tests: we made our CI process enforce a code coverage rule that code coverage cannot decrease. Whether we were at 5% coverage or 85% coverage we blocked completing pull requests if the changes would reduce code coverage.

None of this information is a magic bullet. But it’s the only way I know how to do it and have confidence that we will get results.

Good luck.

I faced this very scenario at a recent conference when I found myself cardless, missing out on the chance to connect with potential collaborators and clients. Frustrated with this missed opportunity, I decided to take matters into my own hands.

I’ve created this user-friendly web application to provide you with a quick and hassle-free solution for generating digital business cards. It’s a versatile tool that can be your go-to resource for creating and sharing your professional contact information with ease.

Why Choose FreeBusinessCardGenerator.com?

1. Instant Creation: With just a few clicks, you can design and generate your personalized digital business card. No need for design skills or complicated software.

2. No More Missed Opportunities: Whether you’re at a conference, a networking event, or just a casual meetup, you can instantly share your card by saving it as an image on your device or displaying it on your phone for others to snap a picture.

3. Customizable: Tailor your card to match your personal branding and style. Add your name, title, contact details, and even a QR code for a quick scan.

4. Always Accessible: Your digital business card is always within reach, ensuring you never miss an opportunity to connect with someone.

5. Eco-Friendly: Say goodbye to the need for physical business cards. Reduce paper waste and contribute to a greener planet.

I invite you to give FreeBusinessCardGenerator.com a try and see how it can transform your networking experiences. It’s a tool I personally created to solve a problem I faced, and now I’m excited to share it with you.

Stop missing out on valuable connections due to the lack of a business card. Visit FreeBusinessCardGenerator.com and create your digital business card today. It’s quick, it’s easy, and it’s the modern way to share your professional identity.

Let’s network smarter, not harder!

FreeBusinessCardGenerator.com

Preparation is Key

The first step in preparing for an interview is, unsurprisingly, preparation itself. But how much is too much, and what’s the minimum? Here’s what you need to know:

• Keep a Log of Interview Questions: Whenever you’re interviewed, jot down the questions you’re asked. After the interview, re-answer those questions in writing. This practice helps you refine your responses over time, as interviews often revolve around storytelling.

• Watch Mock Interviews: YouTube is a treasure trove of mock interview videos. These can give you a sense of what to expect and help you prepare effectively.

• Mindset Matters: 30-60 minutes before your interview, get your mind in the zone. Eliminate disruptions, review your resume, notes, and common interview questions, and ensure you’re ready for the interview.

• Punctuality is Non-negotiable: Don’t be the reason an interview starts late. Be on time or even a bit early if possible.

Dress Code and Appearance

What you wear to an interview is important, and it’s better to err on the side of professionalism:

• Dress Professionally: It never hurts to look professional. Aim to match your attire with what you think the interviewers will be wearing, and then take it up a notch.

• You Can’t Overdress: It’s challenging to be overdressed for an interview, but you should definitely avoid being the most underdressed person in the room or on a video call.

Come Prepared with Questions

Never underestimate the importance of asking questions during an interview. It’s not only appropriate; it’s essential:

• Ask Questions: Prepare thoughtful questions about the company, team, and role you’re interviewing for. This demonstrates your interest and engagement.

The Interview Itself

The interview can be a nerve-wracking experience, but here’s how you can navigate it smoothly:

• Small Talk: Keep the initial small talk light and let the interviewers lead the conversation.

• Quick Evaluation: It often takes just 10-15 minutes for interviewers to gauge if they want to hire someone.

• Body Language: Body language and eye contact are crucial during an interview. Maintain good eye contact and display confident body language.

• Cultural Fit: To determine cultural fit, interviewers look for candidates who are self-aware, understand their “Why,” and genuinely want to join the team or company.

• Red Flags: Avoid red flags such as blaming others, denigrating previous employers, providing surface-level answers, or making the interviewer work too hard to extract information.

Explaining Career Transitions

For those with limited development experience or transitioning careers, focus on these key points during the interview:

• Attributes for Junior Developers: Be motivated, someone your interviewer wants to work with, and someone they want to help grow.

Bootcamp Graduates vs. Computer Science Graduates

When comparing bootcamp graduates to computer science grads, remember this:

• You’re Given a Chance: If you’re in the interview, the employer is willing to give you a chance. They know your knowledge might be incomplete compared to traditional computer science graduates.

• Emphasize Motivation: Highlight your motivation and enthusiasm for the field during the interview.

How to Be Memorable

Being memorable in a sea of candidates doesn’t mean panicking. Instead:

• Stay Composed: Keep your cool, maintain a professional demeanor, and be yourself.

Handling the Salary Question

When it comes to discussing salary:

• Ask About the Range: Inquire if there’s an established salary range for the position. If they like you, they might be willing to push the upper boundary by 5-15%.

• Research the Market: Research industry standards to ensure your salary expectations are reasonable.

Remote Interviews

With the rise of remote work, remote interviews have become common. Here’s how to prepare and what to expect:

• Prepare for Remote Interviews: Find a quiet, distraction-free location, test your setup, and ensure you have the necessary equipment.

• Expect Functional Setup: Interviewers expect a functional audio and video setup. They may reschedule if there are technical problems.

• Tools May Vary: Different companies use different interview tools, so familiarize yourself with these tools to focus on the questions rather than the technology.

The Technical Interview

Technical interviews can vary widely from one company to another:

• Expect Variability: Every company has its unique interview process, and every team believes they have the perfect approach.

• Behavior in Interviews: It’s perfectly fine to ask about the next steps, timelines, and other logistics during a technical interview.

Follow Up

After the interview, it’s crucial to follow up:

• Ask About Next Steps: At the end of the interview, inquire about the next steps and their associated timelines.

• Practice Patience: Recruiters have other commitments, so be patient when waiting for follow-up.

Parting Advice

As you embark on your job search, keep these words of wisdom in mind:

• Persist and Learn: Keep pushing forward, learn from your mistakes, and don’t let failures deter you.

• Take Notes: During the interview, make notes of who was present, their roles, and any notable information shared. This will help with follow-up.

In closing, remember that interviews are a two-way street. You’re not just being evaluated; you’re evaluating the employer as well. Find the right fit for your goals and values, and be willing to walk away from opportunities that don’t align with your vision. Best of luck in your job search!

Know your worth.

• Know your salary and benefits requirements. YOU are the only person who can decide. DO NOT let anyone else define your worth.
• I believe this to be the best answer when the recruiter asks “What are your salary requirements?”:

I’m targeting positions with a base salary of at least $N per year. I understand there are many factors that go in to making an offer including salary, equity, performance bonuses, health benefits, time off, etc. Those are difficult to quantify as they vary so much from company to company. If an offer materializes, I will consider all aspects of the compenstation package to make my decision.

Create a respectable LinkedIn page.

• Include respectable profile picture.
• Has relevant career information.
• Connect with people you know.
• Solicit your peers for valid endorsements (if you’re comfortable with this step).

Targeting Open Positions

• Target positions within your a) salary range or slightly above b) experience and skill level or slightly above.
• Many companies are willing to exceed their max salary range for the right candidate.
• Salary ranges aren’t always published so it’s your choice to skip this job (their loss) or pursue at the risk of wasting time because you already know your worth. If you are willing to risk wasting time, then it’s super important to use the initial phone screen as the time to level set on salary expectations.
• Apply to positions you find! DO NOT LIMIT YOURSELF simply because the job description wants 15 years of experience! Hiring managers are looking for candidates to tick several requirements boxes and very few candidates (if any) will tick all the boxes.
• Many companies have multiple positions they are trying to fill that vary only by experience and skill. For example, Company A wants to high four software engineers and they want the hires ratio’d at one senior, two mid-level and one junior. Your goal should be to get in to their pipeline and a solid recruiter will get you to the right door (assuming you’re not a brick).

Connect with the recruiter and/or hiring manager after submitting an application.

• Do your best recon to find THE hiring manager (or a hiring manager if all else fails) for the position.
• Connect with this person on LinkedIn with a message similar to “Hello, my name is and I applied for position XYZ on . Just putting my face to my name. Looking forward to talking with you about this opportunity."

The Phone Screen

• Your GOAL is two-fold: 1) get on the phone with another human and impress this person 2) determine if this is a position you want to pursue and company to whom you want to offer your services.
• Take notes.
• Write down the questions they ask. After the interview, re-visit these questions and write down an answer. This will pay dividends for future interviews with any person or company. Writing is the most legitimate and sincere habit to determine your thoughts on a subject or topic.
• Have 2-3 questions prepared. The worst thing to do is end the call with “nope, no questions from me.”
  1. Tell me a little bit about your relationship to the team for where this position will be?
  2. The job requirement lists a lot of wants and needs for their ideal candidate. Which of these needs and which are wants? What are the non-negotiable skills this person must have?
  3. Tell me about the culture within this team, group, company, etc. without using the word “collaborative”.
  4. Tell me about the interview process starting with where we are right now and through the final interview process. Do your best to get specific details. Some companies have a thorough and well established process. And some companies seem to make it up as they go.

Company Research

• As a hiring manager, it is great to see candidates that show some level of preparation.
• Your decision on when to perform company reconnaissance. Recon can be a time consuming activity for positions that may never materialize beyond the first or second interview. If you do recon at the beginning - great. I think it’s acceptable to defer recon until you’re sure you actually want the job and it seems like a good mutual fit. In this case, the answer during early interviews is “I have not done much research about the company. The position sounds quite appealling so I prefer spend my time on the initial phone screens to ensure a mutual fit and interest. I will read up on the company in preparation for my final interview or interviews.”

Wait

Now, cross your fingers and hope your effort pays off with response. If so, great! If not, on to the next company : )

Got the RabbitMQ blues? It’s overwhelming even for experienced engineers.

REAL WORKING CODE!

Check this YouTube video for a crash course on RabbitMQ. You’ll be up and running within five minutes running RabbitMQ with sending and receiving messages.

My watch is buzzing and in my pre-dawn stupor I cannot decipher if this is an alarm or a phone call. The time is 4:45 AM. I pull it together to realize it’s a call from a number I do not know - never a good sign. I answer and it is a coworker - my peer who runs our support team that is engaged in nearly all production issues for our customers. “Hi Ryan. Sorry to wake you, I know it’s early. Our biggest customer is reporting their requests are taking over two hours to return results. We think it’s because of our messaging system but we aren’t sure where to go from here. We need your help. Please join our call.” A few moments later my watch buzzed again as my morning alarm sounded. Today, this morning will not be for a workout.

For nearly three years we have been running RabbitMQ for our production systems and 99.5% of the time has been a total non-issue. Throughout that time we have scaled to 200+ concurrent consumers running across a dozen virtual machines while coordinating message processing (1 queue to N consumers) and processed hundreds of millions of messages in our .NET application. Our primary use-case is making HTTP calls to another web service either retrieving JSON data or downloading PDF documents. I will tell you that I recommend RabbitMQ and that’s because I do. For the most part it’s been great to work with and it’s performing well in our application. But, and this is a big but, all of this has come at a cost that we did not know at the time we made our architectural decisions.

RabbitMQ is the backbone of our polling architecture to check for job results. The typical action sequence is the user submits a request via the web application and the backend handles that message by adding a message to RabbitMQ. The consumer gets the message and makes a HTTP call to another web service to actually submit the request. From there, the polling logic takes over and subsequent messages on the queue each represent a polling attempt to retrieve the results. If a job has no results, the consumer places a message back on a queue so we can delay the next polling attempt by a (customer configurable) amount of time. Our delay logic uses a network of queues with a time-to-live (TTL) and dead letter definitions.

Our non-prod clusters use either two or three nodes while production clusters use three nodes. Every cluster has a load balancer and the application strictly addresses only the load balancer. At run time, the publishers and consumers use the same load balancer.

Back to business, you’re reading because you want the goods and not this poorly written synopsis of our application.

What You Should Know

Three years after implementation, this is what I would tell myself before I wrote a single line of code interacting with RabbitMQ.

Engage an expert at the beginning

For probably $2000-$3000 (guesstimate) you can engage a RabbitMQ consulting firm and get time with an expert. Use this opportunity to vet and verify your assumptions, plan, ask questions, get recommendations and perform due diligence so you can minimize future headaches, problems and most likely save money in the long run by making the correct decisions now. Or you can take our route, engage the expert when shit is going sideways.

Use a library like EasyNetQ or NServiceBus

Our application uses the RabbitMQ.Client library from RabbitMQ and these abstraction libraries (ex. EasyNetQ, NServiceBus) use it too. However they’re better and know way more than I ever will about interacting with RabbitMQ at such a low level. The driver from RabbitMQ is low level, primitive and expects you to understand nuance about RabbitMQ. If this is your first time with RabbitMQ then I guarantee you will not have experience to appreciate this nuance.

Before you ask “Why didn’t you use a wrapper library?” let me tell you. In my case, our RabbitMQ project landed in my lap when the original developer left the company near the end of the implementation and he decided to use the RabbitMQ.Client library directly. I did not have enough time to make that swap (nor did I know I should have made a case to swap for a wrapper library!).

There’s this Network Partition thing, it’s kind of a big deal

For common terminology, your RabbitMQ system is called a cluster. A cluster is comprised of one or more nodes. A node is simply a server/container running the RabbitMQ software. All nodes in a cluster must run the same exact version of RabbitMQ.

RabbitMQ provides a mechanism called clustering so that you can link other RabbitMQ instances so they function together as a single logical broker. You can address any node in the cluster with any request and the nodes will cooperate to publish the message or send the message to a consumer.

The nodes are communicating with each other constantly by exchanging data about messages, queues, exchanges, etc. If (and when) that communication is interrupted even if only for a few milliseconds then RabbitMQ enteres a partitioned state and looks to the configuration file to determine what to do about this communication interruption. The default partition handling strategy is ignore which means to just enter the partitioned state and keep trucking along in this “split brain” mode thereby thrusting your cluster in to total chaos. This was hell for us (and a lot of hell for me). The only way to exit the parition to restart the nodes of one side of the partition so it will then rejoin the other side and assume their data thereby discarding it’s own data set that it accumulated while the cluster was partitioned.

I have personally experienced network partions happening in two ways: all nodes in the cluster being updated at the same time through Windows update and firewall rules. The fix for Windows update was the ensure that nodes in the cluster are patched at different times.

I have to stop myself as I could continue to rant and rave about this topic for countless words. The correct configuration is to set the partion_handling strategy to pause_minority. When the cluster is partitioned, one side of the partition will simply turn itself off thereby totally avoiding the split brain scenario. The side that is off will continue to monitor the cluster for resumed communications and will rejoin itself at that time. Now all you have to do is ensure your code properly handles disconnected connections and you will have a fairly robust queuing solution.

From CAP Theorm, ignore means to sacrifice Consistency at the expense of Availability while pause_minority is to sacrifice Availability at the expense of Consistency. The latter is quite worth it, if you’re asking me.

How are you going to upgrade RabbitMQ versions?

The day will come when your version of RabbitMQ has reached end of life. Then what do you do? Continue to operate the unsupported version? Create a new cluster? What will be your plan to migrate traffic from the legacy cluster to the new cluster? Recall my note (above) that all nodes in a cluster need to run the same exact version. Hopefully you can see how this will be tricky if your plan will be to upgrade the nodes in-place.

I leave you only with questions, no answers. This is because every decision is highly dependent on your organizational and operational strategies. In other words, everyone might have a bit of a different approach to solve for these problems.

What’s your plan if you lose all messages in RabbitMQ?

How screwed will you be if you were to lose all (or even a third) of your messages in RabbitMQ? Is RabbitMQ your system of record? Do you have a recovery strategy for getting your application back in to a functional state? What happens when you move your on-prem servers to the cloud - how do you get your RabbitMQ messages flowing again?

Build your application to support different connection addresses for publishers and consumers

At some point in the future (perhaps during an upgrade) you will want the flexibility to independently publish to and consume from different clusters and/or load balancers. This is a zero-risk-high-reward pattern you can early-on build in your application for where you will pat yourself on the back in the future.

The log files will grow to consume dozens of gigabytes of disk space

The log files from RabbitMQ will, over time, grow to consume dozens of gigabytes of disk space. It’s easy enough to rotate those files using rabbitmqctl rotate_logs but strive to automate a process so that “running out of disk space” never causes an outage.

Conclusion

RabbitMQ has been a long-term solid addition to our infrastructure and you are probably making a good decision to use this tool. However, you should also take seriously what I have brought up and at least have a conversation with your peers and stakeholders to decide what about these pain points you should try to address.