Share

Introduction

Logs are a crucial method of obtaining historical context for the application state, as streams of time-series data. They are usually composed of a human-friendly message along with contextual information, providing a timeline and insights on the application state at each step of a given flow. Using a structured format for attaching context means that we can create a document database where attributes conform to certain types and structure and allow for more complex, efficient and consistent queries.

Structured logging helps us interface with logs as a read-only, document database storing a time-series event stream.

Although observability has advanced significantly, log records allow for extended retention and providing precise, application-wide historical context, spanning queue/stream message processing, HTTP request processing and scheduled tasks.

Design Patterns

Log Entry Association

In order to properly group multiple log entries that belong to a specific flow, a common identifier must be embedded in the context. For HTTP requests, this is usually referred to as Request ID, although the same correlation concepts apply to asynchronous, background processing (scheduled tasks, queue message processing). Each unique identifier marks a new request-local context.

Nested contexts enable correlations that facilitate tracking the timeline and state transitions in extended sub-flows. We can create nested contexts in a simple way, by using the parent context ID as a namespace and a constant delimiter that will allow us to deterministically parse the composite context identifier, for example:

• Main Flow: a3003c58-f723-41e3-b50a-a0659d484b60

• Nested Flow: e14f9f9e-003b-4376-bc65-e5e122a4e097

• Composite Flow ID: a3003c58-f723-41e3-b50a-a0659d484b60/e14f9f9e-003b-4376-bc65-e5e122a4e097

You can then select all log records for a given flow or sub-flow using prefix matching (SQL syntax):

request_id LIKE 'a3003c58-f723-41e3-b50a-a0659d484b60%'

and for the sub-flow:

request_id LIKE 'a3003c58-f723-41e3-b50a-a0659d484b60/e14f9f9e-003b-4376-bc65-e5e122a4e097%'

The concept of trace identifier chaining is similar to what AWS ALB does in a more structured form.

Well-Defined, Stable Log Entry Schema

Although often overlooked, it is very beneficial to maintain models for each log entry and treat them as Message or HTTP API Response Schemas. The benefit? Safeguard against making backwards incompatible changes which can have severe downstream implications, such as breaking alert queries, log metric extraction and incident/troubleshooting runbooks.

Schema Definition

Let’s assume a user signup event which we need to log. We decide that the following fields are important:

• The email domain, e.g. gmail.com.

• Whether the email address is valid.

• Whether the password (if provided) is valid.

• The provider: Google or self (application built-in user management).

package main

import (
	"net/mail"
	"os"
	"strings"

	"github.com/rs/zerolog"
	"go.uber.org/zap"
	"go.uber.org/zap/zapcore"
)

// Log is the base type for all log entries.
// Note: only include one interface, depending on your logger library!
type Log struct {
	zerolog.LogObjectMarshaler
	zapcore.ObjectMarshaler
	Type string
}

const (
	SignupProviderSelf   = "self"
	SignupProviderGoogle = "google"
)

// LogEntryUserSignup provides the log context for a user signup event.
type LogEntryUserSignup struct {
	Log
	// HasValidPassword is false for non-password-enabled sign-up methods.
	HasValidPassword bool
	// HasValidEmail reports that the provided address is valid.
	HasValidEmail bool
	// EmailDomain stores the domain part of a valid email address.
	EmailDomain string
	// Provider is an optional signup source: can be one of "self,google".
	Provider *string
}

We will also be using some simple helper functions:

func getDomain(email string) (string, error) {
	addr, err := mail.ParseAddress(email)
	if err != nil {
		return "", err
	}

	at := strings.LastIndex(addr.Address, "@")
	return addr.Address[at+1:], nil
}

func strToPtr(s string) *string {
	return &s
}

Application Logic

func main() {
	email := "tester@domain.com"
	domain, err := getDomain(email)
	le := LogEntryUserSignup{
		Log: Log{
			Type: "user_signup",
		},
		HasValidPassword: true,
		HasValidEmail:    err == nil,
		EmailDomain:      domain,
	}

	zeroLogger := zerolog.New(os.Stdout).With().Timestamp().Logger()
	zeroLogger.Info().Object("context", le).Msg("new user signed up")

	
         // Assuming a non-nil provider scenario
         le.Provider = strToPtr(SignupProviderGoogle)
	zapLogger, err := zap.NewProduction()
	if err != nil {
		panic(err)
	}
	zapLogger.Info("new user signed up", zap.Object("context", le))
}

Note: only a single logging library should be used, here we utilize both for demo purposes!

We receive the following logs:

{
  "level": "info",
  "context": {
    "type": "user_signup",
    "email_domain": "domain.com",
    "has_valid_password": true,
    "has_valid_email": true
  },
  "time": "2026-01-25T10:33:15+00:00",
  "message": "new user signed up"
}

and:

{
  "level": "info",
  "ts": 1769329995.225628,
  "msg": "new user signed up",
  "context": {
    "type": "user_signup",
    "email_domain": "domain.com",
    "has_valid_password": true,
    "has_valid_email": true,
    "provider": "google"
  }
}

Note how in the second case (zap) we also define the provider context field.

zap

Structs implementing the zapcore.ObjectMarshaler interface allow for controlled, structured serialization. Example code:

func (l LogEntryUserSignup) MarshalLogObject(enc zapcore.ObjectEncoder) error {
	enc.AddString("type", l.Type)
	enc.AddString("email_domain", l.EmailDomain)
	enc.AddBool("has_valid_password", l.HasValidPassword)
	enc.AddBool("has_valid_email", l.HasValidEmail)
	if l.Provider != nil {
		enc.AddString("provider", *l.Provider)
	}
	return nil
}

Note that within the MarshalLogObject implementation we can control which fields should be part of the context.

The struct instance can then be added as log context using zap.Object:

// le is an instance of LogEntryUserSignup
zapLogger.Info("new user signed up", zap.Object("context", le))

zerolog

Structs implementing the zerolog.LogObjectMarshaler interface allow for controlled, structured serialization. Let’s see the example code:

func (l LogEntryUserSignup) MarshalZerologObject(e *zerolog.Event) {
	e.Str("type", l.Type).
		Str("email_domain", l.EmailDomain).
		Bool("has_valid_password", l.HasValidPassword).
		Bool("has_valid_email", l.HasValidEmail)
	if l.Provider != nil {
		e.Str("provider", *l.Provider)
	}
}

The struct instance can then be added as context using zerolog.Event.Object:

// le is an instance of LogEntryUserSignup
zeroLogger.Info().Object("context", le).Msg("new user signed up")

Logger Instance in Context & Child Loggers

Although passing dependencies via the context.Context parameter is often debatable, primarily due to hiding dependencies, creating optional branches and additional handling, the logger instance might be a good candidate for context injection because it changes per call; the logger can indeed carry request-scoped local context per goroutine, such as in HTTP request handling.

Using the context container allows any function to obtain this dependency without making it a specific, static dependency in service objects. The pattern is as follows: Cloning the logger, attaching extra context (see zap.With) and creating a context.Context copy with the new logger value, will allow any subsequent layer to emit log entries with the corresponding parent context.

logger := logger.With(zap.Bool("signup", true))
ctx := logging.WithContext(ctx, logger)

Best Practices

HTTP Request ID as Message Metadata

HTTP Requests often generate background tasks (database), publish messages in message queues (SQS) or message streams (Kafka). When task processing fails or you need to investigate the activity that resulted in each task (e.g. due to volume/high backlog), it is useful to embed the message publisher request ID in the message metadata. This will allow you to include the originating request ID in the task log context. By filtering with the HTTP Request ID later on, you can obtain a full view of the timeline (when was each task processed?) and correlate failures with actual requests.

What about tracing tasks that were generated by other tasks?

Tasks that publish messages themselves can create nested contexts, resulting to a composite request ID. Let’s break it down with example values:

• HTTP/ChangeUserProfile: a3003c58-f723-41e3-b50a-a0659d484b60

• Task/ProcessUserNotification: 64b051a2-dff3-4f64-9ca0-c40ee138d55e

  • Task/SendEmail: 2141e184-fe61-4d83-ba1e-6ef04d78283f

Sample log for Task/SendEmail:

{
  "time": "2026-01-25T08:47:02+00:00",
  "trace_id": "a3003c58-f723-41e3-b50a-a0659d484b60/64b051a2-dff3-4f64-9ca0-c40ee138d55e/2141e184-fe61-4d83-ba1e-6ef04d78283f",
  "message": "email sent successfully",
  "email_domain": "gmail.com",
  "task_processing_time_seconds": 0.24
}

Note that a Message ID is a different entity and can be a persistent property of the message itself (e.g. see SQS). Here we are referring to an execution-scoped identifier, which will change in case of retries.

Querying all log entries for which the trace_id starts with the HTTP Request ID, will provide the full context about HTTP & task execution.

Avoid Arbitrary Object Logging

Context definition should extract and define specific fields and translate application-level object fields to logging fields. Although it seems convenient at the time to log arbitrary objects (e.g. the entire User object), it has the following drawbacks:

• Unnecessary log volume increase.

• Noise-vs-signal ratio increase/log pollution.

• Unexpected secret or privacy leaks due to extensions and changes in the object fields.

In essence, something like the following should be considered an anti-pattern and be avoided:

type User struct {
    Email string
    FirstName string
    LastName string
}

u := User{Email: "tester@domain.com", FirstName: "Tester"}
zapLogger.Info("user signup", zap.Any("user", u))

See also the section above regarding maintaining specific log schemas.

Timezone

Regardless of your application timezone settings, the timestamp included in the log entry should be in UTC. Keep in mind that log aggregation systems maintain their own ingestion timestamp, however there might be some latency between generation and ingestion, usually in the order of only a few seconds, but could potentially extend to minutes at times in case of maintenance or resource contention.

Sampling

Production scale, external inputs, transient issues with dependencies, or a bug can cause extremely high amounts of logs being generated. Large log volumes can cause:

• Each application process consumes too much CPU/memory on serialization and output buffers.

• Excessive infrastructure costs in the log aggregation system (i.e. ELK, Grafana, Datadog).

• Too much log noise makes it more difficult to filter for other important logs.

This is where sampling takes effect. Most logging libraries will maintain a small in-memory cache which groups and monitors log entry generation rate per group. The sampler decides whether each message group has exceeded their quota and will start dropping messages after a certain threshold. New logs will start reappearing at a later moment in time due to the rate reset. You can read more on the reasoning and how zap sampling works here and here.

Sampling should usually be disabled for development purposes since it can unintentionally hide logs.

Avoid logging in low-level layers

It is often tempting to use logging in several functions, emitting logs where valuable local context is present. However, it might be best to propagate context to higher layers in the stack, possibly within error types or other return values. For an illustration of this practice you can check this library.

Some possible drawbacks of this anti-pattern are:

• Reduced control over the log volume. A function that is later reused within a loop, starts emitting log entries for each call.

• Passing parameters that do not have functional use for the function, only utilized in logging context. At the call site, this is non-obvious and pollutes the API and function signature.

• Testing side-effects: error handling should generally be unit tested which is more difficult and brittle compared to typed return values.

Conclusion

Logging may sound trivial until we start considering maintainability & ergonomics and maximing its efficiency across code & service components. The above practices and patterns should at least provide a good starting point for highlighting the possible issues that may arise within a large-scale service.

Cache Gain & Entry Eligibility

Caching gain increases as read/write ratio becomes much higher than 1: write once, read many times. An example is caching a blog post after it has been published or edited, since we expect that it will be read many times after users have been notified about the changes.

Aggressively caching state can quickly remove entries with high read/write ratio and replace them with low-value entries which will be read very few times. Since memory is limited, insertion operations will try to make room for new entries.

More sophisticated cache eviction policies can mitigate the effect of incorrect removal of high-gain cache entries. For example, weighted LRU can take into account (a) the cost of regenerating the cache value and/or (b) hit rate.

In order to determine whether an entry is eligible for caching, an assumption can be made that at least two queries must be performed before an element is eligible as a cache entry. A Bloom Filter can help ascertain that at least two queries occurred, in an efficient manner; if the element is found by the filter, the query value can be cached.

Rate Limiting

A simple rate limiter identifies requests according to an identification token (session ID, user ID, API token etc). A potential algorithm here is:

1. Extract request token.

2. Check if the rate limiting threshold is exceeded.

   1. If no, continue recording activity.

   2. If yes, insert into time-limited blocklist.

3. On blocklist expiration, requests with the same token start being re-evaluated.

Rate-limited tokens can be quickly validated with an error-free hash. Assuming 32-byte tokens the memory storage is of the order:

For 1M entries we need to store at least 32MB, plus the size of the hash table structure which we assume is small in comparison to the actual data storage. Note that:

• We will need to reset the blocklist state in regular intervals, thus requiring garbage collection of now unused objects.

• We will likely need more than a single rate limiter. For example, for HTTP server endpoints, we may need to maintain multiple rate limiters since thresholds often differ depending on the (a) the verb and (b) the mutation operation impact.

Primer on Probabilistic Data Structures

What are probabilistic data structures and why are they useful?

Probabilistic Data Structures are data structures that provide approximate answers to queries about a large dataset, rather than exact answers.

The key here is the tradeoff between scalability and accuracy. If our application logic can tolerate certain loss of precision, while achieving 100% accuracy would amount to extreme computational or memory requirements and thus become prohibitive, then perhaps introducing a probabilistic structure is the solution.

Two well-known algorithms with wide applications are:

• HyperLogLog: a cardinality estimation algorithm, present in several database implementations. Counters usually serve as indicators of magnitude, e.g. X number of unique visitors per day where 100% precision is not mandatory.

• Bloom Filter: allows efficient set membership testing and is applicable where low false positive rates can be tolerated, for example in browser URL safety checks.

Building a cache for invalid JWT tokens

Web session and API authentication often use signed/encrypted JWT as a serialization format for context-aware tokens. Verifying an invalid token can still be computationally intensive which can cause Denial-of-Service conditions. In order to protect application servers against this type of condition, we need to find a reliable, lightweight way of quickly marking and detecting previously evaluated invalid tokens.

How does a bloom filter work?

• A bit array (or bit set) with fixed positions is preallocated. All elements are initially unset (zeroed).

• A new token is hashed and the corresponding matching bit positions are set.

• Whether a bit was already set or not is not taken into account.

• Queries perform the same hashing process and evaluate if bits on matching positions are set. If all positions are set, then the test returns a positive result - the element is a member of the set.

As you might observe, a false negative is never possible!

You’ll notice that if the bit array nears a state where most positions are set, then the filter degrades to a high false positive rate, gradually becoming unreliable. However, by choosing the hashing functions and the bit array size carefully it is possible to obtain an acceptably low false positive rate.

A Bloom Filter cannot grow after its original construction in order to increase its capacity.

With regard to memory requirements, a Bloom Filter estimated to accept 10,000 elements with a false positive rate of 0.01% will occupy approximately 200KB.

For our use case, we will try to construct an algorithm that avoids expensive computations (signature verification) on signed JWT tokens that have already been evaluated as invalid. Our probabilistic cache will automatically reset after a predefined duration. New tokens are expected to be constantly issued and different values would otherwise fill the bloom filter over time.

The following implementation uses a read/write mutex and is goroutine-safe.

Let’s also perform a benchmark to measure how a Bloom Filter can reduce computational overhead from incorrectly signed tokens.

Results of this benchmark on a M2 Pro:

cpu: Apple M2 Pro
BenchmarkJWTVerification_HS256_Failed
BenchmarkJWTVerification_HS256_Failed-10         	  368486	      3197 ns/op	    2993 B/op	      56 allocs/op
BenchmarkJWTVerification_HS256_BloomFilter
BenchmarkJWTVerification_HS256_BloomFilter-10    	21016758	        56.43 ns/op	       0 B/op	       0 allocs/op

We observe a drastic reduction on computational time (~60x) and memory allocations dropping to 0.

Conclusion

Bloom Filters provide an efficient way to perform set membership tests. We examined a use case where invalid JWT tokens can be cached in order to avoid further unnecessary computations on subsequent requests using the same invalid token.

References

• https://en.wikipedia.org/wiki/Bloom_filter

• https://github.com/bits-and-blooms/bloom

• https://en.wikipedia.org/wiki/Randomized_algorithm#Data_structures

• https://www.geeksforgeeks.org/dsa/introduction-to-the-probabilistic-data-structure/

• https://eli.thegreenplace.net/2025/bloom-filters/

• https://www.researchgate.net/publication/266650658_Weighted_Cache_Replace_Algorithm_for_Storage_System

Switching to a new language is not an easy decision and it may affect the entire software development lifecycle. Time & effort needs to be spent to ramp up the engineering team’s understanding of the language syntax, design/testing patterns & optimal code architecture. Perhaps more importantly, the team needs to familiarize with the operational behavior of the runtime under production workloads!

The more widespread adoption of service-oriented architectures and the advances in operational automation nowadays allow the use of a wider range of languages and runtimes. Some example factors that may affect this design decision can be the following:

• A runtime with efficient concurrency support can provide very high throughput with a very low resource footprint, depending on the workload characteristics (e.g. NodeJS, Go).

• A strong ecosystem and community may supply mature, well-tested, efficient libraries and frameworks that can boost development considerably. For example, Python has been so far a very popular choice for building Machine Learning services and enabling Data Science research.

Learning a new programming language and familiarizing with the runtime and the ecosystem can be both interesting and challenging at the same time. Obviously acquiring expertise requires time and frequent hands-on work on real use cases and tasks. Let’s see if we can approach learning in a way that helps us become efficient faster!

Getting Started

Finding resources and reading material. Several languages offer abundant web-based resources for running code snippets, a structured language reference and extensive tutorials. I would recommend starting with a book that’s focused on teaching the fundamentals of the language and allows you to gradually build up your knowledge. The Learning series by O’Reilly is usually structured this way, but if you’re already an experienced software engineer perhaps you’d be more comfortable with more fast-paced, advanced material. I would definitely advise against skipping the fundamentals completely though, as you’ll keep stumbling on minor issues.

Find a technical book that’s focused on teaching the language fundamentals.

Installing the language runtime is our next step, as we need to be able to execute any code snippets built during the learning process. While you can use a browser-based, language playground, you might find it beneficial to familiarize with runtime installation and its behavior.

Prefer a local installation of the language runtime.

Setup your editor/IDE to work with the specific language. A recommendation here is that your configuration allows you to quickly jump to type & function definitions of standard or third-party libraries and read the code. You can see an example with Jetbrains Pycharm here. There are some significant benefits of having this capability:

1. You can read the documentation and actual code for a standard library function, without leaving your editor window. The standard library tends to be also highly optimized and idiomatic, since it’s maintained by experts on the language.

2. If you have enabled a debugger, you can do start a step-by-step debugging session to confirm how the code works.

Start a small project. While writing simple functions, loops, if statements is definitely a good first step, you’ll probably need a small toy project, such as a Todo List manager or something similar that requires CRUD operations and working with data.

The Basics

First things to cover:

• Primitive types

• Variable initialization

• Arithmetic and logical operators

• If-else statements

• Loops

• Iterables

While not ideal for production-ready applications, translating already well-understood code from a different language is still useful for learning, since you’ll start mapping already familiar concepts right away!

Mathematical problems are usually the easiest to implement at first, e.g. calculating the sum or the median of an array of integers which will require some use of loops, conditional statements and data structures like arrays. It’s better to start with pure functions at first and avoid any kind of state management. Usually such problems are ideal since they can be modelled using a procedural approach. Procedural programming is universally available across languages and even on heavily object-oriented languages (e.g. Java) you can define functions as static methods attached to classes.

Do not spend effort on idiomatic syntax and performance optimizations at this stage.

You can then proceed with learning about custom type & function definitions, inheritance and object model (if applicable).

More Advanced Topics

• Writing tests: familiarizing with test runners and assertion libraries. Some languages have built-in testing frameworks some others don’t.

• Errors/exceptions and stack traces: being able to identify the root cause for a failure is essential for your development workflow and production deployments alike.

• Mutable and immutable types: mutability is an important concept and differs among languages. It affects how code is structured and when not thoroughly understood can lead to serious bugs or memory issues.

• Concurrency model: most languages support concurrency either via green or OS threads, multi-processing, async I/O. It’s useful to familiarize with the fundamentals, just to be aware of what capabilities are supported.

• Variable scope: some languages can create locally scoped variables in statements and blocks. Incorrect scope understanding can become a source of errors.

• Closures

• Error-prone anti-patterns: for example you can find here a list of common Python gotchas.

Common Mistakes & How to Avoid Them

Reinvention

The uncertainty of entering an unknown ecosystem can often lead to some tendency to reinvent. If you find yourself implementing something seemingly very fundamental, you may need to take a step back and do some research first. Unless you’re solving a very unique problem, the chances are that there are already available libraries that can help partially or fully address your requirements.

Translation

You can sometimes notice that a codebase is heavily influenced by a language different than the one it’s being written in. It takes some time to get used to another language’s optimal code architecture & patterns. How do you know if you’re writing something in the recommended/most popular way? Well, sometimes it won’t be easy to understand whether something seems too difficult because it shouldn’t be done this way or because you’re just struggling due to inexperience. Sometimes linter tooling can help point you to the right direction, but it’ll require mostly reading (a lot!) existing code from the standard library and very popular OSS projects, as well as articles from language experts.

Full Software Development Lifecycle

Do not underestimate the effort and learning curve to make the code testable, in addition to learning the whereabouts of the testing frameworks. In-depth knowledge of the testing frameworks is necessary in order to give you confidence that the code works as expected. Although your knowledge on testing practices is definitely transferrable, what often differs greatly is testing frameworks, stubbing/mocking libraries and injection patterns, and of course assertions.

Different performance characteristics and runtime design may also require different production stack. For example, NodeJS is very performant for async I/O but is single-threaded when it comes to CPU-bound operations and the same applies to Python with ASGI. Thus, production-ready application servers must be capable of spawning multiple sub-processes in order to take advantage of multi-core machines.

In the upcoming articles the following topics will be explored:

• Message Structure and Serialization Formats

  • Headers & Versioning

  • Schema Validation

  • JSON, YAML, Thrift, Avro, Protocol Buffers

  • Language-specific marshalling

  • Security: Signing and Encryption

• Essential Data Structures

  • FIFO/LIFO/Priority Queues

  • Sorted Sets

  • Sharded Queues

• Messaging Patterns

  • Event Notification

  • Topics & Streams

  • Task Queue

  • Pub/Sub

• Message Delivery

  • Fanout

  • Early / Late & Negative Acknowledgement

  • Failure Handling with Dead-Letter Queues

  • At-least-once/Exactly-once/At-most-once Guarantees

  • Deduplication

• Message Brokers

  • RabbitMQ

  • Redis

  • Kafka

  • AWS SQS

  • RDBMS (MySQL/PostgreSQL)

• Task Design

  • Idempotence & Optimal Parameter Selection

  • Timeouts

  • Error Handling & Retries

  • Acknowledgement Timeout

• Task Processing

  • Worker Pool Concepts & Types

  • Context & Tracing

  • Horizontal Scalability & Auto-Scaling

  • Interdependent Tasks - Directed Acyclic Graphs

  • Scheduled / Delayed Tasks

  • Circuit Breaker Pattern

  • Rate Limiting

• Frameworks, Services and Workflow Management Platforms

  • Resque, Sidekiq, Celery, Faktory

  • AWS Lambda

  • Apache Airflow, Luigi

The above topics should provide a comprehensive theoretical background but how applicable are these concepts to actual use cases? Throughout the article series we’ll be using a sample application comprising several processes, to help us visualize how each architectural pattern addresses real-world scenarios. Our application will be an implementation of a very common System Design question, the URL Shortener Service.

The main goal of the service is to provide one or more unique, shorter URLs that redirect to the target URL:

GET https://url-shortener-service.io/q2csXrZC

HTTP/1.1 301 Moved Permanently
location: https://controlflow.substack.com/async-task-architecture

This is a seemingly simple operation, but let’s try to lay out some of the features and functional requirements for a SaaS offering and determine where async task execution is applicable.

Authentication & Authorization

Email address verification during signup

An email containing the verification URL is sent to the email address of the new user account.

Password Change/Reset Requests

Users will be notified on password changes and must receive a link for password resets.

Redirect Setup

When a new redirect is configured, the change must be propagated to the redirect proxy service, which is likely using a cache of valid URLs. Likewise, when a redirect is disabled/removed, the cache must be updated to stop serving requests for this short URL.

Analytics

The service collects analytics for each redirect, such as total views, view count aggregated by referrer domain or user agent.

The redirect operation should have very low latency, thus the service must forward the HTTP request information to a queue or stream for async processing.

Paid Accounts

Our service will have paid features such as using a custom domain for shortened URLs and will offer a free tier that limits the total active redirect URLs or the total views per month.

Cost calculations occur asynchronously and warning notifications about reaching the end of the free tier quota are also sent via email in the background.

Billing

Assuming a usage-based billing plan, the corresponding amount must be calculated and charged on the account’s credit card each month. A new invoice is issued and the billing contact receives it as an email attachment.

The users may also want to configure billing alerts in order to be informed early about sudden increases in charges.

The above operations are not triggered by a user action, but instead are executed on regular intervals in the background.

What’s next?

The next article in the series will discuss Message Structure and Serialization Formats.

Coding Challenge

In a coding challenge, the candidate will be given a relatively simple problem, that needs to be translated to working code to be solved. Someone may now ask, isn’t that called an algorithmic challenge? As you’ll see next, I reserve this term for challenges where designing and understanding the algorithm itself is a substantial part of the interview.

Some problems that fall into this category:

• FizzBuzz

• Count the occurrences of each string in an array of strings.

• Find the common elements between two arrays of integers.

These problems sound simple but they can provide significant insights into someone’s capacity to effectively communicate their thoughts and display their methodology to problem-solving in general. In addition, there is often an expectation to write unit tests to verify the solution, along with identifying and covering edge cases in the code.

Although not usually crafted for performance-related discussions, there’s still room for such questions in the everyday engineering context. Are we applying micro-optimizations versus favoring readability and maintainability for non-performance-critical code? Are we using a sub-optimal custom implementation that could be replaced by a standard library function?

Another interesting aspect is testing the knowledge depth and experience of a programming language. Are they knowledgeable about the standard library and language constructs? Do they write non-idiomatic or even error-prone expressions? Are they familiar with the error handling and unit testing fundamentals? Note that language selection is often open to a large set of popular languages, but certain companies will narrow the choices down to a smaller subset or even a single language, if they consider these fundamental to the organization and a prerequisite for any position.

You may wonder what happens when the interviewer doesn’t have experience with the language chosen by the candidate. Although a not-so-ideal scenario, an interviewer can still assess the ability of the candidate to explain technical matters to someone experienced in software engineering in general, but having a specific knowledge gap. Being able to narrow down the scope of shared information in this case is also a sign of experience. If you have mentored or collaborated with other engineers closely, you likely had the chance to explain your approach on multiple occasions.

🟡 Scalability/Performance

🟡 Testing

🟡 Problem-Solving

🟢 Language Proficiency/Ecosystem

🟢 Practicality

🔴 Systems Thinking

Algorithmic Challenge

In this case, the candidate is requested to provide an algorithm for solving a difficult problem, such as Tree traversals or other operations, Graph searches, sorting. The solution usually requires a deep understanding of data structures, graph theory, set theory and mathematics. Apart from the inherent difficulty, these problems most likely have performance-related components, which must be discussed during the interview in the form of analysis on space and time complexity.

Although pseudo-code may be applicable, or even requested as an initial step, most interviews will also require writing working code as well, and even reach the stage of delivering unit tests along with the solution. You may now wonder if that’s a more difficult Coding Challenge. There is indeed an overlap, but due to time restrictions, the interviewers will likely not have enough time to dive deeper into all the focus areas that were mentioned in the previous section.

A significant drawback of algorithmic challenges is that they may require prior knowledge or exposure to a specific Computer Science problem (e.g. detecting loops in linked lists). Although working with the interviewer to see how you approach problem-solving still plays an important role, there is a chance that some candidates will not go very far in the discussion unless they have a breakthrough moment.

Interviewers may also face a problem with becoming biased when knowing the solution beforehand. Inevitably the solution seems obvious over time. This effect can be mitigated, by alternating interview questions after a fixed period of time, e.g. every quarter.

There is a major debate on the value of difficult algorithmic problems within an interview scope. They are frequently regarded as not being in touch with everyday engineering practices and possibly dismissing valuable experience of other important aspects of software engineering. A counter-argument may be that they reveal the candidate’s exposure to highly-scalable, performance-critical applications where space and time complexity matters to the extent of rendering a solution non-functional. In addition, a strong CS background is actually required in some positions. For example, if you are interviewing for a position in a team developing a NoSQL database system, you’ll have to display some understanding of how Trees work.

🟢 Scalability/Performance

🟡 Testing

🟢 Problem-Solving

🟡 Language Proficiency/Ecosystem

🟡 Practicality

🔴 Systems Thinking

Project Assignment

While the two previous types are almost exclusively performed synchronously in collaboration with the interviewers, a project is usually given in the form of an assignment. It is often criticised as requiring a significant time investment on behalf of the candidate, though this varies on the scope of the project: some assignments can be fully completed in 1 or 2 hours, others will imply an expectation of spending 14-20 hours within a calendar week for a high-quality deliverable.

The goal is to simulate a real-world feature and evaluate the candidate’s experience on several important aspects of software engineering, such as language ecosystem knowledge, framework experience (e.g. React, Rails), Continuous Integration and testing, documentation. The usual requirement is to create (or modify) a service with an HTTP interface or write a CLI tool.

Usually, the deliverable is the complete application code either as a private repository or as a compressed archive. Candidates are expected to write clear and concise documentation on how their app is deployed or used in the command-line. For Git repositories, interviewers may also want to take notice of version control specifics, such as commit messages, sequence and content.

The project specifications may also require that a specific third-party API is utilized. Interactions with several services are commonplace in today’s architectures that depend heavily on Infrastructure-as-a-Service providers for core operations (e.g. object storage with AWS S3/Google Cloud Storage) and integrate with other service vendors for their actual business logic, for example, transactional email or payment gateways.

Upon successful completion of the assignment, there is usually another technical interview round where the candidate presents the solution to the interviewers with some follow-up questions. The candidate may also be asked to implement additional functionality during the interview, which tests both their understanding of the delivered solution and their ability to adapt to new requirements along with their communication and collaboration skills.

An interesting variant of the technical assignment is to be granted access to a draft of the project solution and apply enhancements or new features there. The Code Review workflow may also be followed: the candidate notes some comments in a change request and is able to communicate with the interviewers asynchronously in writing. During the interview each comment is discussed and any agreed suggestions are then implemented by the candidate. An example of this variant is GitLab’s technical assignment.

🟡 Scalability/Performance

🟢 Testing

🟢 Problem-Solving

🟢 Language Proficiency/Ecosystem

🟢 Practicality

🟡 Systems Thinking

Conclusion

The approaches mentioned here display several differences, but what they do have in common is that communication always plays a key part. As a candidate, you’ll need to explain your ideas, ask for clarifications and work together with the interviewer to reach the optimal solution. As an interviewer, you should also focus on evaluating how a candidate deals with unknowns, adapts to new information, communicates and approaches problem-solving in general.