dev.in.the.shell

How to SUCK at pair programming

Thu, 03 Jul 2025 23:00:00 GMT

Pair programming is a wonderful technique where two developers come together to accomplish one task with half the productivity and twice the resentment.

For those looking to derail this process with style and master the art of collaborative sabotage, here's your step-by-step guide to making every session as painful and unproductive as humanly possible.

Establish Dominance

The driver role is sacred, and by sacred, I mean yours. Hold on to it for dear life. When your partner timidly suggests a role swap, laugh softly and say, "No worries, I’ve got it".

You didn't get here by being a team player, you got here by pushing people around until you get your way.

Let 'em know who's boss.

Ignore Your Partner

They mention a typo? Nod silently and keep typing. Suggesting a better approach? Let out a passive-aggressive "mhmm" and proceed to do it your way (the best way).

Don't miss the opportunity to make your partner feel like there is no point to even talk.

Human Rubber Ducky

Tired of writing code? Time to piss off time looking at your phone. This is a great moment to mercifully allow that other poor soul to type.

Your primary role here is to be utterly, completely, and silently useless.

Your partner types, you stare. They debug, you breathe. They ask for input, you offer a cryptic grunt or perhaps a well-timed yawn.

The goal is to be less helpful than a syntax error, essentially transforming yourself into a warm body occupying a chair.

The ~Navigator~ Distractor

Since you are not typing, hence not paying any attention whatsoever, it's a great moment to bring up anything that comes to mind. The less relevant the better.

That meeting that nobody asked for? Complain about it. You ate some crap last night and got mild diarrhea? Let 'em know. Planning your next trip? Give them all the details.

Bonus points if they are debugging prod.

Nitpicks For Days

Conversely, if you can't manage complete apathy, go for the opposite extreme. Every keystroke is an opportunity for critique.

"You missed a white space!" "Do you really need a while loop?" "I don't like that variable name!"

Your partner should feel like they're undergoing a highly aggressive driving test, with you as the perpetually disappointed instructor. The key here is to offer no constructive alternatives, just relentless, nitpicky condemnation.

Weaponize Questions

You already know it all, so questions only serve one purpose: to traumatize your co-worker.

"You know how this works, right?" "Did you not see this coming?" "Who would write crap like this?"

Passive-aggressive is your middle name.

Praise, But Not Really

Sprinkle in some motivational comments like:

"I like it...as a first approach." "That's quite good...for someone your level." "That's an...interesting way to do it."

Make them feel like there's hope...and then crush it.

Your Own Best Practices

Use the "best practices" hammer to strike down on any and all approach to programming you personally dislike.

Don't like functional programming? Best practice is to use OOP. Don't like layers? Best practice is to do everything in one file. No modules, no nothing. Call it something fancy like "locality of behavior" to hide the fact that it's just bullshit. Doesn't matter if you use the term incorrectly, just shoot some fancy-sounding words at the problem.

"Best practice" is whatever you want it to be.

Code Style Is A Weapon

Spacing? Braces? Tabs vs spaces? Semicolon? You are the Judge, Jury, and Linter. Repeat with me: "I am the law".

Declare a new law every 15 minutes. Retroactively shame them for not knowing the law.

If they question the law, come up with some "industry standard" that confirms the law and refuse to cite sources.

Make It Weirdly Personal

If they make a mistake, use it as a segue into a dysfunctional psychotherapy session.

"You are very thorough with your tests. Do you feel insecure about your ability to write bug-free code?" "Interesting variable name. What made you come up with it?"

Don't evaluate the code, judge the person.

Conclusion

Remember, pair programming isn't about collaboration. It's a psychological endurance sport, and you don't need to be the best: you just need to make your partner collapse faster than you.

It's a great chance to do fuck all while pretending to work, a chance to show someone how inferior their thought process, syntax choices, and entire existence are compared to yours.

Have some fun, do nothing productive, make them suffer.

Try, Catch, Repeat

Thu, 05 Jun 2025 23:00:00 GMT

Quick overview of different ways to handle errors in software development.

Error Codes

The simplest and oldest way to handle errors. You would usually return an integer or null.

Unix shells famously follow this convention by returning either 0 for success or 1-255 in case of error. This way, not only can you inform the caller of an error, but also specify the kind or severity of it.

In C, -1 is commonly returned to indicate an error. Whenever the return type cannot be constrained to an int, NULL is returned instead.

Of course there's an issue here: it's just a matter of convention (what code should I use?). There is no specific error type or compile time error that tells you how to handle the error. In fact, the caller might very well just ignore it!

Plus, this approach limits the signature and design of a function call. What if I try to fetch a user from a DB and fail to connect? I can return NULL, but then what should I do when I just don't find the user? Also NULL?

Exceptions

A much more popular approach is to "throw an exception" or "raise an error", depending on the specific language jargon.

This relies on the language providing some construct to create a separate, conditional execution flow. Sort of like an if/else statement, only in this case, the if branch continues normal execution while the else branch unwinds the call stack halting the rest of the program.

This takes the form of a try/catch in Java or C#:

try {
    doABarrelRoll();
} catch (Exception e) {
    // handle error
}

Or a try/except in Python:

try:
    do_a_barrel_roll()
except Exception as e:
    # handle error

More often than it should, this gets used as a clever way to not pay attention to errors: I can now throw them wherever I want, as long as there's a try/catch somewhere up the stack I'm golden.

As the call stack grows and the number of throws grows with it, following this conditional logic gets incredibly complicated, especially when these Exceptions/Errors are not explicitly declared by the type system at compile time (looking at you C#).

Callbacks

In functional style languages and async heavy programming, functions are passed to the main function being called, which will then be run whenever the main function finishes. These are, of course, callback functions, as in they will be called back at a later point in time.

Typically found in and popularized by pre-promises JavaScript, it looks something like this:

doABarrelRoll((err, result) => {
    if (err) {
        // handle error
    } else {
        // use result
    }
});

Of course, nothing stops you from calling other functions from within the callback function. And more functions in the callbacks for those functions... Welcome to the infamous Callback Hell:

getUser(function(user) {
  getPosts(user.id, function(posts) {
    getComments(posts[0].id, function(comments) {
      sendNotification(user.email, function(response) {
        console.log('Notification sent:', response);
      }, function(error) {
        console.error('Failed to send notification:', error);
      });
    }, function(error) {
      console.error('Failed to get comments:', error);
    });
  }, function(error) {
    console.error('Failed to get posts:', error);
  });
}, function(error) {
  console.error('Failed to get user:', error);
});

Of course, this is mostly seen in legacy codebases. JavaScript (thankfully) now uses async/await and try/catch, which is far from perfect, but miles better than this.

Returning the Error

As an alternative to error codes, some languages (most notably Golang) allow for multiple return values.

res, err := doABarrelRoll()
if err != nil {
    // handle error
}

This way, the error doesn't interfere with the actual result of the operation, and both can be independently typed and checked accordingly.

Pretty nifty, but also quite verbose: the code gets littered with if err != nil everywhere. Then again, at least with this approach devs are forced to confront and handle errors, which might be tedious but also necessary.

Also of note, nothing is stopping you from writing a function that returns User, Post instead of User, Error, so there's a whole other way to shoot yourself in the foot here. In fact, some languages simulate this pattern by wrapping the return values in a list, so that one "value" is return containing n values. Complexity here can get out of hand real fast.

Returning a Wrapper

Similar in spirit from that list approach, we can be more explicit about the return values by wrapping them in a dedicated type.

A Result type is often used for this, such as in Rust:

match do_a_barrel_roll() {
    Ok(success) => /*handle success*/,
    Err(error) => /*handle error*/
}

Where the function signature looks something like this:

fn do_a_barrel_roll() -> Result<i32, String>

And the definition of the Result type is:

enum Result<T, E> {
    Ok(T),
    Err(E),
}

Or in Kotlin:

when (val result = doABarrelRoll()) {
    is Ok -> /*handle success*/
    is Err -> /*handle error*/
}

Or Haskell:

let result = do_a_barrel_roll
    case result of
        Right val -> -- handle success
        Left err  -> -- handle error

Notice how Haskell is different in that instead of Ok and Err it has Right and Left.

This is because, instead of having a Result type like Rust and Kotlin, it uses Either as its return value. In fact, Result is little more than a specialized version of Either, where the semantics more clearly indicate the meaning of each possible value.

This paradigm is kinda hard to get accustomed to, as it forces you to think about the error just as much as you do about the happy path. Which is awesome IMO, but needs some getting used to if you are used to the "throw an error and hope for the best" school of thought.

Also, this requires either a language with a decently strong type system or very diligent developers.

Conclusions

At the end of the day, the language you use often defines the error handling approach for you.

That being said, as long as the rest of the team is on board, I would suggest playing around with different paradigm from the ones your stack defaults to.

Not everyone finds the same error handling system equally intuitive, and some contexts are just better suited for some paradigms than others.

Your User Stories Suck

Thu, 22 May 2025 23:00:00 GMT

I have mixed feelings about User Stories. Not the structure, necessarily, but the idea that the end user is the only perspective that matters. I feel this is distilled pretty well in User Stories, so I'll use them as scapegoats here.

At a high level it's pretty reasonable: focus on the user when defining a problem/need.

"As a [type of user], I want [some goal] so that [some reason]."

However, it's not immediately clear how "we need to update the database" or "we are asking to get hacked" fit into this frame. To be more specific, if a dev's job is to take care of an iceberg worth of "shit that needs doing", User Stories seem to only fit well for the exposed tip of the ordeal. Plus, a user can want an infinite amount of features, including features that don't fit well into the product.

What do we do with those other tasks? How can we stop the feature creep? Are user stories not fit for purpose?

User Stories

"User Stories are chunks of desired behavior of a software system." <cite>Martin Fowler[^1]</cite>

[^1]: User Story

Sounds like a great way to ensure the team focuses on adding value to the product instead of tinkering endlessly with not-so-relevant parts of the system.

It seems particularly useful in the context of Lean Startups, or greenfield projects more broadly. Especially in regard to build-measure-learn, where the assumption is that users will guide product direction through measured behavior and feedback.

That same approach kinda falls apart when you’re knee-deep in a legacy codebase or a mature product. These systems are often full of past compromises (made to get to market fast) that now need attention, but The User doesn't know or care about this.

So what's the approach here? How do we make space for that work?

Dev Stories

You might try to create a somewhat separate backlog of "Dev Stories" to ensure technical stuff doesn't get drowned out by shiny new features. But how would that look like?

"As a developer, I want to fix known issues so that I don't get emergency calls at 2 AM." "As a developer, I want a secure, up-to-date tech stack so that my job doesn't suck as much."

Doesn't quite have the same ring to it, does it?

Jokes aside, this would mean juggling two separate backlogs, when one is already hard enough to keep at bay.

Also, how is your Product Owner/Manager/Analyst/Thing supposed to prioritize stuff they don't even understand? How important is keeping your server OS updated? Is it more urgent than the feature marketing wants by Friday?

It's a hard sell without a clear link to user value.

The Temptation of More

The average user seems to want one tool to rule them all. Technically minded people (usually) know that to be a bad idea. A tool that tries to do everything is often great at nothing, and since we (hopefully) want our products to be great, there is a conflict between what the users ~think they~ want and what is best for the product.

Add to that the constant marketing "necessity" to add flashy features to more easily sell the product, and you end up with a great recipe for feature creep.

This manifests as a near-infinite pile of User Stories that often begs the question: Who actually asked for this and why?

I think that question is pretty key.

Do we know what The User wants, or are we just guessing? Is marketing chasing trends or chasing value? Are we conflating The User with the marketing team?

Anecdotally, it's hard for me to map this feature obsession with actual people using actual software. 100% of my non-tech friends and family have 0 interest in The Cool New FeatureTM and would much rather have any given software be better and faster at what it already does, than poorly perform new tricks on every update.

Do we actually know what percentage of our users want/need that new feature, or are we spending time and money chasing hunches?

Hierarchy of (Software) Needs

I find it useful to adapt Maslow's pyramid to software, where each additional level only adds value if the previous one is in place.

In my mind, a software product should be:

Functional: It does what it should.
Reliable: It can be trusted to do so.
Usable: It's accessible to non-tech users.
Secure: It protects user data and privacy.
Performant: It's reasonably efficient.
Scalable: It adapts to reasonable increases in workload.
Delightful: Modern UI, quality-of-life features, polish, etc.

The point here isn't to undervalue #7. It's critically important. But using a beautiful UI to entice a bunch of new users to use a service that can barely cope with the current ones makes no sense. And showing off shiny new features when user data is simply not safe in the system due to poor security practices is not only disingenuous, but immoral.

The issue here is that the average user only seems to see tips #1, #3 and #7 of the iceberg.

Blind User Is Blind

Most users are too technically illiterate to ask for quality software. But that desire exists, it's just 'hidden' as a combination of implicit assumptions and unconscious expectations.

This is often visible after the fact: Once a service gets hacked and its users start getting spam calls all day, they care about security. If the software starts behaving subjectively "too slow", they will complain. If the team takes too long to adapt to market changes due to horrible developer experience, they will leave.

By the time The User asks for security, reliability or scalability to be "added" to a system, it's usually too late. These are not things you add later, they must be built and designed into the system from the start and maintained constantly. They can hardly be just bolted on.

However, I don't think that makes User Stories unfit. It's just a matter of reading between the lines, surfacing implicit requirements instead of waiting for The User to manifest them. These implicit needs should bubble up as any other item in the backlog.

"As a paying user, I want the system to scale well so that broad adoption doesn't hinder my experience." "As a privacy-conscious user, I want my data protected, so future attacks don't affect me even if successful."

If these User Stories don't fit in the backlog, that's just the team not caring about their users. At least not in those regards.

So What Do We Do?

It's always good to question new features:

For whom is it?
How does it fit the system?
What's the cost to stability and maintainability?

Learn to say no. More features is not "more better".

I think we should, on one hand, expose those 'hidden features' as any other User Story, they are just as important if not more. When raising a technical concern about the system, tie it to real user impact.

On the other hand, if you can't do this, either try harder or admit there is no actual reason to be concerned. You might just want to improve something and that's valid too.

Make the terminal great again

Thu, 24 Apr 2025 23:00:00 GMT

The terminal is a very powerful tool, but it seems to scare developers into using less productive GUIs. This guide provides tips, tricks, and tools to enhance your terminal experience, making it more efficient and enjoyable.

Choosing a shell

The shell is what interprets the command you write in the terminal.

This is not to be confused with the terminal emulator itself. When you change the background of your terminal, you are configuring the emulator; when you set up an alias or run commands you are interacting with the shell.

There are a lot of shells out there, but the most frequently used ones are bash and zsh. While bash is the most ubiquitous, zsh offers some worthwhile advantages that make it IMO the go-to option:

Its tab completion is much better than bash
It's bash compatible (unlike some other shells)
It's designed to be extensible via plugins

OMZSH sometimes gets mixed up with zsh itself. OMZSH is a big framework built on top of the z shell, with tons of functionality built in.

While using it might make sense as a first step, I would suggest you move past it as soon as you feel comfortable doing so. Realistically, there are two features you care about here: plugin management and fancy prompts.

As a plugin manager, OMZSH is incredibly overkill. Consider that zsh plugins can be handled simply by cloning and updating them. There is really not much management needed. Alternatives like zap are much, much faster and simpler.

As a prompt, it's clunky as all hell and doesn't really help with customization all that much. Dedicated solutions like starship are much faster and more customizable.

Just in general, it pollutes your zsh config with a bunch of settings you might not really need (or want), as well as a huge amount of aliases you wouldn't even notice are there, but might alter how commands behave.

So yea, I'd say use it to help you get started but leave it behind as soon as you get comfortable. More often than not, less is more.

Plugins

There are (at least) three zsh plugins you should use to greatly improve your experience with the shell.

Use zsh-syntax-highlighting to add pretty colors to your commands as you type. This will help you catch typos and mistakes faster.

zsh-autosuggestions suggests past commands based on what you type, as you type.

You can use fzf-tab to fuzzy find through zsh's built-in tab completion.

Tips

People seem to think that using the terminal requires you to manually type out long commands or have a near-infinite amount of aliases.

Here are some other things you can do to reduce redundant typing.

Clipboard

Copy/pasting to/from the terminal seems to cause more issues than one might expect. Most modern emulators support Ctrl+Shift+C for copying and Ctrl+Shift+V for pasting, but remember that you can always just pipe the results of a command to your clipboard:

some --cool command | xclip -sel clipboard # or wl-copy, pbcopy, ...

Or pipe the content of your clipboard into a command:

xclip -sel clipboard -o | some --cool command # or wl-copy, pbcopy, ...

History expansions

If you use the command line long enough, you will eventually get a Permission denied message after forgetting to use sudo to run a command. You don't need to re-write or copy the whole thing: sudo !! will re-run the last command with sudo at the start of it.

Here are a bunch of other useful expansions:

!! # expands to the previous command
!$ # expands to the last argument of the previous command
!^ # expands to the first argument of the previous command
!* # expands to all arguments of the previous command
!echo # expands to the most recent `echo` command
!?echo? # expands to the most command containing the string `echo`

If these look like random symbols, understanding some basic regex syntax might help.

Simple but useful

Moving around the file system with the command line can be a bit of a pain. Always remember that running cd with no arguments sends you to your home directory, while cd - sends you back (and forth) to the previously visited directory. So you can swap between two distant directories easily.

Similarly, you might find yourself running the same set of commands in the same order multiple times, especially while troubleshooting issues. This hints to a shell script being a better approach, but if that seems like overkill, you can use ; and && to automate this a bit:

echo one && echo two # runs the second command only if the first one succeeds
echo one; echo two # runs the second command even if the first one fails

You can combine as many commands as you want, and go for a coffee while they run unattended.

Tools

As much as those tips help, some things are better handled by some clever command line tools.

z.lua

z.lua "learns" which directories you move to the most, and suggest them by frecency.

In practice, this means that after a while of cding to directories like these:

cd ~/Documents/repos/work/legacy
cd ~/Pictures/trips/colombia

You'll be able to run z leg or z col to go to those directories no matter where you are. It really does feel like it's reading your mind.

Using zsh, you can install it directly as a plugin by adding something like this to your config:

plug "skywind3000/z.lua" # adapt syntax to match plugin manager

fzf

fzf is a command line fuzzy finder. What does it fuzzy find? Anything.

Running fzf in your home directory will list every single file in it, typing something will filter the results and pressing enter prints out the selected entry.

This might not seem like much at first, but consider that anything can be piped into fzf, and its output can also be piped into any other command. We'll see this in action with some useful aliases. For now, adding these env vars to your shell config will make it behave more intuitively:

export FZF_DEFAULT_COMMAND='rg --files -g "!.git" --hidden' # show hidden files but still ignore .git/ dir
# export FZF_DEFAULT_COMMAND='find . -type f -not -path "./.git/*"' # if not using ripgrep
export FZF_DEFAULT_OPTS='--height 70% --layout=reverse --border' # "better" (?) layout

bat and eza

bat is a prettier cat with git integration, while eza is a prettier ls with icons. Plain and simple.

File management

mmv allows you to do bulk rename of files:

More broadly, consider installing a terminal file manager like yazi, ranger, or even good old vifm, especially if you often have to fiddle around with the file system.

Utilities

As far as system monitoring goes, you'd be hard-pressed to find a better solution than btop.

If your struggle to handle git though the command line, lazygit might help.

Same goes for docker and lazydocker.

Aliases

Having gone through those tools, you can see how I cope with my crippling allergy to unnecessary typing:

alias cat="bat"
alias cb="cd .."
alias cc="z"
alias cl="clear"
alias cpd="cp -ir"
alias fm="yazi"
alias l="eza --group --all --icons --long"
alias mkdir="mkdir -p"
alias rmd="rm -rf"
alias sctl="sudo systemctl"
alias tre="eza --all --icons --group-directories-first --tree --git-ignore"
# maybe add `--level=2` to reduce output

Which especially applies to git:

alias ga="git add -A"
alias gamen="git commit --amend"
alias gc="git commit"
alias gcempty="git commit --allow-empty --allow-empty-message"
alias gcom="git add -A && git commit"
alias gfs="git fetch && git status"
alias glg='git log --graph --abbrev-commit --decorate --format=tformat:"%C(yellow)%h%C(reset)%C(reset)%C(auto)%d%C(reset) %s %C(white)%C(bold green)(%ar)%C(reset) %C(dim blue)<%an>%C(reset)" -15'
alias gmkb="git checkout -b"
alias gmv="git checkout"
alias gp="git pull"
alias gpush='git push'
alias gpushf='git push --force'
alias grmb="git branch -D"
alias gs="git status"

Aliases work as long as they are self-contained, but you can't be specific with the arguments. For more complex use cases, we can use functions instead, which for this particular use case will act just like aliases.

Fancy funcs

How often do you create a directory only to then have to cd into it?

mkd() { mkdir -p "$1" && cd "$1" }

Or cd into a directory and instantly run ls?

c() { cd "$1" && ls }

Adding this to your zsh config will allow you to run these functions as if they were built-in commands.

You can fuzzy find files in the current directory and open them with your favorite text editor in one go:

vo() { file="$(fzf)" && nvim "$file" }

Or fuzzy find your zsh history to look for that command you barely remember:

hist() {
    eval "$(fc -l -1 0 | awk '{$1=""; print substr($0,2)}' | awk '!seen[$0]++' | fzf)"
}

How about installing packages? Package managers are awesome, but you need to know the exact name of what you're after. We can use fzf to make our lives much easier:

# Arch
install() {
    package=$(paru -Slq | fzf --preview 'paru -Si {1}') && paru -S --skipreview "$package"
}

# Debian
install() {
    package=$(apt-cache pkgnames | fzf --preview 'apt-cache show {1}') && sudo apt install -y "$package"
}

This will present all available packages in fzf for you to fuzzy find the one you need. Pressing enter on the result will install the package.

Similarly, we can list all installed packages in fzf for easy removal:

# Arch
remove() {
    package=$(paru -Qq | fzf --preview 'paru -Qi {1}') && paru -Rns --noconfirm "$package"
}

# Debian
remove() {
    package=$(dpkg --get-selections | awk '{print $1}' | fzf --preview 'apt-cache show {1}') && sudo apt remove --purge "$package"
}

If fills me with no joy to admit that more often than I'd like, I need to debug tests or commands that only fail sometimes. This is always a pain in the ass, but something like this can make it easier:

loop() {
    local cmd=("${@:2}")
    for i in {1.."$1"}; do
        eval "$cmd"
        if [[ $? != 0 ]]; then
            echo "\nCommand failed on run $i"
            return 1
        fi
    done
    echo "\nCommand never failed"
}

Now, I can run loop 5 make flaky_tests once and have make flaky_tests run 5 (or 5000) times, reporting the failed iteration if present.

You can see how this can get out of hand fast. Beware of complexity!

Global aliases

Some pipes and/or redirections are used quite often, but aliasing them won't work as you'd expect.

You can use -g to declare them as global aliases (as in they can be placed anywhere in the command, not just the beginning):

alias -g C="| xclip -sel clipboard" # or wl-copy, pbcopy, ...
alias -g NOER="2> /dev/null"
alias -g NOOUT="> /dev/null 2>&1"
alias -g S="| sort"
alias -g SU="| sort -u"

Now I don't need to remember how to send the output to my clipboard.

Shell setup

There are some other minor tweaks you can do to your shell config to make it nicer. For starters, chose a text editor and set it as default, so you don't unexpectedly get thrown into an unfamiliar environment:

export EDITOR=nvim

Explicitly set your desired terminal emulator so things don't open in some odd default terminal:

export TERMINAL=kitty
export TERM=kitty

Don't ask why you need to set it twice...

Also, you can ensure the command history behaves sensibly:

# Number of commands to store in memory
export HISTSIZE=10000
# Number of commands to store in disk
export SAVEHIST=10000
# Ignore duplicated commands during session
setopt HIST_IGNORE_ALL_DUPS
# Ignore duplicated commands when saving to hist file
setopt HIST_SAVE_NO_DUPS
# Append commands to history file instead of overwriting it
setopt append_history
# Append commands to history file as soon as they are run (instead of when the session ends)
setopt inc_append_history

Key bindings

With zsh, we can assign any function to a key combination by first turning it into a widget:

custom_func() {
  echo "Hello from custom function!"
}
zle -N custom_func
bindkey '^H' custom_func

Here we use zle -N to register a function as a widget, and bindkey to assign it to Ctrl+H. A widget is, keeping it simple, any ZLE (Zsh Line Editor) compatible command, which is what bindkey expects.

There's much more you can do with widgets (this is a great place to fall down a rabbit hole!) and zsh comes with a bunch of useful ones built-in.

There are two in particular that I find super useful when going up and down the command history: up-line-or-beginning-search and down-line-or-beginning-search.

By default, pressing the up and down arrow keys allows you to scroll up/down the command history. These two widgets will scroll based on what you already typed. So if I type e and then the up arrow, only commands starting with e will be shown (so echo would appear but ls would be ignored).

These need to be loaded in memory before being registered, which then allows you to bind them:

autoload -U up-line-or-beginning-search down-line-or-beginning-search
zle -N up-line-or-beginning-search
zle -N down-line-or-beginning-search
bindkey $key[Up] up-line-or-beginning-search
bindkey $key[Down] down-line-or-beginning-search

This binding syntax is slightly different from the previous one. Special keys (arrows, backspace, tab, delete, etc.) are handled this way.

Hopefully this helps you enjoy your time in the command line a bit more!

How to git without hub

Thu, 10 Apr 2025 23:00:00 GMT

How did people use Git collaboratively before GitHub was a thing? The two are often coupled, but some projects that don't rely on GitHub (or any centralized service for that matter) for their git hosting needs. Not just any old project too: the Linux kernel, Debian, Apache, GNU core utils and Golang are some examples of well known projects that handle their git repositories on their own infrastructure.

Ever wondered how they manage? The simplest and most rudimentary way of using Git collaboratively is by sending changes to the maintainer in a file via email.

Let's go over what that workflow might look like.

Where are my changes?

We need a way to get a set of changes (often called change-set) out of your local repository so that they can be sent elsewhere easily.

A commit is represented by a hash or an alias, which wouldn't be very useful by itself in our case. We can however get the underlying changes using diff.

Creating a diff

We can create a diff between two commits with something like git diff <hash-1> <hash-2> and send the results to a file:

git diff <hash-1> <hash-2> > mypatch.diff

You can also use a range of commits or something like HEAD~2 to get the diff for the last 2 commits. If given only one hash, git diff will create a diff between it and your working directory (so git diff HEAD on a clean working directory prints nothing).

This gives us a neat file with the changes we want to send upstream.

Applying the diff

Just run git apply mypatch.diff!

This will apply the changes in the diff to the working directory, but they won't be staged. The maintainer would have to stage them and create a commit to actually add the changes to the source tree.

Shortcomings

So this is great, but there are a couple of glaring issues here:

The original author and metadata of the changes got lost
The original commits all got squashed into one (or re-organized however the maintainer decides)
A person that didn't write the changes (maintainer) got to commit them and appear as the author in the log

This is fine for a quick POC or draft to share during development, but doesn't really scale well.

We need a way to maintain the original commits and their metadata, so the original contributor ends up in the log and their work is merged as provided (assuming no changes are needed from the maintainer).

Creating a patch

A patch is similar to a diff but keeps all the relevant metadata. Since diffs are sometimes colloquially called patches, these can be called formatted patches.

You can create one using git format-patch:

git format-patch -1 <commit-hash> --stdout > my_patch.patch

Here, -1 represents the number of commits before <commit-hash> to be added to the patch. So for a series of commits like:

A -- B -- C -- D

git format-patch -1 C and git format-patch B..C would achieve the same thing: a patch of the changes between B and C. Conversely, git format-patch C would produce a patch of the changes between C and HEAD, which in this case is D.

The --stdout > my_patch.patch bit is just to send the data to a file.

It would be nice if git diff and git format-patch had consistent interfaces, but consistency is not really a strong point of git's UI...

Anyway, if you inspect this file you'll see that it not only contains the changes but also a bunch of metadata about them.

Let's see how an actual formatted patch can be applied (because of course it's not git apply like before...).

Applying the patch

Running git apply on a formatted patch "works" but leaves all that metadata out of the picture, just like before.

So instead, we use git am my_patch.patch (apply mailbox if you're curious).

Now, all the original commits in the patch (with their timestamp and messages) will be added to the tree, with the original contributor/s as the author/s. This of course might create conflicts (although it shouldn't if the patch was created with care, more on that later). These can be fixed like any other merge conflicts, using git am --continue to resume the process.

In this case, the maintainer applying these changes will not appear anywhere in the commits. This might be fine, but for later reference it might be useful to use the --signoff flag. This way, the maintainer will be referenced at the end of the commit/s message/s with something like this:

Signed-off-by: Some One <[email protected]>

Where is my fork button?

There is none, you just clone the repository, work on your local copy and send the patch to the maintainer/s.

Here's the thing: forks are not really a git thing, they are a GitHub ~complication~ abstraction.

When you click that fork button, what essentially happens is that GitHub creates a copy of that repository under your user, with a reference to the original for ease of integration. You would then clone your copy of the repo, work on that, push to your remote and then handle the merge request to upstream through GitHub's UI.

Here, you just clone the original (upstream) repo, work however you want, and send the patch to the maintainer.

GitHub-less Workflow

So here's what the full workflow might look like, from both perspectives:

As contributor

Clone the project and create a branch.

git clone [email protected]:Upstream/Repo.git
git checkout -b cool_branch

Do and commit the work.

git commit -a -m "no idea what i'm doing"

Pull any new changes and rebase master onto your branch. This makes sure your changes don't cause conflicts and are up-to-date with the main branch. The maintainer will thank you if you do this and yell at you if you don't.

git checkout master
git pull
git checkout cool_branch
git rebase master

Create the formatted patch:

git format-patch master --stdout > the_patch.patch

Like we saw before, git will produce a patch of the diff between the head of the current branch and master. This is the data you would see in a GitHub Pull Request.

Send the patch to the maintainer, and you're done!

As maintainer

Get the patch somehow (email, curl, etc.) and apply it to a newly created branch.

git checkout -b dont_trust_that_guy
git am --signoff the_patch.patch

Hope the contributor did a rebase to avoid conflicts, yell at him if he didn't. Review the work and merge it if correct and up to standards.

git branch master
git merge dont_trust_that_guy
git push

Done! Now the contribution is in the main working tree, yay!

WTF do I care?

Why would anybody care to work like this when we have lovely, Microsoft-provided, green "Merge" buttons?

Well, for starters some projects started before GitHub was a thing. Some projects are so big and distributed in nature that the GitHub workflow isn't really fit for purpose (such as the Linux kernel). Some might argue that having the biggest repository of free and/or open source software hosted in Microsoft's servers might not be the brightest idea...

In any case, as a contributor you might not really have a say in this. If you want/need to contribute changes to these kinds of projects you'll have to adapt to how they work.

Apart from that, I just think it's pretty cool to be able to send a quick diff or a patch to a co-worker or a maintainer/contributor without the usual rigmarole of creating a branch, pushing to remote, fighting with the pipeline, etc. It's just a file you can send via Matrix or Slack.

Simplicity has a charm all of its own.

Remote branches and how to handle them

Fri, 28 Mar 2025 00:00:00 GMT

Often enough, the confusions with git arise not when working on local repositories, but when collaborating with others and handling remote ones. In this post, we'll go over some basic concepts and commands to better understand what's going on when we push, pull and merge commits/branches.

Remote origins

So what are remotes? And how come there's more than one?

A remote is simply a place where we can pull from and push to. When you first clone a project, you clone the remote repo to your local machine.

Since git can handle multiple remotes, these are named. By default, the URL from which you cloned a repo is set as the origin remote. We can change that URL with something like git remote set-url origin [email protected]:User/Repo.git.

How is that useful? Well it's useful as a feature when working on Open Source Software, since more often than not you'll have your own remote for the project (a fork) as origin and the actual upstream project as a separate upstream remote. This is done to keep your 'copy' up to date with the 'original'.

More broadly, it's useful to understand these concepts firstly because you'll find references to origin and remotes when looking for information online (including this post), and secondly, because the following commands can be told which remote to operate on.

For simplicity, it will be omitted wherever possible, just know that there's nothing special about origin and that a remote is nothing more than a git server somewhere.

Fetch

To fetch the 'state' of the remote repo, we can run git fetch. This allows other commands like git status or git log to show the full picture, since these only work with local information.

This hints to the fact that whatever git fetches has to be stored locally somehow.

Indeed, just like there is a main branch on a local repo, there's also a origin/main tracking branch. A remote tracking branch's only job is to locally store the state of the corresponding remote branch. You cannot, for example, git checkout to one.

Of course, there are other ways git uses the fetch command, or rather the underlying plumbing. More on that in a bit.

Merge

Pretty explicit: It merges two branches together, specifically the second one into the first. If only given one branch, git merge will integrate the given branch into the current one.

This is the main selling point that made git so popular in the first place and although nowadays, there are plenty of reasons to limit the use of branches (and thus, merges), it's still worth understanding what's happening and how.

Since this is where a bunch of git issues arise, and there are multiple ways git might handle a merge, we'll go over the different ways this might happen (merge strategies) and the implications.

Fast-Forward Merge (ff)

Git's default behavior when possible. Whenever a branch A has newer commits than another branch B, and A needs to get merged into B, these newer commits will be 'copied' or fast-forwarded into B.

For example, given these branches and commits:

main:    A -- B
feature:       ↘-- C -- D

The command git merge main feature would produce the following result:

main:    A -- B -- C -- D
feature:       ↘-- C -- D

Since the new commits on feature 'come from' the last commit on main, they simply get put on top of it.

This is usually the best approach whenever possible, because it doesn't create new commits, conflicts or other complications.

Recursive Merge or Merge Commit

This is the default alternative to ff, and it's also what the green 'Merge' button does on GitHub by default.

In this case, git creates a new commit that points both to the last commit of branch A and to the last commit of branch B.

So for this setup:

main:    A -- B -- C -- D
feature:       ↘-- X -- Y

Doing a fast-forward merge is not possible, so merging feature into main would look like this:

main:    A -- B -- C -- D -- M (Merge commit pointing back to both D and Y)
feature:       ↘-- X -- Y --↗

With this approach, the whole history is preserved and the merge point is marked with its own commit. This is also called three-way merge, because on top of (in this case) commits D and Y, commit B is also involved in the merge as it is the common base for both branches.

Notice how the merge commit has two parent commits, as in it points to two different commits as it's 'previous' one. This is why 'undoing' or reverting a merge commit is a bit more involved than a usual git revert [HASH].

Squash

Another strategy we might use is creating a squash commit, or squashing the changes into a single commit. This is similar to the previous approach, only in this case the commit history will not be preserved like before.

When we do a squash merge, the changes in all the commits of (in this example) feature will be 'compacted' into one new commit in main.

So for the same setup we had before:

main:    A -- B -- C -- D
feature:       ↘-- X -- Y

Running git merge --squash main feature would produce this output:

main:    A -- B -- C -- D -- S (Squash commit containing changes in X and Y)
feature:       ↘-- X -- Y

Be careful when using this: if a big branch with a bunch of commits is squashed this way, and a bug is introduced in one them, it won't be easy to spot which one is the culprit. Remember, there will only be one commit on the main branch after the merge.

Rebase

A rebase is not really a merge strategy, but it's vaguely related and will be relevant further down.

In this case we literally 'change the base' of a (set of) commit/s. That is, we assign them a different parent commit.

So given the same old setup:

main:    A -- B -- C -- D
feature:       ↘-- X -- Y

Running git rebase main feature would rebase feature onto main:

main:    A -- B -- C -- D
feature:                 ↘-- X' -- Y'

So before the rebase we had a X commit with B as it's parent, but after we ended up with a X' commit with D as its parent.

Notice how the commits in the feature branch are actually different now. In git land, commits are immutable. Once we change the parent we change the whole commit. New parent, new hash, new commit.

As mentioned before, this is not a merge: commits X and Y simply got substituted for new commits, they didn't get merged anywhere. This does, however, enable you to do a ff merge onto main, which was not possible before the rebase.

Rebase should be used with caution when working with other people, as these new commits might conflict with other people's work. Generally speaking, this is usually done to update your local branch to the latest commit in main or in a remote. So to keep a local feature branch up to date.

In fact, as a rule of thumb, never rebase branches where other people might be working and never, ever rebase main.

Pull

When we pull changes from a remote, (simplifying things a bit) we are fetching recent commits and merging them into our local branch. This is done using the remote tracking branch we mentioned before to store these new commits, and merging that into the branch we are in.

As such, the different approaches to merge (strategies) also apply here:

--ff fast-forwards local changes onto the new changes coming from the remote
--squash squashes new remote commits into a single commit into the local branch
--rebase rebases local commits onto the remote ones.
If no flags are given it will follow the same defaults as merge, or whatever is configured in the .gitconfig file.

On top of these, the pull command also has --only and --no versions of these flags. This indicates that a pull should either only or never follow a given strategy when pulling changes.

Push

Of course, we can also push our local changes to the remote repo.

What happens when we push a branch is that the remote tracking branch is updated (fetch) and if your local commits can be fast-forwarded onto the tracking branch, these commits (not the whole branch) will be pushed to the remote.

It doesn't really make sense to have multiple strategies here, since anything other than ff is bound to mess with other contributor's work.

We can however, in dire circumstances and hopefully not in the main branch, make a forced push using the --force flag. Be careful with this, since all remote changes will be overridden with whatever is on your local branch.

We also use the push command to delete remote branches once they are no longer useful: git push origin --delete feature-branch

Are IDEs worth it?

Fri, 14 Mar 2025 00:00:00 GMT

Much of the eternal online debate about IDEs and code editors is either out of date, out of context or out of both.

Let's go over some of these misconceptions and some less talked-about points, and evaluate which alternative makes more or less sense and in what context.

Speed

IDEs are slow, or so the trope goes.

While this is most noticeable at startup, this slowness is not necessarily limited to startup times. It makes sense: IDEs are constantly trying to help and make suggestions. This has a compute cost.

Of course, given good enough hardware or simple enough projects, this might not really be an issue. Still, that doesn't change the fact that, ceteris paribus, a code editor is faster.

Some might say that it doesn't matter, that it's not noticeable or that it's more than worth it. Those are some "that's like your opinion dude" kinda points more than anything else.

Whether you notice it or not is a matter of how sensitive you are to things like input lag or what you consider fast enough. Whether it's worth it or not depends on a bunch of different things, like the stack you are working with, the particular IDE you are using or how close you can get by adding plugins to a code editor.

In other words: it's context dependent and up to you if this is an issue or not.

LSP

Traditionally, the only way to get decent code completion and navigation was using an IDE. In that sense, the developer experience used to be much, much better for IDEs than for code editors.

Since LSP came out however, this is no longer the case. Long story short, the Language Server Protocol standardizes how an editor or IDE interacts with a language, which enables any LSP-capable editor to perform much like an IDE.

Newer languages have official LSPs available on day one and even older ones often have unofficial alternatives made by the community. To be clear, not all languages benefit from this. Obvious examples are Java and C#: good luck using them without Visual Studio or IntelliJ.

This used to be a big difference between using an IDE or not, but nowadays, its more a matter of what else can an IDE provide.

Distractions

Some of us need a distraction-free environment.

First thing I do after installing an app or subscribing to a service is disabling all non-critical notifications. I don't want to deal with useless popups, sounds or flashing icons.

Clearly, IDEs are not made for people that have trouble focusing.

An IDE is always trying to help the user, which means bringing stuff to your attention. Constantly. This is great if you need the help and can handle the distraction, but absolutely unbearable if you don't.

You can of course tone all of this down, just keep in mind that you are swimming against the tide: the IDE doesn't really want to shut up.

Learning

As the name suggests, IDEs integrate everything you needTM under a single interface.

It's super convenient, but has the unexpected side effect of allowing you to treat all of those "things you need" as a black box.

You abstract away complexity, but lose the understanding of the underlying systems.

I've personally met and worked with people that only know how to interact with a database through their IDEs UI. Not an IDE, their IDE. The same goes for docker, running tests and, worse of all, running the code itself.

Let me repeat, some professional developers can only run the code they write through the green "Play" button of their IDE. Talk about vendor lock-in.

Now, this is a human issue, not an IDE issue. People that behave like that tend to have the same exact problem with all the software they use. Still, humans gravitate towards comfort and more often than not the (apparently) easier path will be chosen. Even if that means settling for ignorance.

People that have only ever driven automatic are unlikely to learn how to drive a manual out of curiosity, even though they might benefit from the knowledge at some point.

This might be a worthwhile trade-off, or it might not. But it is a trade-off, and we should be aware of it.

Death by feature

Code editors are usually pretty bare-bones out of the box: the user is supposed to add features as needed. This usually results in a (sometimes unreasonably long) list of plugins. The idea here is to end up with the features needed by the user and nothing else. Keep things relatively simple.

Conversely, IDEs come with all the features: the ones you need and the ones you don't. This of course is far removed from the good old "do one thing and do it well" UNIX principle, and may introduce unexpected complexity or interactions under the hood. The point is for the user to find everything he might need right from the get-go.

If you use a code editor expecting to not have to configure anything up front, you are going to have a bad time. If you use an IDE expecting it not to pollute your dev environment with opinionated solutions, you are going to have a bad time.

IDEs actual use case

This all seems pretty negative towards IDEs, but there are a couple of very key areas where they shine so much more than code editors it's not even fair.

Refactoring

This is not necessarily true for every IDE out there, but once you learn to refactor code using any of JetBrains IDEs, there is absolutely no turning back.

The amount of manual labor it eliminates, of refactors you wouldn't even dare to do by hand, the time it saves: there's simply no comparison with any code editor. Yes, there are plugins that somehow try to mimic these features. But it's not even the same league.

There's just no way around it: If you do frequent heavy refactoring, right now there is nothing that even comes close.

Intelligent analysis and suggestions

The counterpart of all that annoying slowness: most of the time, it feels like the IDE is reading your mind. It just knows much more about the project than what is currently possible through the LSP.

Yes you can get good suggestions in any LSP-capable code editor, but a decent IDE will suggest stuff like variable names so cleverly you'd think your computer is actually doing your job. The suggestions regarding code structure, repetition, standards and general best practices are just not there for code editors. Yes, you can use linters, but it's just another level of awareness.

Mind you, this was the case even before the AI craze.

IDEs as a crutch

The less you know about a stack or project, the more useful an IDE is. Which is not to say that senior devs shouldn't use them or they are "for noobs", far from it.

This is precisely why we love them so much in consulting: we often get thrown head first into new projects with god knows what weird or ancient stack, using languages we are not super familiar with, while expecting us to refactor, stabilize or improve the codebase. Using an IDE allows me to focus on the system first, and leave the stack/language details for later. It fills any gaps in my knowledge and allows me to start being helpful faster.

This might hurt some people's ego or pride. You should leave those out of your workplace.

Use both

For whatever reason, this issue is often presented as a false dichotomy (as with most other online debates): either use a fully featured IDE for everything or a dementedly minimalist setup with barely any software apart from a terminal emulator.

This is a bit silly, there's no reason why you shouldn't use both. Choose the best tool for the job, don't fall in love with software.

You don't need an IDE to edit a bash script or a yaml file. You don't want to work on a legacy java codebase with a text editor.

Selfish reasons to engage in Open Source

Fri, 07 Feb 2025 00:00:00 GMT

There are many reasons to use, make or contribute to OSS. While the ethical and societal benefits might be enough for some, I'd like to argue there are also purely selfish reasons for a developer to get involved.

Indeed, just like running Linux in your workstation deepens your knowledge of operating systems and broadens your tool-set basically for free, so too can OSS make you a better developer pretty much by osmosis.

Here are a bunch of ways this osmosis happens.

Other people's code

Contrary to what I thought when starting out, the bulk of our work consists of navigating, understanding and modifying other people's code. And yes, "other people's code" includes the code you wrote a year ago.

This skill can hardly be trained or practiced as it should: you can type faster, practice problem-solving or read a bunch of books, but nothing trains you to deal with systems you didn't write. In OSS however you'll be doing this all the time.

Very different people from different walks of life contribute to OSS, I've seen ways of writing and structuring code that I could have never imagined (both good and bad). Participating in this space will allow you to improve this skill set.

You might think something like "no, thanks, I get plenty of that at work", but is that really the case? Mostly, I see people getting accustomed to their teammates code, creating a sort of cesspool of the lowest common denominator. At work, we learn to navigate a specific codebase, written by specific people, instead of learning the skill of code navigation more broadly. This is a shame.

Documentation

Maybe I've just been very lucky (or unlucky, depending on how you look at it), but I have never, not even once, seen documentation half as good in proprietary projects as I have in OSS projects. Both dev-facing and user-facing.

This not only means that the latter ones are a joy to work with by comparison, but also that I have a chance to learn what makes documentation good and how to write it myself.

There is no asking around co-workers to understand how to build the project, what ritual to follow when modifying the system (if any) or the general structure of the code base. Any reasonably complex OSS project that actually expects and welcomes contributions will at least have a CONTRIBUTING.md file to document all these issues and many more. We should normalize this for proprietary projects as well.

The same goes for user-facing documentation: where companies have on-boarding teams and an endless supply of outdated docs that often force users to open support tickets, OSS projects have a compact, to-the-point README.md for basic instructions and very little incentive to offer free support (which means the docs are usually kept up to date and users are expected to actually read them).

Sure, on-boarding teams often serve a more service-oriented purpose. Something like "tell me what to do, I don't want to waste my time reading your docs". Still, concise, to the point documentation is often just as, if not more, valuable than having someone to directly ask for help.

Troubleshooting

There are plenty of ways to help OSS projects, well written bug reports are one of them, and they don't get the respect they deserve. A good bug report can literally be used as a test case. Writing a decent, well-structured bug report is non-trivial and can teach a lot about the troubleshooting process in general.

It is common for projects to have some sort of template or required information for the bug report. Something like versions of relevant software, configurations, expected behavior and minimum reproducible setup.

Not only can one learn a lot by paying attention to what information they require, but the act of gathering it and writing the report usually already starts the troubleshooting process for the maintainer.

I'm not sure how to describe it, but this process trains a sort of mind-set that can save a lot of time when things go wrong and help you get to the crux of the issue instead of fumbling around blindly.

Soft skills

These are often average at best in developers and understandably so: they are only indirectly related to writing code and are quite difficult to train. They are however really important and often make the difference between a good dev and a great one. OSS can be used as a playground to improve them, including but not limited to:

Communication

When participating in OSS projects, you will receive criticism. This is needed to keep the projects afloat and while it often comes with care and good intentions, this is not always the case. This is a great chance to learn how to handle and respond to it, with the added benefit of not putting your job at risk.

Giving and receiving constructive feedback during code reviews hones your ability to communicate professionally and respectfully.

You will also have to communicate with clarity, since there is no "let's quickly hop on a call" when talking to strangers on the internet. At least not usually. Writing issues, participating in discussions, writing or improving documentation: these all force you to articulate complex ideas clearly and succinctly, including all necessary information and absolutely no unneeded padding.

Interpersonal

More often than not, you'll have to adapt to the preferences of either the maintainer of a project or the wider community. This might not seem like a great thing, but learning to disagree and commit is rather important especially in a business setting, as the alternative often consists of endless, bitter discussions with teammates and co-workers.

It takes some empathy to understand that people you'll never meet might have the same general goals and good intentions you have, while disagreeing with this or that particular point. It takes some more empathy to not get mad at a user telling you that the default behavior of your software makes no sense, or that a given feature is actually a bug. Sure it's frustrating, but how often do we have the luxury of talking directly with the end-users of our software?

Another source of interpersonal friction is a lack of cultural awareness. It's easy to forget that you might be interpreting as rude something that is totally respectful for a different culture. It's even harder to think of a better environment to get used to this than OSS, short of a language exchange bar.

Sure, communication is mainly in English (which already cuts out non-English speakers), but you'd be surprised how many of them are not using their first language. Open any issue tracker or online discussion, you might be reading a German dev responding to a Chinese user. This matters a lot and changes one's approach to a conversation entirely. Rarely have I been in a work environment with comparable diversity.

Better ssh client setup

Fri, 31 Jan 2025 00:00:00 GMT

You probably already know how to create and register an ssh key pair.

ssh-keygen -t ed25519 -C "[email protected]"
# Press enter a bunch of times
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

Simple enough. You can then copy the pub key into [insert relevant UI here] or just send it to your sever with ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server.

However, when using more than one service or handling more than one server, it might be best to use different keys.

Why use multiple ssh keys

On one hand, it limits the impact of a compromised key. If a machine gets compromised, but the ssh keys it used only granted access to a limited set of services, one can rest assured that the other services are probably fine. This is especially relevant when separating work from personal keys.

It's also useful for revocation. You can cut access to one key/machine without affecting others. Or set the ssh key for an untrustworthy machine to expire on a short cycle.

Fine, you create multiple ssh keys, and now find yourself having to run your ssh commands like ssh -i ~/.ssh/key user@server or even worse, having to ssh-add ~/.ssh/key before managing your remote git repo.

This kinda sucks, but there is a better solution.

Config file

You might not be aware that ssh will look for a client config under ~/.ssh/config. The file follows this basic structure:

Host [address or alias]
    IdentityFile [path to the ssh key]

You can have as many of these as you want. The file is read up-to-down and the first match is used, so the least specific sections should be at the bottom.

This config for example, would use the first section for that IP, while using the wildcard at the end for all other ssh connections:

Host 192.168.1.69
    IdentityFile ~/.ssh/nice_key

Host *
    IdentityFile ~/.ssh/other_key

Handling multiple services and keys

In practice, you might end up with a config that looks like this:

Host github.com
    IdentityFile ~/.ssh/github

Host gitlab.com
    IdentityFile ~/.ssh/gitlab

Host vps
    HostName 209.85.231.104
    User vps_usr

Note here that, since a HostName is provided, the Host in the last section acts as the alias one would use to refer to that service. So for that section, you would run ssh vps instead of ssh [email protected], much less verbose.

Since the first two sections will be used by git, adding a HostName and a User doesn't make much sense. For starters, git will always default to the git user, no need to explicitly set that. Plus when working on GitHub/GitLab hosted repos, one would clone something like [email protected]:ORG/REPO.git, so the Host would always be the same as the HostName, thus making it redundant.

Of course, other options can be set in the config file, like Port or ConnectTimeout. But there are more clever things that can be done.

Advanced options

There are, of course, many more options than these. These are just used to show the usefulness of an ssh config file.

Proxy Jump

Depending on the security requirements of an organization, a ProxyJump might be needed when connecting to a server.

This simply means that the outgoing connection must go through a dedicated server before being redirected to final one, which might for example not be exposed to the internet. To do this, one would ssh -J jump_host target_host.

As you might imagine, each might have different settings, so that command is bound to get messy. You could create a shell alias, or you could use the ssh config:

Host target_host
    HostName target_host.com
    User target_user
    IdentityFile ~/.ssh/target_key
    ProxyJump jump_host
Host jump_host
    HostName jump_host.com
    User jump_user
    IdentityFile ~/.ssh/jump_key

This allows the ssh command to look like ssh target_host, no need to worry about who jumps where and with which credentials.

Port Forwarding

Another security-related config is ssh tunneling or port forwarding.

This is done with something like:

ssh -L localhost:3000:localhost:3306 example.com

Which means 'please take all traffic going to localhost:3000 and send it to example.com on port 3306' The second localhost is from the perspective of the remote server.

This might seem a bit silly. Why not just point directly to example.com:3306? The interesting bit here is that the traffic is being sent through the ssh connection (so port 22 by default). The server would receive the traffic on :22 and re-route it to :3306.

This might be interesting not only to reduce the number of exposed ports in a server, but also to ensure cryptographic security. There's no need to use SSL/TLS here, OpenSSH is plenty secure and comes for free with no work needed on either side of the communication.

Of course, one could route more than one ports, and point to more than the remote "localhost", with a designated user, etc.:

ssh -L localhost:8000:[IP_ADDRESS]:8000 -L localhost:8001:localhost:8001 [email protected]

This is not exactly easy to type, but a config like this might help:

Host what_a_mess
    HostName example.com
    User username
    LocalForward localhost:8000 [IP_ADDRESS]:8000
    LocalForward localhost:8001 localhost:8001

One would only need to run ssh what_a_mess.

As you might tell, LocalForward implies there is also a RemoteForward, which indeed is used to send traffic the other way around (from the server to the client).

Just be careful when committing this config to your dotfiles repo: make sure no sensitive information is public!

A different approach to AI code assistants

Sun, 04 Jun 2023 10:57:51 GMT

AI code-generation might usher a future where TDD becomes a strict requirement for software development. In that future, code quality will only be relevant for tests.

Let's do a thought experiment.

Hallucinations

AI generated code kinda sucks.

It's often messy, buggy, over-complicated or just simply incorrect. You need to be super careful with it, double check every line it produces. Troubleshooting bugs is a huge pain with generated code.

This is a shame because the efficiency is undeniable: I doubt that the average dev can produce code at a similar speed that your Copilot/GPT thing can.

Of course, speed is not everything, but we all want to be more productive.

How can we take advantage of the AI's ability to generate code quickly while ensuring the code actually works, and does so as expected/required?

If only there was an approach to software development that could guarantee the behavior of a piece of code...

Testing

What if we add AI to the TDD cycle?

I write a little test, the AI produces code to make it pass.

I write another little test, the AI makes both pass, either adapting current code or generating new one.

This way, there's at leas one thing we could always be certain of: the generated code makes the tests pass.

Given enough quality tests, this would allow us to ensure the system's behavior.

This is not far away from conventional TDD: You just don't write (or read) production code. You treat it as a black box and only focus on the tests.

Evolution

Take two equally capable developers: Peter and John. Similar experience, similar skills.

Peter uses the previously described TDD+AI approach while John doesn't.

While only writing tests, Peter would produce working software that is guaranteed to behave as expected (so far as the tests correctly describe its behavior). This obviously can be done in less time than the alternative, he is only writing tests.

John faces a difficult choice: either use AI and embrace its quirks or avoid it altogether. While the former might seem faster at first, he could soon find himself spending more time fiddling around with generated code than actually writing it.

Avoiding AI altogether will be significantly slower than Peter's with the possible exception of just not writing tests at all, which wouldn't be a fair comparison and has plenty of other downsides.

Which one seems more productive/employable? Remember, we are considering two equally capable devs.

Neglect the black box

As you might imagine, the proposed approach would imply a near-total neglect of production code.

This is a big jump from our current notions of clean code or maintainability.

If you can produce code in a matter of seconds, does it really matter if it's easy to understand and modify? Wouldn't you just tell the AI to rewrite the thing if your requirements change?

Remember, you'll have your test suite to ensure no current behavior is lost. Just add more tests for the new behavior and the bugs you need to fix.

We wouldn't be spending much time (if any) with production code at all: that will be the AI's territory.

We would mostly work with the tests, using them to ensure the AI behaves correctly and doesn't make stuff up.

This doesn't make clean code or maintainability concepts obsolete. Rather, they will find their place within the tests. Those notions where meant for us humans anyway, not for the machine. If we focus our efforts on the tests, they'll come along for the ride.

Grim future? Depends on how you feel about TDD I guess ¯\_(ツ)_/¯.

Diagnosing Digital Patients

Fri, 21 Apr 2023 08:03:27 GMT

Unless building a greenfield project, devs spend a lot of time troubleshooting buggy systems.

I find it odd that we seem to have no method to this madness, no procedures, no nothing. Just smash your head into the keyboard until something clicks.

Medical professionals have to 'troubleshoot people' all the time, maybe they know what they're doing.

Gather data

There's not always a sensible bug report to start with.

Pinpoint the issue

People often don't realize the full extent of their symptoms. They just know it kinda hurts around here sometimes.

Good, sensible questions need to be asked to get a full picture of the unexpected behavior.

What exactly is not working? Does it fail all the time? How does it fail exactly? What's the expected behavior?

Get to know the system, understand the failure.

Clinical History

Look at the context surrounding the error, problems don't come out of nowhere.

When does it happen? What makes it fail? What happened before it started? Can you find a pattern?

If a system hasn't changed recently and a bug was 'introduced yesterday', either the user is the bug, or it's been there for a while.

Physical Exam

Well, digital really but you get the point.

Once a general understanding of the behavior and context is reached, try to go deeper.

'It hurts when I…'

Get your user to reproduce the bug for you.

Yes, this is not always possible. But patient and therapist should be on the same page. Maybe it's not a bug but a missing feature.

What input(s) causes the unintended behavior? How do we get it to happen consistently?

This aims at a low level, I/O approach to reproduce the issue. Reason about the bug like if you were to write a test around it (which you might actually want to do).

If you can reproduce it consistently, you'll fix it eventually.

Does this hurt?

The 'prod it with a stick' part of the process.

Does Y seem to make it any better? Does it also break if you X? What makes it worse?

This might come off as a bit sadistic (and sometimes it is), but it's analysis by I/O: Give the system a bunch of different inputs and see how it affects the output.

What if you press this button/use that plugin instead?

Fear no consequence, break the thing: Software (unlike people) can be rolled back.

Tests and Data Analysis

There's no MRI for software, but we do have logs, metrics, user data, observability, etc.

If they are not present in the system, yesterday is a good time to add them. There is never too much information, you can always filter out irrelevant data.

While test results are a very important part of any objective analysis, they should not be the only base for a diagnosis. Use this data to complement the information gathered in the previous steps.

Make a bet

Gathering data is alright, but how does one actually reach a diagnosis? In any reasonably complex system, it's hard or impossible to actually know what is happening e2e. There are often unknowns, black boxes we don't fully understand.

Even so, we can do better than guessing.

Pattern recognition

If you feel tired, your nose is running, and you have a fever, it doesn't take a rocket scientist to bet on you having the Flu.

If you updated your Nvidia drivers yesterday, and today you got a black screen on boot, your OS is probably fine, the drivers are likely broken or incompatible.

This doesn't mean there cannot be any other issue, it's just so likely to be the cause that focusing on any other possibility as a first guess makes no sense.

Of course, this requires some experience: you can probably only recognize these pattern if they are not new to you.

Differential diagnosis

It involves finding all possible causes and eliminating them one by one, leaving only the (most likely) root cause.

A PC may not boot for a bunch of different reasons, but if you can hear the fans spinning and see some lights turn on, you can eliminate the power supply as one of them.

This can be, especially with software, a long and tedious process. But is accessible with or without previous experience, and allows you to be methodical in the process.

Treat the damn issue

Medical and IT professionals both face a critical choice: Either find the root cause and treat it, or simply treat the symptoms.

In medical fields, the latter is exclusively reserved for three scenarios:

There is no treatment, so best we can do is alleviate the symptoms.
The treatment is unavailable/unaffordable.
The system is overstretched, and we lack the time/resources to diagnose and/or treat properly.

Unfortunately, in the IT space, treating the symptom is a near-ubiquitous practice and the third scenario seems to be the norm.

We should keep in mind that software serves the needs of people, and even if users are not visible, they are still affected by inadequate diagnosis and treatment.

Would we behave the same way if the user was sitting by your side? What if the software was used by medical professionals? Are we considering the impact that software has in the lives of people and the choices they make?

We should make sure there is a valid reason not to diagnose and treat the root cause.

Follow-up

Once a diagnosis is reached, and a treatment is prescribed, follow-up appointments are scheduled.

This is done to confirm that the diagnosis was correct, the treatment is effective and that there are no unwanted surprises or further actions needed.

Try to reproduce the bug, press the same buttons as before, stress the system.

The issue is not fixed until proven so.

Myths and clichés

Sun, 02 Apr 2023 15:22:31 GMT

Conventional wisdom can go a long way and is often a useful guide. However, it is usually best served with a healthy dose of skepticism and scrutiny.

Here, we explore the good, the bad and the ugly behind some common clichés I see floating around.

TDD

Probably the most misunderstood of the bunch.

Requires clairvoyance

A simple (but unfortunately common) reading of TDD lead some to believe that all tests need to be written before any of the relevant production code is.

This would require developers to plan ahead in some sort of whiteboard, which of course is usually a bit silly, since we learn about a problem domain as we build software around it.

Does it though?

Let's take a module or a Class for example: what part of TDD dictates that all tests for that module should be written as step 0?

Is it not TDD if I write a test for one of the functions, write that function and keep going bit by bit?

TDD is not a way of planning out your code with boxes on a whiteboard, quite the contrary: it incentivizes you to design as you go, to think about the public facing design of your code even before you think about the code itself.

Its usefulness comes in part from the fact that it helps detect and resolve unknowns before writing a piece of code, where they can cause issues.

Testing

Makes no sense in an MVP

An MVP needs to be built fast and only has to prototype the behavior of the system in a narrow scope.

Edge cases are often (purposefully) overlooked, so testing would slow down the process without adding much.

Who cares? It's just an MVP anyway, it will get re-written if it works.

MVPs are forever

MVPs often turn into the (nearly incomprehensible) core of Legacy projects that have to be maintained 20 years down the line. There is always 'not enough time' to re-write them.

What would you rather do: write tests as part of the process or convince the Business team to stop new developments for a while to test what has already been "proven" to work in production?

You don't need 100% test coverage here (or ever). Just make sure your code is testable to begin with. Further testing can come later down the line.

CI/CD

Means having a pipeline

It's not uncommon to come across teams that think having a pipeline as part of their workflow qualifies as Continuous Development.

Somehow, the principle and the tool got mushed together.

You could make-do without a pipeline

Assuming you wanted to make this as inefficient and unreliable as possible, you could argue that Continuous Integration can be achieved without a pipeline.

After all, it's about integrating your code with everybody else's as frequently as reasonably possible: just push to master, see what happens.

You could also pay a poor soul to continuously hit the big red 'Deploy' button all day long as well, no need for a pipeline.

The point of having a pipeline is to act as the gatekeeper for code quality, reliability, performance, stability, etc. when sending it to production.

This is the only sane way I can think of doing CI/CD, but pipelines are a tool, not a set of practices.

You can have the most over-engineered pipeline in the world, if you deploy and merge branches once a week well... I have bad news for you.

Automated Deployment is Continuous Deployment

Some think of CD as the absence of manual deployments.

Like with pipelines, automated deployments are the only sane way to do CD.

But if you really want to suffer, nobody is stopping you from manually producing and uploading all required artifacts by hand.

Don't miss the forest for the trees.

Clean Code

Horrible performance

The "clean" code rules were developed because someone thought they would produce more maintainable codebases.

Even if that were true, you'd have to ask, "At what cost?"

<cite>Casey Muratori^1</cite>

Let's not dwell on the fact that the generation of developers that basically founded this industry are not someone, and rather focus on the claim regarding the cost.

Nowhere in the book does the author advocate for clean code at all costs.

Quite the contrary, it is mentioned several times that code efficiency must be taken into account:

I will avoid [the more efficient solution] if the [efficiency] cost is small.

The claim about performance is either missing the point entirely, based on a woefully misguided reading of the book or simply clickbait.

No one is advocating for a complete disregard to performance.

Rather, it is suggested that you should write your code with other people in mind.

Eventually, someone will have to maintain your code: make sure there is a good reason to make it hard to work with.

The reality is that in a lot (if not most) contexts, performance is far down the list of things to worry about.

In most cases, the real world, practical performance differences between clean code and whatever the alternative is (performance-focused code?) are far outweighed by the maintainability of the former.

As with most things: it's a trade-off.

Your code doesn't need to be text-book-clean, but you should aspire to keep it reasonably clean considering the circumstances.

It's in the eye of the beholder

One might argue that what's clean to one person might not be clean to another.

Advanced programmers might find easy to read code that seem incomprehensible for beginners. Each language has different idioms.

This might indicate that clean code is too relative a thing to be of any help.

It not always is

On the one hand yes, what is or isn't clean/readable depends on the context.

Your team might be used to working with 20000+ LOC files. They might find this normal and desirable.

This is not the end of the world: if it works for the team then it's all good.

On the other hand, not everything is relative.

Calling a variable x is objectively less clear than giving it a decently descriptive name.

Writing your code in a way that can be understood by a beginner, someone coming from a different background/language, or your future self, has clear and obvious advantages.

Using you language's latest super-fancy, concise, ultra-functional gimmick is often less about clean code and more about showing off.

There is a fine line between taking advantage of a given language's features, and gate-keeping the codebase to only those 'smart/versed enough' to follow it.

Clean code doesn't look the same in all projects/teams/contexts, but that's not an excuse to disregard best practices and write code like you want to.

Linux

Is free if you don't value your time

Some find Linux (Desktop) to require too much time to set up/configure/maintain to be worth the effort.

To them, all possible gains from using Linux are offset by the amount of time and attention it requires.

In fairness, Linux can be as much of a time sink as you want it to. That being said, nothing prevents the use of a ready-to-use distribution. These come already set up/configured and require little to no maintenance.

Things don't actually break for no reason (except when updating Windows).

Since this is quite obvious, a more charitable reading of the claim might be something like: "The amount of stuff you have to learn is not worth the effort."

Is it really not worth it?

Learning a new anything, a new OS in this case, implies... A learning process, which quires time and might be frustrating.

Linux being FOSS furthers the amount of learning required, since we all come from using proprietary software and things are quite different here.

If you are not interested in learning a new skill set... Don't. Stick to what you already know.

If instead you are, this can be a lot of fun.

And if you are a developer, I cannot stress enough the amount of times I have been able to solve a problem or help a co-worker just by virtue of having a deeper understanding of how an OS actually functions.

If you come with an open mind, this stuff makes you a better dev. For free.

Breaks all the time

Some consider Linux Desktop to be unstable.

I'm still not sure what makes it a reliable server, but an unstable client. Maybe someone will point it out eventually.

Still less of a headache

I've been personally working from a rolling release distribution, widely considered unstable and breakage-prone for more than 4 years ATOW.

In this time, it broke twice:

The first one as a consequence of me doing things I didn't understand as sudo.
The other one due to a combination of my being silly and upstream changes (a borked update that should have been easy to recover from).

In contrast, I've lost count of how many times I've had to 'unfuck' my Windows machine after an update.

MacOS has been less of a headache in that regard, but in that land you either do it 'The Apple Way' or you don't. I like it my way, thank you very much.

There is a kernel of truth though: Linux allows the user to break things, while the alternatives usually limit what the user can do so much that the only reasonable way something breaks is if they (Microsoft, Apple) break it.

This is easier on the user's ego because it makes him inherently blameless. The system likely 'breaks less' because the user is out of the equation.

Plus, what's so terrible about breaking things? It's fun and the best way to learn!

Just have a backup and you'll be fine.

TBD in action

Sun, 05 Mar 2023 16:41:12 GMT

As mentioned previously, you can be as lax as needed when adapting TBD to a team's workflow.

We'll go over do's, don'ts, and how-to's to help adjust this approach to software development for each context.

Follow the principles, not the rules

Branch wisely

While the point of TBD is obviously to only work on one main branch, this is an ideal that some teams strive for, but might be out of reach when starting out.

Start by ensuring no branch lives for more than a day, the shorter-lived the branch, the better.

Crucially, remove all long-lived branches that run in parallel to the main one.

Don't branch out of habit, do it if/when you actually need to. A POC might be a good example of a valid reason to branch.

Distrust PRs

While you work your way to a branch-less workflow, PRs are still going to happen.

Some restrictions might be worth considering:

The pipeline should define the standards for code validity, quality and security. Not the PR.
Reviews are optional: Reviewers shouldn't prevent code from reaching production, reviews should be requested by the submitting dev if needed (and should be done synchronously if possible).
Use PRs as tools, not rituals: You might feel more confident blocking all pushes to the main branch and having the pipeline run on merge. This is little more than implementation detail, and is fine as long as it doesn't interfere with the workflow.
Keep them small: The easier they are to revert, the better. Prefer multiple small PRs over one big PR per user story.
Consider whether pair programming is a better alternative than a PR.

In general, try to see PRs as little more than 'a thing that happens' semi-automatically when pushing commits.

Stay in sync

The less time you spend away from your main branch, the better.

Constantly ask yourself: Could this be merged?

Doesn't matter if the feature is done or if the bug is fully fixed. If the answer is yes (as in "tests pass, code compiles and doesn't break prod"), do it.

Even if using branches: merge with master, open a new branch and keep going. Make this a normal part of your workflow.

PRs and branches should end up feeling more like a chore than anything else.

Deploy whenever

The more, the merrier.

Keep your code deployable while you work, don't break the build, keep your tests green.

Test yourself and your team by deploying at least once a day. See if your code really is "always in a releasable state".

Automatically deploying every commit might be a bit much to begin with, but the closer you get, the faster you gather user feedback, the faster you can make informed decisions.

For the bold and brave: have a pipeline that automatically deploys every evening/morning. You might be surprised how much that can change how you work.

Work in small steps

Commit code frequently, multiple times per hour. Doesn't matter if the code is not perfect or if it's a "Work In Progress".

Wrote a test? Commit. Made it pass? Commit. Made it compile? Commit. Refactored a module? Commit.

If it compiles and passes the test suite it's good to go.

Reverts are easy when working in small increments, trust your VCS, think of commits as checkpoints.

Must-haves

Of course, there are some technical must-haves to make this work. You might not be able to just take your existing codebase, and go for a TBD workflow.

Here are some things to consider.

Pipeline

You need a solid, cared for, efficient and stable pipeline.

This should be a primary focus of the team: Issues with the setup (build, tests, containers, pipeline, etc.) should be resolved immediately.

Ideally it would take care of building, testing, code analysis, security tests, deploying to production and any other task that can possibly be automated.

The pipeline should be fast and efficient. Builds and tests should run as fast as possible, ideally in parallel.

Only changed code should be built, and only relevant tests should run. Of course this requires enough modularity to make this viable: you can't only run the tests for module A if you expect the changes to affect other parts of the system.

Cache your dependencies, optimize anything that comes to mind. Every minute wasted here will add up really fast.

Fast builds and tests

You need to have a comprehensive and meaningful suite of automated tests, mostly unit tests with a more selective approach to e2e and integration tests.

These need to be fast and reliable and the team should trust them enough to consider the code deployable as soon as it passes them. They should be the judge of what is or isn't production ready.

Ideally, building the project and running the tests shouldn't take more than a few minutes from start to finish. If there are tests or builds that take longer than the rest, isolate them.

Fuzz tests for example might run after the code is deployed or in parallel to it, slow builds might be avoided for patches that don't involve that specific part of the system.

Locally reproducible

When doing following this line of work, braking trunk slows down the rest of the team.

Since mistakes will inevitably happen, ensure the system can be quickly and fully built and tested locally. This should be done regularly before pushing changes.

Tests that "only work in Jenkins", flaky tests, or slow/complicated builds incentivize devs to run very little checks locally. This might not be out of carelessness, they might trust the pipeline so much that they count on it spotting errors, they might think it's not a smart use of their time (why bother if the pipeline is going to do the same thing?).

This is a good thing, but not good enough to slow everyone else down. Make sure these things are not a chore, but a quick check one does without even thinking about it.

Fine-grained deploys

Ideally, especially with monolithic applications, one wouldn't need to re-deploy the whole thing. Rather, deploys should only involve the parts of the system that have been updated.

This is easy enough when working with microservices (if done right), but can be challenging with monolithic systems.

Modularize your code in a way that allows for partial deploys. Ideally, the selection of which part to deploy would be automatic based on the git diff, but human selection might be a good or even better idea depending on your system.

If for example the codebase is fragile and changes in one place are bound to affect other places, automatically selecting which piece to deploy might be a bad idea, while a human might have the context needed to make that decision.

Tips and tricks

When coming from a branch based workflow, it is likely unclear how exactly to make changes without breaking things.

There are multiple tricks you can use to protect the system from your code:

Feature Flags

A feature flag is a way to hide a functionality or a piece of code unless certain criteria is met.

What these criteria are is up to you and context dependent. Feature flags can be as simple as "only available to user X" and complex enough to require a purpose build solution just to manage them.

// old code
if (user.flags.newFeature || user.email === "[email protected]") {
    // cool new feature!
}
// old code

This allows you to easily "turn your code off and on" for one or more users, handle Betas or simply to manually test out the code in production.

It also has the added benefit of allowing work in progress code to live in production without affecting the application or the users in the slightest.

On top of that, it can pave the way for A/B testing. This can be enough of a reason to implement feature flags on its own.

As you can imagine, there is much more to feature flags. You can learn more here.

Branch by abstraction

When making changes to a piece of code that other developers or teams depend on, branching from that code by abstracting the API is very helpful.

To use a simple example: If changes need to be done in function_foo() but someone else is using it, extracting N functions from it can allow for easy swapping of the parts that need work or a new implementation, without having to go for more invasive approaches, like changing the usages of the original function to another wip_function_foo().

This might seem needlessly complex for functions, classes or interfaces/traits, but in complex systems and/or big enough changes we might be talking about whole modules. Even changing a function signature might entail an unmanageable amount of merge conflicts.

You can think of this as a type of Parallel Change, although on top of allowing you to keep the tests passing, it allows other team members to keep working uninterrupted.

Dark Launches & Canary Releases

Dark launches simply refer to releases that are hidden and only made visible/usable for a subset of users.

Similarly, canary releases are only meant to go out to a select group of users.

The former is used when all users necessarily run the same version of the software (a web app, SaaS, etc.) while the second might make more sense in the opposite case (a phone or desktop app). Both have the same purpose and hold the same value.

The point here is to only give access to the feature to a predefined group of trusted users (or a small percentage of the total user base), usually with the help of feature flags.

By doing so, you can see a feature in action (not only in production but in use by actual users), gather feedback, evaluate how it performs and decide if a full-scale release makes sense or more work needs to be done.

Push to prod, do it often

Sat, 04 Mar 2023 12:00:13 GMT

Software development might mean different things to different people, but at the end of the day the whole point is to satisfy the needs of its users, whichever they might be.

Think of it in that light, and some gaps will appear in the way those needs are conveyed:

What the user actually wants != what he thinks he wants.
What the user thinks he wants != what the business team thinks he wants.
What the business team requests != what ends up as a Requirement or User Story for the dev team.

This is unfortunate, but apart from trying to get in touch directly with the user, there's little we can do as devs to close that gap.

What's maybe more relevant to us is the gap that runs on the other side of the equation, between code being written and the user actually using it (the way the needs are met):

Write the code
Make a PR
Wait for the reviewer and discuss the code
Deploy to a staging env
Wait for QA to validate it
Mark as ready to release
Wait for the release cycle/window
... Is the user still there?
Fuck I pushed a bug to prod -.-U
Hotfix? Rollback and goto to step one?

All this with the accompanying mess of branches, merges and possible conflicts with other co-workers or teams.

In an ideal world, the user would tell us what he needs directly and clearly, peek over our shoulders while we code, understand what we are writing, and let us know if we are on the right track.

Sadly, this world is not ideal. But there is a lot we can do to shorten the time between writing code and receiving the users feedback on it.

Trunk Based Development is one approach we might use to achieve this.

What even is TBD?

Trunk Based Development is, in a nutshell, the practice of writing software avoiding branches as much as possible, streamlining the development in one main "Trunk" branch, also used for deploys, QA and the likes.

The fundamental importance of this approach is in recognizing that the main "Trunk" branch is the only source of truth.

There's no 'my version vs your version' here, we all have the same version. No 'what commit is prod right now?': the answer is ideally always 'the last one'.

While using branches is not necessarily blasphemy (it might be more practical to use them), no long-lived branches should exist.

Of course for this to be viable, commits should be small and happen as frequently as possible. Code should be thoroughly tested and always releasable.

Simply put, you should aspire to constantly commit correct, tested code straight to production.

What's wrong with branches?

Nothing per se, but we often forget the trade-offs they bring. Here's a refresher.

Merge Hell

In big projects with multiple dev teams working on shared code, Merge Hell is a very real issue.

Nobody wants to spend half a work-day resolving merge conflicts, much less pay for someone's salary to do so.

The feeling one gets after resolving merge conflicts for 2 hours, only to find out more conflicting code was pushed in the meantime is... not nice.

It's just not a productive use of time.

Partial Truths

If you are working on a branch created more than a day ago, you know how your code behaves with yesterday's project. Your 'Truth' got stuck in time.

Today's version of the project might not behave the same, and you might need to re-think what you are doing. Wouldn't you want to know if that's the case as soon as possible?

The other side of the coin might be even worse: While your code is not in the main branch, you are hiding information from the rest of the team.

Nobody knows what your code looks like, how it behaves or how to work with it. You are hiding your 'Truth' from the rest of the team(s).

Speed (or lack thereof)

Do we really need long-lived branches at all? When a critical hotfix is required, we clearly have no issue pushing directly to the main branch.

Even with a protected main branch, we create a short-lived branch on the fly before deploying it directly to production, no fluff involved.

This approach allows us to quickly update production software, delivering immediate value by squashing a bug or addressing a critical need.

So why not aim for the same speed when rolling out new features or UI updates? Why should we go fast only in emergencies?

TBD? CI? CD?

Some might argue that these problems are avoided by practicing CI/CD, no need for this TBD business. The thing is, you are probably not really doing CI/CD if you aren't also doing TBD.

On the other hand, if you’re doing TBD, you're either practicing CI/CD, incredibly smart, or incredibly dumb.

CI/CD is a somewhat ambiguous term: It depends on what one considers "continuous".

Teams coming from a monthly release cycle might consider weekly integrations to be "continuous", while some might argue that daily integrations are the bare minimum to be considered "continuous".

CI/CD is also quite often mistaken with "having a pipeline", which is indeed a necessary and important part of the deal, but not the whole thing.

One can hide behind these ambiguities when talking about CI/CD, but there's no hiding from TBD: You either push to prod, or you don't.

If we want to deliver software continuously (CD), we need to integrate our work continuously (CI). At some point one has to evaluate if the rituals involved in a branch-based workflow really allow any of this, or if we are better off getting rid of the paperwork and focusing on what matters.

The fundamental assumption of CI is that there's only one interesting version, the current one.

-Wiki

If the only interesting version is the current one, why waste effort, time and resources in other versions?

PRs kinda suck

This does not mean we need to ditch PRs completely: As mentioned before, branches are not the devil, PRs have their place.

TBD is a general way of approaching development, not a strict dogma.

We'll see that these principles can be followed even in a PR centric workflow. So long as their use is reasoned and makes sense in context.

Still, it’s worth considering why we use and (supposedly) "need" PRs in the first place, especially given their downsides:

Reviews are a pain: PR reviews are often treated like chores: rarely does one enjoy doing them and more often than not, they are done as an afterthought in some spare time or hastily before doing the 'actual work'.
Context change: They demand frequent context switching, especially if one is expected to prioritize them. This is not productive and should be avoided if possible.
Slow pace: Reviewing PRs, if done with care, is a time-consuming process (more so with long-lived branches), and even with the best intentions and effort, it still significantly slows down the pace at which we deliver value to users. After all, perfectly good code might be sitting in a PR right now just waiting for someone to review it while it could be adding value to the project.
Better alternatives: In most situations, live code reviews and pair or mob programming sessions are a much faster way of ensuring code quality (or asking for input/help), with much less chance of overlooking mistakes, introducing bugs or creating unnecessary friction between co-workers.

Gatekeeping

More often than not, PRs are used as a form of gatekeeping. We assign a keeper for the 'Security' gate, one for the 'Testing' gate, another one for the 'Efficiency' gate and so on.

To be clear, this can make sense in some cases:

Many juniors, few seniors: PRs are a nice way of managing these team layouts ensuring quality standards are met and bugs avoided. Ideally the senior would pair-program his way out of this situation, but it is a useful temporary crutch.
Open Source Software: In this context there are only a handful of maintainers with a complete picture of the codebase and sometimes hundreds of occasional contributors. It just makes sense for them to inspect the code before merging it and PRs are the best way to do so. The maintainers will be the ones... maintaining the code in the long run after all.

Gatekeeping might be done with good intentions and might be necessary in certain moments and contexts. Usually though, especially in a business setting, there's a deeper underlying cause.

Trust

You might feel uncomfortable letting "anyone push to prod".

It might be worth digging deeper here. What would you expect to happen?

Do you expect your teammates to knowingly introduce bugs? Are you worried that they aren't 'good enough'? 'Professional enough'? 'Smart enough'?

If these doubts sound silly: Congrats, you trust your teammates!

If instead they sound reasonable, you might want to ask yourself if you are comfortable working with people you don't trust or respect, well before thinking about TBD.

A team can't work effectively if their members don't fully trust each other. Each member should feed like the rest of the team has their back, and that everyone is capable of doing at least as good a job as they do.

If this isn't the case, no, TBD is not for you. I would argue the team itself is not for you either.

Write secure software

Fri, 17 Feb 2023 23:26:08 GMT

Bolting security onto a system as an afterthought is about as effective as retroactively adding tests to meet coverage goals, which is to say, not very.

Testing a system that wasn't designed for testability is a huge pain and often limits both the scope and method of testing. Similarly, securing a system that wasn't built with security in mind can feel like using a sledgehammer to crack a nut.

What follows is a non-comprehensive list of heuristics to ensure a system is built with security at its core.

Minimize complexity

Unnecessary complexity is the enemy of good software. It is also the enemy of secure software. The more complex the system, the more surface area, the more nooks and crannies that can be exploited.

Complex UIs are more likely to produce invalid states, keep them simple. The more user flows your app has, the harder it becomes to keep track of all of them.

Think of each software integration, SaaS or dependency as a possible security risk, each supported platform is a new Pandora's box to be opened. This doesn't mean these things should be avoided altogether, just keep in mind that they imply risk, evaluate if it's worth it or not.

Software minimalism is a bit of a meme, but it is true that having less (features, versions, size, etc.) is a surefire way to minimize possible attacks.

Clearly define boundaries

It's surprising how often complex systems don't have clearly defined points of entry. This often leads to incomplete or inconsistent input validation.

Defining what pieces of your system will communicate with external systems (including but not limited to The Users) is a key step to securing it. This determines what exactly you are securing, where to focus your efforts.

Another team's microservice sending incorrect input to your part of the system is an issue, but usually benign and solvable with a Slack message. A public API receiving incorrect input might be user error, but might also be something else.

This doesn't mean internal services can disregard security, but security requirements vary based on what it is you are securing. Defining where a system ends and another one starts is key to detecting what parts need more or less security.

What are you defending against?

This doesn't need to be a full threat model for the whole organization, but it is useful to ask certain questions.

Is the business B2C or B2B? Is the API public or for paid users only? What kind of data is being stored? Is the government involved somewhere? What relationship does the business have with its clients? Are possible competitors also using the software?

You don't need detailed answers to all of these questions, but the more information/context you have, the better you can define security requirements.

You might have none of the answers you need to make an informed decision. First: Really? How do you have no context of the software you work on? In any case, you can look at the most critical security risks for a good baseline.

Defending against everything often ends up defending against nothing. Defining what you are defending against is key.

Restrict access

Both at the user and at the code level.

Users should not see parts of the system they shouldn't interact with. Giving a user a big red "DELETE ALL" button and telling them not to use it is like handing a child a crayon and expecting them not to draw on the walls.

Making internal functionality available for others to use (think making a function public instead of private) and expecting them not to, is equally naive. This goes back to the previous point about defining boundaries: limit the ways a piece of code can be interfaced with.

Grant as little access and as little visibility as needed. Not more, not less.

Whitelisting > Blacklisting

Ideally, one would prefer the former over the latter. This is kind of what is usually done with admin users: only these 'whitelisted' users have access to certain things.

The same goes with IPs: don't wait for a DOS attack to start blacklisting IPs, block everything except the ones registered by the users/clients.

Of course, this isn't always possible as is the case with public facing APIs or services. Blacklists should not be avoided, but Whitelists should be preferred when possible.

Create alarms

You'd be surprised how often inappropriate uses are discovered when investigating logs trying to squash a bug.

Consider creating alarms for unexpected execution flows. Of course, handle the error in the code, but also give it a thought: Could this behavior suggest more than a user error?

An IP address constantly being rate limited should not go unnoticed. A delete operation performed 1200 times in 1 minute in the main production database is likely more than a bug (and even if it is, you probably still want to be notified ASAP).

Set up a useful logging system, have alarms to cover edge cases and/or critical user flows (logins, payments, etc.). This will help minimize the time it takes for a possible attack to be detected and dealt with.

Test for security

Pentesting is great, but often time-consuming and expensive. While writing security driven tests is no substitute for this, it can alleviate some of the work.

Write your security requirements as tests, like you would write an acceptance tests for a use case. Use fuzz testing to discover unwanted behavior.

If you fix a security issue, write a regression test to ensure it doesn't happen in the future. Or even better, use TDD to fix it.

Organizational aspects

While a lot can be done as a dev, ensuring the whole organization is aligned on security practices is key. Here are some things worth considering.

Involve your business team

Security is always and implicit requirement, but often not an explicit one. Users might not ask for it, PMs might not add it to the board, but they do expect the system to be secure.

Make sure everyone involved is aware of this. Resources should be allocated to security.

Have a plan

Don't assume that nothing bad can happen, nor that you will be able to recover from it.

Have a reliable backup system, some sort of contingency plan: When something goes wrong, you should be able to recover from it.

Ensure you can roll back the state of your software: When a vulnerability is introduced, you should be able to quickly roll back even before you start fixing it.

Audit your system

Reacting quickly is key, but preventing security issues can be far more cost-effective.

Consider having a bounty program, call a pentester once in a while. Even better, learn the basics of pentesting yourself!

Remove trust from the equation

Often, particularly non-technical people, think that proprietary software is more secure by virtue of being opaque. After all, if I can see how it's made I can see its weaknesses, right?

This is "security by obscurity" and, while it can make sense in some contexts, it is generally not recommended (especially not on its own).

A secure system is not one that is only safe if the attacker can't see it, but one that is safe even in that case.

Either outsource security to a third party based on the guarantees it offers (and it's SLA), or prefer open standards and software.

The chance of auditable, open and widely used protocols and/or software being insecure is slim: everybody is using them, depending on them and auditing them. Plus, issues with these systems are, by nature, instantly public and have a vast pool of talent willing and invested in solving those vulnerabilities if/when they occur.

Software licences

Fri, 10 Feb 2023 16:15:44 GMT

An overview of different open source software license types with a quick run down of the most common ones.

TL;DR

Code with no license attached is under exclusive copyright of the creator.
- This applies for public GitHub/GitLab repos: users can only see and fork them.
GPL requires all derivative work to be licensed with the same license.
MIT has basically no restrictions and is the most widely used.
Apache is like MIT, but patent rights are explicitly granted to the users.
MPL is like MIT, but modifications must share the same license.
BSD-3 is like MIT, but the project name cannot be used as endorsement.

You can read more about the various licenses here and here.

Permissive vs Restrictive

These words get thrown around a lot, but their meaning might seem counter-intuitive.

In a nutshell, restrictive licenses work a bit like viruses: They require derived work to share the same license. They restrict derivative work to that specific license. In contrast, permissive licenses have a more "do what you want" approach.

So in this context, these concepts are not from the point of view of the user (none of these license restricts the user in any way) but from the perspective of other developers/business, which may or may not do what they want with the software.

MIT & similar licenses

MIT licensed code can be used by whoever to do whatever. The only restriction is that the original "owner" cannot be held liable.

A business that stumbles across some MIT licensed code can feel free to use it however it wants.

Clearly, this license is the most business-friendly, in the sense that businesses are not restricted in any way by it. The story is however quite different from the developers point of view, as this license is perfectly fine with BigTechCorp using your code for profit and giving nothing back.

It's by far the most widely used, not least due to its simplicity.

Code licensed under the Apache license is functionally in the same situation, with the added technicality that patent rights are explicitly granted to the users. This is done to prevent patent-holding contributors from possibly suing users of patented code.

The BSD-3 license is a lot like MIT, with the added restriction that the original project name cannot be used as endorsement of derivative work. So if a business decides to redistribute a piece of BSD-3 licensed software, it could not use the name of that software to promote or endorse their product in any way.

MPL is technically a restrictive license, but it only is so for modifications of licensed code. This is to say, all modifications of an MPL licensed code must also be licensed under an MPL license.

GPL & friends

GPL is a family of restrictive, copyleft licenses that broadly require all derivative work to be distributed under the same license. Code under these licenses is generally considered not only Open Source, but Free and Open Source.

Code under these licenses can be used freely only in projects under the same or similarly restrictive licenses. So if a business wants to use a piece of software under a GPL license, it can do so with the condition all code derived from it is distributed under the same license.

This is why it is traditionally considered less business-friendly than the previously mentioned licenses: I can't just "steal" your work and profit, I have to "give away" the code as well. Making money from GPL code requires the business to actually provide a service, add value, instead of merely hiding the code behind a paywall.

Note that internally or privately used codebases don't apply here: These licenses are concerned with code that is distributed (freely or not) to end users. Also, the notion of derivative work here generally refers to any code that depends on and is distributed with the original work. This however may vary from one type of GPL license to another.

Lesser GPL and Affero GPL

A more permissive and restrictive version of the GPL respectively.

LGPL is non-restrictive as long as the licensed work is being used "through interfaces provided by the licensed work". AGPL is more restrictive in that when the licensed work is being used "to provide a service over a network, the complete source code [...] must be made available".

Simply put: LGPL draws an exception for libraries. AGPL also considers it "derived work" if it sits behind a network.

GPLv2 vs GPLv3

Similarly, while GPLv2 includes no particular consideration regarding hardware, GPLv3 does:

If the software is part of a consumer device, you must include the installation information necessary to modify and reinstall the software.

So in this case, distributing software as "part" of hardware does not exclude it from the GPL restrictions.

Why does this matter? Well some companies thought that only software distributed by itself (think CDs or floppies in the olden days or a software download nowadays) would be subject to the license restrictions. For the more curious, look up "Tivoization".

The new version of the license explicitly prevents this "misunderstanding".

Other licenses

There are of course as many licenses as one is willing to look for.

The Unlicense for example states that the work is dedicated to the public domain. No copyright, no restrictions, no strings attached.

Other honorable mentions are the Do What The Fuck You Want To, the I Don't Give A Fuck, or the Good Luck With That Shit licenses. Some people just can't be bothered.

No License?

So what happens if a piece of code is not licensed at all? By default, the author has full exclusive copyright: nobody should do anything with that work without his/her permission.

This is irrelevant if the work is kept private, but might lead to interesting situations when sharing it. Say a user stumbles upon an unlicensed piece of code on the internet. This user has three options:

Don't use the software.
Negotiate a private license/bring a lawyer.
Yarr me salty seadog!

This is awkward and can be messy to deal with, so please just license the work!

GitHub TOS

This leaves us with a final consideration on GitHub TOS:

By setting your repositories to be viewed publicly, you agree to allow others to view and fork your repositories.

So in trusting Microsoft, we agree that our unlicensed, public repos are free to view and fork, but nothing else. This leaves us in the same situation described above so again: Just license your work!

Parse JSON with jq

Sun, 05 Feb 2023 13:25:18 GMT

Our beloved GNU utils, especially sed and awk, work better with some file types than others. JSON, YML or XML files can be a bit of a pain to work with.

Jq is a parser specifically designed to handle JSON files, and there's a bonus tool at the end for YML and XML files as well!

The basics

Let's take a simple JSON as an example: run curl https://til.hashrocket.com/api/developer_posts.json?username=doriankarter on your command line to see the data.

Since this data is presented as a one-liner, we can use jq to format the output:

curl https://til.hashrocket.com/api/developer_posts.json?username=doriankarter | jq

We can query the interesting bits and remove some noise simply by referring to its node name:

curl https://til.hashrocket.com/api/developer_posts.json?username=doriankarter | jq '.data.posts[]'

To output the data as an array we can just enclose the query in []:

curl https://til.hashrocket.com/api/developer_posts.json?username=doriankarter | jq '[.data.posts[]]'

Or we can do some interesting manipulation to the data and present a parsed version:

curl https://til.hashrocket.com/api/developer_posts.json?username=doriankarter | jq '.data.posts[] | {id: .slug, formatted_title: ("THIS IS A TITLE - " + .title)}'

Again, we are accessing the data by their node name and doing some string concatenation.

Notice how we use a pipe (|) to pass the data from one command to the next.

The not so basics

This tool has a bunch of very useful functions available, we'll go over a few of them.

From now on, there will be no reference to the curl command to keep the code blocks more concise.

Delete node

Use it to clear out unwanted noise:

jq 'del(.data.posts[].slug)'

Filter data

Select only the entries that match the given condition:

jq '[.data.posts[] | select(.title | length > 30)]'

Add a node

You can add nodes to the JSON:

jq '.data.posts[] | (. + {hi: "mom"})'

Conditional logic

Following the previous example, we can use if statements to add a node with variable content.

Here we create a new one called IS_VALID with the value "Too short!" or "yes" depending on the length of the .title.

jq '.data.posts[] | (. + {IS_VALID: (if .title | length < 30 then "Too short!" else "yes" end)})'

Perhaps more useful, we can add the new node or not depending on the condition:

jq '.data.posts[] | (if .title | length > 30 then . + {IS_VALID: true} else . end)'

Group by

Group nodes by values using group_by():

jq '[.data.posts[] | (. + {IS_VALID: (if .title | length < 30 then "Too short!" else "yes" end)})] | group_by(.IS_VALID)'

Notice how in this case we create a new array with the data before sending it to group_by().

Sort by length

Sorting is also possible and can the result be reversed if needed:

jq '[.data.posts[] | (. + {len: (.title | length)})] | sort_by(.len) | reverse'

Notice that we add a .len node with the result of passing .title to the length built-in function.

Modify in place

So far we've always focused on the content of the posts array, losing it and the data node names in the process.

This might be what you want, but in some cases one needs to modify the data 'in place', keeping the original data structure.

This can be done swapping the pipe operator (|) for the modify-in-place operator (|=), so for this simple example from before:

jq '.data.posts[] | (. + {hi: "mom"})'

If we wanted to modify the original data structure including the data and posts node names, we could instead do:

jq '.data.posts[] |= (. + {hi: "mom"})'

Handle other file types with yq

Since this is so useful, someone took the time to create yq (as in YAML query). It actually doesn't just handle YAML files, but also XML, CSV and TSV.

Not only that, you can easily use this application to convert one file type into another! Check the docs to find out more.

Keep in mind that apart from what is shown below, all the previous operations can be applied to any of these file types. Since yq uses similar syntax as jq, I'll keep it out of the examples to keep things simple.

This is just a quick overview of how you might want to use the tool, it can achieve much more than I'm showing here.

YAML to other types

For a cool.yaml file of the structure:

pets:
    cat:
        - purrs
        - meows

The command yq -o xml '.' your_cool.yaml would output it with XML structure:

<pets>
  <cat>purrs</cat>
  <cat>meows</cat>
</pets>

Or you can run it like yq -o json '.' your_cool.yaml to get a JSON instead:

{
    "pets": {
        "cat": ["purrs", "meows"]
    }
}

Any Input, Any Output

Say you have a cool.csv file of the structure:

name,numberOfCats,likesApples,height
Gary,1,true,168.8
Samantha's Rabbit,2,false,-188.8

Convert it to YAML with yq -o yaml -p csv '.' your_cool.csv:

- name: Gary
  numberOfCats: 1
  likesApples: true
  height: 168.8
- name: Samantha's Rabbit
  numberOfCats: 2
  likesApples: false
  height: -188.8

Again, use the -o flag to change the output format yq -o json -p csv '.' your_cool.csv:

[
    {
        "name": "Gary",
        "numberOfCats": 1,
        "likesApples": true,
        "height": 168.8
    },
    {
        "name": "Samantha's Rabbit",
        "numberOfCats": 2,
        "likesApples": false,
        "height": -188.8
    }
]

Notice the use of the -p flag to indicate the input format, since by default it will expect a YAML.

Pair Programming 101

Sat, 04 Feb 2023 10:10:33 GMT

A quick overview of things to keep in mind and best practices when pair programming.

Values

Not too far away from the ones described in XP, the key values to foster when working in pairs are as follows:

Humility

Appreciate feedback. Learn to give and receive it. Don’t be afraid of being wrong, ask questions when needed (especially if you feel blocked).

Trust

Believe in yourself and (especially) in your partner. Understand that everyone solves problems differently. This is a good thing, trust that someone else's path will lead to a desirable outcome.

Grit

Muster the courage to stick together, challenge and motivate each other.

Care

Foster teamwork, care for your partner. Be there for one another.

Roles

The two main roles in pair programming are the Driver (the one at the keyboard) and the Navigator (the one helping out). Contrary to popular belief, these roles entail much more than "I type you look".

Driver

As a driver, you should do one thing and to it well: Focus. Worry only about the smallest step possible, "forget" about the feature/task as a whole.

Think small: naming, algorithms, implementation details, etc. should be your main practical concerns.

Two key and often overlooked aspects of being the driver:

Think out loud It's hard for the pair to be on the same page if the navigator doesn't know what the driver is thinking. Narrate your thought process.
Be open to criticism/improvement/discussion Part of the point of working in pairs is to question one another and reach the best approach possible. Don't take suggestions as an insult, your navigator is just trying to help!

Navigator

A good navigator should practice tactical thinking, reaching compromises when required. On top of that, think about the bigger task at hand, constraints, API, performance, etc.

The navigator should be asking good questions to ensure the development follows a desirable path. In this, rabbit holes should be avoided.

This role is in an advantageous position to catch typos, errors, etc. and should beware of complexity (especially if avoidable).

Forcing breaks when needed, keeping an eye out for meetings and urgent Slack messages are tasks generally best suited to this role.

If you are in this position, be sure to help out as much as possible and take notes for future reference. You should have the relevant resources and documentation at hand, and should remember (or have a list of) the things that came up during the session that might need further investigation or analysis.

Practice

On a more technical note, here are some things to keep in mind as far as "how to's".

A pair of what?

Are Juniors supposed to pair? What about Seniors? There are three possible combinations here, each with its strengths and weaknesses:

Novice–novice

Significantly better results than two novices working independently. Still, this makes it hard for novices to develop good habits, since they lack a proper role model.

Expert–novice

Possibly the best practice for mentoring and for getting fresh air into the system, as the novice is likely to question established practices. This is great, but the expert needs to have the patience to handle the situation and the novice might feel too intimidated to have a productive session.

Expert–expert

Best for productivity and, given that both understand pair programming, very likely to produce great results. Still, two experts are unlikely to question established practices, so novel ways to solve problems will likely not come up in these sessions.

Alternative styles

Ping-Pong

Best served with a side dish of TDD. Simply put, switch roles on every TDD step:

I write a test (You are the navigator)
You make it pass (You are the driver)
I refactor (You are the navigator)
You write a test (You are the driver)
...

Strong style

The one with the idea (and/or knowledge) sticks to being the navigator.

I've got an idea, please take the keyboard!

Very useful to ensure knowledge transfer.

For an idea to go from your head into the computer it MUST go through someone else's hands.

Here, the senior should be navigator and the junior should be driver. This can also be applied with two experts while handing over a task, as it ensures the objective is achieved.

Mob-ish

The line between the two roles gets blurred out.

With some tasks the role separation might not fit very well, but it might still be advantageous to tackle the issue as a pair. Simulating a mob programming session, the pair as a whole takes care of the responsibilities of both roles.

Time management

It is extremely important to consider breaks as a natural part of your work as a dev, much more so when pairing.

This practice can be quite tiresome and is not sustainable if frequent breaks are not taken. And proper breaks at that: no slack, no emails, no checking tech-support. Get up, go for a glass of water, step outside, etc.

There are multiple ways of ingraining breaks into your pairing routine:

Methodical approach

The classic "Pomodoro" style:

Work for 25 min
Take a 5-min break
Work for 25 min
Take a 5-min break
Work for 25 min
Take a 20-min break

The time windows should be adjusted as needed, but some consistency should be kept (you either take 5-min breaks or 10-min breaks, not 5 now, 10 later and 8 afterwards...).

Organic approach

If (and only if) the team is well-adjusted to pairing and there is a safe work environment, going by "gut feeling" might work quite nicely. Take breaks as needed, have a "feel" for when a good moment to take a break arrives.

Rotation dependent

Another approach is to sync your breaks with your role rotations. This can work really well in teams that are used to pairing and tend to swap roles very frequently.

Of course, not all tasks will allow this and not all teams are okay with it. But especially in a remote environment, it is often quite nice to do so when possible.

Rotations

Role Rotation

Really helpful when people are new to pair programming, as they can quickly get a feel for how each role works. This also keeps you on your feet, since it requires you to fully focus on you current role's responsibilities.

As to when to rotate, it can sync with your time management schema or go by user story. Make a conscious decision about when and how to do this, avoid switching roles for no reason.

Pair Rotation

Useful to spread knowledge between people, and facilitate collective code ownership. It requires that everyone on the team is willing (and capable) to work with each other.

It keeps things fresh, as different ideas and new point of views are likely to come up on a regular basis.

Usually it doesn't make much sense to rotate pairs multiple times a day. Think of doing it each Sprint, every X days, or maybe with every new user story.

Different teams and workflows will allow for more or less frequency in this regard.

Don'ts

Each team does pair programming in its very own unique way. This is a good thing, practices like this should be adapted to the team's needs and/or abilities.

There are however a few things that are better avoided.

Don't...

Drift apart, zone out, loose focus, look at your phone...
Micromanage what your driver should do. This is sometimes welcome, especially if the driver is new and/or blocked.
Be impatient. Leave some room for your driver to figure out the error.
Stress out your partner.
Marry the keyboard. Sharing is caring.
Pair all the time. Your job is much more than coding, pairing does not apply when writing emails or researching topics.

Tame your dotfiles with Stow

Fri, 03 Feb 2023 17:21:42 GMT

As soon as you start putting some time into your *nix system configuration, you'll probably notice your config files getting out of hand.

Dotfiles all over the place, no good way to back them up, versioning is a pain, keeping your configs in sync in multiple OS or multiple computers becomes a chore.

You can use stow with git to tame them once and for all!

TL;DR

Make a ~/dotfiles/ directory using this repo as a reference for the dir structure.
Move your config files to the corresponding directory in ~/dotfiles/.
Run stow * from ~/dotfiles/.
Profit?

Context

Symlinks

Symbolic links are a way for us to make the OS believe there is a file where there's not. It's a reference to a file that might be wherever we want.

You can create one from the command prompt with ln -s <path_to_file> <path_to_link>.

Naturally, the link will always show the same content the file has.

Stow

This GNU command line utility is probably already installed on your *nix system. At a basic level, it's a symlink farm manager. Give it a directory, and it will create a mirror directory with the same structure and links to the files in the original one.

Stow is a symlink farm manager which takes distinct sets of software and/or data located in separate directories on the file system, and makes them all appear to be installed in a single directory tree.

This can be used as a really basic package manager, but we can make better use of it.

However, Stow is still used not only for software package management, but also for other purposes, such as facilitating a more controlled approach to management of configuration files in the user’s home directory, especially when coupled with version control systems.

The Plan

Say our favorite text editor expects a vimrc file in .config/nvim/.

If we run stow from a directory that mirrors the expected one, it will create the corresponding directory structure and symlink.

Let's see it in practice:

~
├──dotfiles [YOU_ARE_HERE]
│  ├── nvim
│  │  ├── .config
│  │  ├── nvim
│  │  │  ├── vimrc
...

Running stow nvim from ~/dotfiles/ will create the following structure, directories and all (assuming no conflicting files or directories where already there):

~
├── .config
├── nvim
│  ├── vimrc
...

This vimrc only references the one under ~/dotfiles/nvim/vimrc.

⚠️ Careful

By default, stow will clone the dir structure from cwd (configurable through the --dir flag), to its parent directory (configurable through the --target flag).

This is why the examples assume that the dotfiles directory is in ~/dotfiles. I would advise you to follow this structure to avoid unpleasant surprises.

The Execution

This means we can have all our config files in one neat repository like this and deploy them with stow *.

The deploy command will only be needed once, since editing the files from the dotfiles directory will update the symlinks as well.

Whenever you update your config on one machine, just git commit and push the changes. You'll be a git pull away from syncing them across other machines!

Some programs expect their configs in ~/. Handle them like this.

Others expect them under ~/.config/. No problem: just as we explained, create a mirror image and let stow take care of it.

This, apart from being incredibly easy to keep track of these files, allows us to easily use git to version (and share!) them. Moreover, it makes it a lot easier to automate the config process of your favorite OS.

Migration

More than likely, if you try to use stow on a system already quite configured it won't be happy.

This is because, if it detects files where it wants to establish symlinks it will let you know and abort.

There are flags available to tell stow to stomp over whatever it finds, but I would advise you not to follow this approach.

Take your time when setting this up. Go one by one, moving the file or directory over to your dotfiles directory. Ensure the structure is correct, remove the old config and run stow DIR_NAME one by one.

You'll be happy you did!

How to Awk

Thu, 02 Feb 2023 16:13:59 GMT

Not only a command but a full-blown scripting language, awk is a powerful tool for text processing.

It's a great way to quickly search through text files, extract and format data, and even perform basic calculations.

Dive much deeper into awk here and here.

Keep in mind

Not all awk implementations are created equal. This post references the GNU implementation.

The basics

Awk operates on records and fields. By default, a record is a line (uses \n as a separator) and a field is a "word" (uses \s or 'space' as a separator).

It performs an action based on a pattern, as in "if it matches this, do that".

Your basic awk command looks something like this:

awk '/himom/ {print $0}' file
        |         |
     pattern    action

Patterns will always be delimited by / while actions will be within {}. Note also the use of single quotes.

This reads: "On each record (line) that matches the pattern himom, run the action print $0 (which prints the whole line)."

You can omit the pattern to perform the action in all lines, or omit the action to print each matching line.

So awk '{print $0}' file would print the whole file, while awk '/himom/' file would do the same as awk '/himom/ {print $0}' file.

Positions

As you might imagine, changing $0 for $n will print the nth field (word) instead of the whole record (line).

Regex

The pattern '/himom/' is a shorthand for '$0 ~ /himom/'. This means that patterns can be applied on a per-column basis.

If you know your regex, you might expect the previous pattern to match lines containing only the word himom.

This is not the case, the default behavior is to match anything containing the given pattern.

Also, while ^ and $ usually designate beginning and end of line, here they indicate beginning and end of word (field).

This means that for awk '$1 ~ /01$/', the line 01 02 03 would match

Unlike sed or grep, you don't need the -E to use extended regular expressions since this is the default behavior.

Variables

Ignorecase

Awk is case-sensitive by default, but this can be switched off by setting this variable:

awk -v IGNORECASE=1 '/fooBar/ {print $1}' file

We use the -v flag to set the IGNORECASE Variable to true.

Filename

When processing multiple files in a script, it might be useful to also print the current file name:

awk '{print FILENAME}' file.txt

(Input) Record and Field separator (RS & FS)

As mentioned before, the default RS is \n while the default FS is \s. This can be configured to fit different file structures.

A CSV for example might not behave as expected:

Tonia,Ellerey,[email protected],firefighter
Joleen,Viddah,[email protected],police officer
Cherilyn,Kat,[email protected],firefighter
Janenna,Natica,[email protected],worker

Something like awk '{print $3}' file will not really work, but awk -v FS="," '{print $3}' file will:

[email protected]
[email protected]
[email protected]
[email protected]

Similarly, we could change the RS variable as well, although that is a less common use case.

(Output) Record and Field separator (ORS & OFS)

These are used to format the output of your awk command.

While for simple commands, something like awk '{print $3" - "$4}' file should do the trick, this can get tedious and unreadable fast with more complex ones.

For such cases, use the OFS variable:

awk -v OFS=" - " '{print $3, $4}' file

Notice the " - " separator in both examples.

There is also printf support in awk, so you can get as fancy as you want.

Record and Field number (NR & NF)

These hold the value of the current line (record) and word (field) numbers. You can print them with something like:

awk '{print "Line num:", NR, "Num of fields:", NF, "Content:", $0}' file

Or use them to conditionally apply the action:

awk 'NF<10 && NR>2 {print $2}' file

"Print the 2nd field of all records whose NR is greater than 2 (3rd line onwards) and whose NF is less than 10 (9 or fewer fields)".

The not so basics

Logical operators

As hinted above, we can use && and || as in most other programming language. Patterns can be mixed and matched using these logical operators.

awk '/bilbo/ && /frodo/ {print "My Precious"}' file
awk '/bilbo/ || /frodo/ {print "Is it you mister Frodo?"}' file

Or you can negate the match, as in "only perform the action on lines that DON'T match the pattern".

awk '! ~ /frodo/ { print "Pohtatoes" }' file

Ternary operations

Since we can use logical operators, you might imagine that we can also take advantage of ternary operators.

awk '/frodo/ ? /ring/ : /orcs/ { print $0" --> Either frodo with the ring, or the orcs" }' file

Which we can write in pseudocode as:

if matches(frodo) AND matches(ring)
    print "Either frodo with the ring, or the orcs"
else if matches(orcs)
    print "Either frodo with the ring, or the orcs"
else
    don't print

So for a file:

frodo
ring
orcs
frodo ring
frodo orcs
ring orcs
frodo ring orcs

The command above would output:

orcs --> Either frodo with the ring, or the orcs
frodo ring --> Either frodo with the ring, or the orcs
ring orcs --> Either frodo with the ring, or the orcs
frodo ring orcs --> Either frodo with the ring, or the orcs

Range

If the file you are working with has some kind of internal sorting, you might want to operate based on that instead of the NR.

You can use multiple matches to create a range on which to perform the action. So on a file like:

first line
second line
third line
fourth line
fifth line

The command awk '/second/ , /fourth/ {print $0}' file outputs:

second line
third line
fourth line

Scripting

Here we only covered how to use awk as a one-liner from the command line, but awk is actually a fully featured scripting language.

The previous point regarding ternary operations skips over the fact that the action per se can include conditional logic.

This for example, is a valid awk script:

#!/usr/bin/awk

/hi/ {
  if($1 > $2){
    print "mom!"
  }
  else print "there!"
}

In fact, if your awk commands are getting a bit out of hand, turning them into a script might make things a lot easier.

How to grep

Fri, 22 Apr 2022 11:47:49 GMT

Grep helps you locate any given pattern(s) within one or more files.

Very useful when parsing logs!

Keep in mind

Not all grep implementations are created equal. This post references the GNU implementation.

The basics

Grep commands have the following structure:

grep [OPTIONS] 'this_string' that_file

This will output the full line(s) where this_string was found, highlighting the match itself.

Context

There is a -n flag you can use to get the line Numbers of the matches.

You might find it useful to have some more Context around your grep results.

Use something like -C2 to tell grep to also print the two lines before and after each match.

Keep in mind that the amount of context lines printed will be limited by other matches as well as the beginning and end of the file, so you might not always get exactly the amount of lines you asked for.

These two flags work well together, since grep will separate line numbers from the line itself using : for matching lines and - for context lines.

So given a grepme file like so:

Lorem ipsum odor amet, consectetuer adipiscing elit.
3183_22_4 -> '3183_22'
Lorem ipsum odor amet, consectetuer adipiscing elit.
3183_22_5 -> '3183_22'
Lorem ipsum odor amet, consectetuer adipiscing elit.
Lorem ipsum odor amet, consectetuer adipiscing elit.
3283_23_1 -> '3183_23'
Lorem ipsum odor amet, consectetuer adipiscing elit.
3183_23_2 -> '3183_23'

grep -nC2 '3183_22' grepme will output

1-Lorem ipsum odor amet, consectetuer adipiscing elit.
2:3183_22_4 -> '3183_22'
3-Lorem ipsum odor amet, consectetuer adipiscing elit.
4:3183_22_5 -> '3183_22'
5-Lorem ipsum odor amet, consectetuer adipiscing elit.
6-Lorem ipsum odor amet, consectetuer adipiscing elit.

Multiple files

You can use dir/* instead of a file name to tell grep to look in all files in dir/ (or simply * to look in all files under cwd).

If there are any directories here, it will print errors since it can't do much with them.

To Suppress these errors, use the -s flag.

Quality of life

Count

More often than not you'll need the number of matching lines, more so than the lines themselves.

You might be tempted to pipe grep into wc -l, but there are better options.

grep 'hi there!' file | wc -l and grep -c 'hi there!' file produce the same output: They both Count the number of matching lines.

Or, use the pipe with the -o flag to get the number of Ocurrances (which will differ from -c if there are more than one match per line).

So following the previous example:

grep -c '3183_22' grepme ➡️ 2

grep -o '3183_22' grepme | wc -l ➡️ 4

-o on its own will simply print the matches themselves, which doesn't make much sense right now, but will once you add regular expressions to the mix.

The classics

There are some combinations that are used so often you might as well create an alias for them.

grep -rinv 'foo' .
grep -rl 'bar' .

The first command will output all lines plus lines Numbers (-n) NOT matching foo (-v). It will look for the match recursively (-r) with case Insensitivity (-i).

The second one will output all files containing a match (-l, -L would output only files NOT containing a match) for bar, recursively (-r).

The not so basics

Multiple searches

Just like sed, you can use -e to concatenate multiple searches in the same grep command.

Using sed, this flag runs the all commands on each line. Similarly, here it will print out all lines that match any of the expressions.

This might be surprising, since when piping grep commands into each other, the result will be the exact opposite: you will get only lines that match all the expressions.

So again, using the example file from before:

grep -e '3183' -e '22' grepme | wc -l ➡️ 4

grep '3183' grepme | grep '22' | wc -l ➡️ 2

Here I use | wc -l instead of -c for clarity/symmetry.

Regex

Again, just like sed and find, grep uses reduced regex by default and the -E flag allows you to use its full regex engine.

If instead you want to avoid regex altogether and look for a literal string with strange characters, use -F.

grep -F '[Hh]ello moto*' file will literally match "[Hh]ello moto*". Not "Hello moto", not "hello moto", and not "[Hh]ello moto, something else".

Exclude and include

You can exclude and include files from the search by a given pattern.

Even better, you can use both flags together to fine tune where you are searching exactly.

grep -s --exclude=*.py --include=main.py 'something' *

Will exclude all Python files from the search, except for main.py.

Grep based on a file

Say you have a list of blacklisted words you want to ensure are not present in a project.

grep -f blacklist.words projectFile will print out all matches for any of the lines in blacklist.words, while also passing it the -l flag from before will print only the problematic filenames.

For this to work, blacklist.words has to contain one expression (or word) per line.

Another neat use case: ls | grep -f blacklist.files

This will output all filenames in cwd listed in blacklist.files.

How to better isolate your tests

Sat, 26 Feb 2022 17:14:45 GMT

Introduction

These are tools used to imitate or substitute parts of the production code in our testing environment. Usually Services, Repositories, event buses, etc. although you can apply the same principles in much simpler contexts (like katas). They are useful to ensure we are testing the different parts of the system in isolation.

In practical terms, we can say that when the system under test (SUT) depends on a separate piece of our application (separate as in should be tested separately), the second one should be substituted. Specifically the parts we are not interested in testing but are required for our SUT to function.

As was the case for the tools we saw on the previous post, these Doubles might take a minute to set up, but not using these tools makes our tests significantly more fragile and less reliable, since when they break we won't have a clear picture of who exactly is at fault.

In the end, Doubles are just a false implementation of production code. Say you want to test UserService, but it depends on and requires a UserRepository with a search() function. You would make an InMemoryUserRepository that implements UserRepository (with its search() function) to test the Service independently of the Repository.

That InMemoryUserRepository is a sort of Double. We'll use this common example going forward.

Note on definitions

The terms Mock, Spy and Double are often used in different ways depending on the source material

When naming things in your code base please make sure there is a consensus within the team regarding what each word refers to. When looking for help online or debating with a co-worker, ensure you understand what they mean by these concepts. You might be using the same words but talking about different things.

Some consider Stubs to be very different from Fakes, some don't include Dummy as a testing Double, and some don't differentiate between Mocks and Spies. Some just consider everything a type of Mock Object.

So here's more fuel to the fire.

Good luck 🙃

Dummy

False implementation of production code with no real behavior. It's literally there to make your code compile.

Use Case

You would use a Dummy to substitute a dependency of your SUT when it's only needed at compile time but doesn't really do anything in your testing scenario.

For Example, if our InMemoryUserRepository were a Dummy, it would implement the production UserRepository and have a search() function that does nothing (or the absolute minimum to compile).

Fake

False implementation of production code with very basic, test specific behavior. It would receive some starting data to simulate operations.

Use Case

A Fake is useful whenever you need some very simplistic behavior.

For example, if our InMemoryUserRepository were a Fake, it would implement the production UserRepository and have a search() function that actually implements production logic, but searches in an Array that it got via constructor (starting data).

Stub

False implementation of production code with basic, use case specific and re-usable behavior. It would use some hard coded data to simulate operations.

Use Case

Use a Stub when you need some basic test independent behavior. Basically as soon as you use the same Fake twice with the same starting data, build a Stub with that starting data.

Following our example, instead of our previous InMemoryUserRepository, you would build an InMemoryAdminUserRepository that implements the production UserRepository and has a search() function with production logic, but searches in a predefined, hard coded Array made up of a bunch of random Admin Users. This hard coded Array substitutes the starting data from the Fake example.

This way you could use the same Stub in multiple tests without rewriting the Array and ensuring they all work with the same data.

Spy

Piece of code (or external library) that allows the tester to check if and how a specific interaction with the spied code has taken place. It can tell the tester how many times its methods were called, what parameters were passed to each of them, in what order they were called, etc.

This is the first concept that only takes into account behavior, disregarding output.

It is also the first tool that allows us to see the inside workings of the systems we are testing. Useful, but prone to coupling. Use them sparingly!

Use Case

One would use a Spy to further detach the SUT from its dependency and/or to only make assertions regarding the interaction between the two, not really caring about the final output.

Common scenarios are the assertions 'if the function was called at least once' or 'if the function was called with X argument'. You'll tend to find this behavior implemented within other Doubles, since it is often insufficient by itself.

So now, our InMemoryUserRepository would implement UserRepository and a search() function that literally does whatever, as long as it notifies the tester that it was called.

In our very simple example, the tester could ask the InMemoryUserRepository for the state of searchHasBeenCalled and use that to assert the expected behavior, no matter what the search() function actually does.

You can hopefully see that this couples our Service test to how our service functions (we test if it calls a given function), rather than to what it actually achieves (simply testing the output).

Mock

Keep in mind that, as noted before, some literature refer to everything we've seen here as 'mock objects'. That being said, the sources I've found that consider them as its own specific thing all agree on Mocks being the more sophisticated of the lot.

Speaking of sources, the actual source material states:

[Mock Objects] replace domain code with dummy implementations that both emulate real functionality and enforce assertions about the behaviour of our code.

Similarly, Martin Fowler describes them as:

Objects pre-programmed with expectations which form a specification of the calls they are expected to receive.

One major benefit of using them is:

It makes it a lot easier to write a mocking tool.

Thus, they are often provided by an external library.

Mock objects give the tester full control over the behavior of the code being mocked, which can even be manipulated dynamically. It usually offers all the benefits from the previous tools as well.

As you might imagine, this can be as complex to implement as one heart's desires (hence the external library), but they are very useful and easy to work with.

A super simple Mock might look a lot like a Spy that, instead of exposing whether a given function was called, has some sort of assert function. Mocks "know what they are testing", they make assertions on their own.

That being said, things can (and usually do) get more complicated than that.

Use Case

Say you want to simulate some specific complex behavior of our UserRepository to see how the UserService responds. Given a complex enough behavior, you might have to duplicate (and maintain) quite a bit of code or give up completely and test both elements together.

This might put you between a rock and a hard place, having to choose between flaky tests or giving up test isolation.

You can use a Mock to abstract that complexity away altogether. In fact, if using an external library, you wouldn't even be implementing a substitute for our Repository (like the InMemoryUserRepository from before), since those usually provide a way to create mocks on the fly based on the interface it should implement.

Example

Suppose that, when our UserService calls the UserRepository implementation, the Repository needs to go fetch some data from the database, wait for an email to be sent, check for authentication with a third party service and call your mom to say you love her. Then, based on the results, the Repository returns either an empty array, an array with 4 elements or null (which as it turns out is the behavior you need for your use case).

You could "re-implement" all that code/behavior, or you could mock the whole thing. With Mockito for example you would annotate the Repository with @Mock and use it like this:

when(mockRepo.doTheThing()).thenReturn(null)
// the rest of your test...

A lot simpler than the alternative! Although you are adding a dependency to your tests. Pick your poison!

Abstractions for convenient testing

Sat, 26 Feb 2022 16:43:01 GMT

Broadly speaking, these tools are used to make things easier when testing by setting up objects/data in a reproducible and reliable manner with minimal effort. They might take a minute to set up, but as soon as the data is needed in more than two tests you'll be thankful you took the time.

Fixture

A test fixture is an environment, a state or a dataset we use to consistently test a piece of software.

This example used by Martin Fowler seems appropriate:

When you write tests in a reasonably sized system, you find you have to create a lot of example data. If I want to test a sick pay calculation on an employee, I need an employee. But this isn't just a simple object - I'll need the employee's marital status, number of dependents, some employment and payroll history. Potentially this can be a lot of objects to create.

All of that data is what we call a test Fixture, no matter where it is or what shape it has as long as it's valid. It could be something as simple as a json file with all the data we need.

Often enough, the term Fixture is used to also refer to the utilities we build to provide that data. Example.

For example, say you need the data from the example above to be persisted in your testing database to see whether a given function can fetch it correctly. You might create an EmployeeFixture with a save() function that receives an Employee and persists it.

Here the naming gets kinda muddy: Although we usually call these types of helpers Fixtures, what they actually do is provide the Fixture itself, they set up the testing environment.

Builder

A creational design pattern that lets you easily construct complex objects step by step as needed. The pattern allows you to produce different types and representations of an object using the same construction code.

They still end up producing a (in memory) Fixture. It's just a more useful and flexible way of getting it.

Example

Say you want to test how your application behaves when saving a user to the database if it has a faulty email address. You could go new User(name, age, id, email, maritalStatus, ...), but you really only care about the email for this test case. Plus, imagine name, age and maritalStatus all go through validations, so you can't just put whatever in those fields. It would be nice if you could use a sort of "default valid User" and just set a faulty email to it.

Something like User myUser = new UserBuilder().withEmail('doesntWork').build() with the Builder setting the rest of the properties to some irrelevant (but valid) default for you. Example.

Details

You'll often want to test behavior affecting semi complex entities.

You can use the Builder pattern to your advantage by having it set some sane defaults to, in our example, the User while also allowing you to customize the Entity at will.

This also gives you a centralized standard 'User maker' for your tests. So as long as this Builder accurately reflects the behavior of the production entity, you can be sure that your tests are relevant.

Plus, if something changes about your User (for example, the age now defaults to 18 if not set) you only need to apply the change in the Builder instead of parsing all the tests that use the User entity.

Object Mother

An Object Mother is a sort of fancy factory pattern, delivering prefabricated test-ready objects via a simple method call.

Again Mr. Fowler:

[...] it makes sense to have a factory object that can return standard Entities. Maybe 'John', an employee who just got hired last week; 'Heather' and employee who's been around for a decade. Object Mother is just a catchy name for such a factory

Example

You might find yourself using our UserBuilder in a few different tests just to end up creating the same type of User. What 'type of User' means depends on context but think of your typical Admin User, Guest User, New User, etc.

You can remove this duplication by abstracting the User creation into an Object Mother and just write testAdminUser = new UserMother.withAdminRole() or testAdminUser = new UserMother.admin() and call it a day! Example.

Details

Object Mothers differ from Builders in that Builders usually create dummy versions of domain Entities with no specific scenario in mind while Object Mothers are meant to Build more specific and complex instantiations of your domain Entities with the necessary data.

As soon as you find yourself creating the same kind of user in two different tests, go for an Object Mother.

You'll use it to reduce code duplication, increase test maintainability and encourage other developers to write more tests by making test objects super-easily accessible.

DDD Strategies

Sun, 13 Feb 2022 18:23:40 GMT

This is part of a series, start here!

We went over the Tactical concepts of DDD here. In this post, we'll cover the Strategic side.

When linking together the two parts for a more comprehensive picture, pay spacial attention to the concepts of Ubiquitous Language and Bounded Contexts, since these are the bits that keep the whole thing together.

Ubiquitous language

The idea is to use the same language everywhere possible, and let that language be dictated by the Domain and/or the Domain experts.

In an ideal world, we wouldn't have to map developer-speak to business-speak: we would all be using the same terms to describe the same things (God knows that almost never happens).

Let the code reflect the business language.

One of the advantages of following this approach is bringing together Domain experts, technical team, and other stakeholders involved in the project, with as little ambiguity as possible.

This is often not easy to do: in order to develop this Ubiquitous Language you need to understand the business and the Domain.

Developers also need to accept that they will often not be the ones in charge of naming things. Which frankly, is a very good thing IMO.

Bounded Context

A Bounded Context is a linguistic and/or conceptual delimitation.

The same concepts might have different implications in different contexts.

These contexts more or less reflect the business structure of the enterprise, or the problem domain.

Bounded contexts define isolated parts of the model with some degree of independence.

The isolation can be achieved by decoupling logic, code segregation, database segregation and also in terms of team organization.

The degree to which we isolate Bounded Contexts depends on the needs and realities of the business, and will often be context dependent.

You don't need tight, completely independent, future-proof, Bounded Contexts.

But you do need enough flexibility in your system to easily promote Modules to Bounded Contexts when needed.

Modules

Bounded Contexts are made up of Modules, which you can think of as mini-Bounded Contexts: Smaller semantic units that make sense within a greater common Context.

It's usually a good idea to have only one Aggregate per Module. The need for more than one might indicate a need for a new Module or for the Module to get promoted to Bounded Context.

So to be specific, you could manifest this in a structure like:

src
├── BoundedContext
│   ├── Module
│   │   ├── Application
│   │   │   ├── ApplicationService (Actions, Handlers, Commands, etc.)
│   │   │   ├── Repositories [Might also be under Domain]
│   │   │   ├── ...
│   │   ├── Domain
│   │   │   ├── AggregateRoot
│   │   │   ├── Entitiy/VO
│   │   │   ├── Domain Service
│   │   │   ├── Repositories [Might also be under Application]
│   │   │   ├── ...
│   │   ├── Infrastructure
│   │   │   ├── MySqlRepository
│   │   │   ├── ...
│   │   ├── ...
│   ├── AnotherModule
│   ├── ...
├── AnotherBoundedContext
├── ...
...

Apps

These should be the entry points to our Bounded Contexts.

They are usually called by API controllers, CLI interfaces, etc. and orchestrate use cases.

They lay outside our Bounded Contexts and call (directly or not) the Application Services to initiate use case execution.

There might be various Applications per Bounded Context, and their directory structure usually reflects this relationship.

So adding this to the previous example:

apps
├── BoundedContext
│   ├── UseCaseOneApp
│   ├── UseCaseTwoApp
│   ├── ...
├── AnotherBoundedContext
│   ├── AnotherUseCaseOneApp
│   ├── AnotherUseCaseTwoApp
│   ├── ...
src
├── BoundedContext
│   ├── Module
│   │   ├── Application
│   │   │   ├── ApplicationService (Actions, Handlers, Commands, etc.)
│   │   │   ├── Repositories [Either under Application or Domain]
│   │   │   ├── ...
│   │   ├── Domain
│   │   │   ├── AggregateRoot
│   │   │   ├── Entitiy/VO
│   │   │   ├── Domain Service
│   │   │   ├── Repositories [Either under Application or Domain]
│   │   │   ├── ...
│   │   ├── Infrastructure
│   │   │   ├── MySqlRepository
│   │   │   ├── ...
│   │   ├── ...
│   ├── AnotherModule
│   ├── ...
├── AnotherBoundedContext
├── ...
...

Something like that anyway, these structures are only here to better illustrate the relations between each piece.

Context Maps

Visual representation of a system's Bounded Contexts and how they relate to each other.

It helps understand the project as a whole (high-level design) as well as showing the communication patterns between contexts.

One of the main benefits of DDD is that it allows multiple teams to simultaneously work on different parts of the same system.

These 'parts' usually, though not always, come down to our Bounded Context and as such, building a context map will also show organizational issues, bottlenecks and team dependencies.

These are some ways Bounded Contexts might relate to one another:

Client - Server (Customer - Supplier)

As you might expect, one Bounded Context is upstream while another one is (or multiple ones are) downstream.

This makes them somewhat independent, but ultimately one of them will dictate the integration contract.

Anti-corruption layer

Another upstream/downstream relationship, where the downstream Bounded Context implements a layer responsible for translating upstream objects/structures into its own.

Mostly used to separate the old, legacy part of the system from a greenfield. It allows you to treat that part of the codebase as a 'black box'.

Shared Kernel

A more thoughtful version of your typical Utils directory.

Here, a common contract is defined and referenced by multiple bounded contexts.

The key to implementing a shared kernel correctly is to keep its scope as small and limited as possible.

Another less thoughtful but common approach is to have a shared kernel that holds only dumb components that are needed in multiple Contexts (or Modules), and only when they are needed.

So a classic example could be the Value Object for user IDs: Their structure will depend on our Domain, and they will most definitely be used all over the place, while not holding significant logic apart from basic validation. They are also very unlikely to change, so they make for a somewhat safe common dependency.

DDD Tactics

Sun, 13 Feb 2022 17:08:24 GMT

This is part of a series, start here!

From Eric Evans 2003 book, this approach to software design aims to couple the design of a system to the business domain it operates in.

This is to say, a system design should reflect the business logic for which it was created.

He broadly separates Tactical from Strategic design. In this post, we'll go over some concepts from the former.

Layers

Broadly speaking, four main layers are considered in this architecture:

User Interface: More or less equivalent to Boundaries in EBI.
Application: Partly in charge of the role of the Interactor, specifically related to use case orchestration.
Domain: In line with the Entities from EBI
Infrastructure: Simply in charge of persistence, messaging, and such.

If 'EBI architecture' doesn't ring a bell, you might want to start here.

Entities / Value Objects

Concrete representations of very basic Domain concepts.

They differ on mutability and identity.

Entity

An employee might change their role within the company, that doesn't make it a different employee.

Apart from their name, you'll likely identify them by some sort of ID, so a change in their attributes doesn't change their identity.

So we would say an employee is an Entity in our system.

Value Object

A phone number on the other hand does not change. Or rather, if it does, we are talking about a different phone number.

It wouldn't really make sense for a phone number to have an ID: The object is identifiable by its attributes.

Thus, things like phone numbers, email addresses, etc. are Value Objects (VO).

Rich Domain

While in general, Entities have IDs and VO don't, not all cases are so clear-cut as these. The context and Domain will dictate which to use in a given situation.

It's also important to remember that neither should be anemic: they should encapsulate as much logic as reasonably possible (usually all logic regarding their individual behavior).

Aggregates

Conceptual elements made up of multiple Entities and/or VO, which only have meaning or make sense together.

The concrete representation of this element is the Aggregate Root. This serves as a gateway to the rest of the elements enclosed within the Aggregate (Entities and VO).

The Aggregate Root should be the only way of accessing those elements, especially when modifying their state.

It's not hard to imagine such a structure getting out of hand. Prevent this from happening by:

Keep them as small as possible.
Allow for easy promotion from Entity to Aggregate Root, in case one of them grows significantly.
Aggregates should relate to one another by ID or directly through Services or Events to maintain scalability.

All logic pertaining multiple Aggregates should be delegated to a Domain Service.

Services

Stateless objects that perform Domain-specific operations that escape the boundary of the Aggregate. Based on their scope, there are two kinds:

Domain services: Executes logic that does not fit nicely within an Aggregate. Orchestrates interactions between multiple Aggregate Roots.
Application services: Orchestrates a Use Case, using Repositories, Domain Services, Aggregate Roots but always within its own Module.

To be clear, the scope should grow the further away we go from VO:

VO < Entity < Aggregate Root < Domain Service < Application Service < Module

If communication is needed between Modules, the Application Services should talk to one another Without accessing other Domain objects from different Modules.

Domain Events

A decoupled way for different parts of the system to indirectly interact with one another.

These usually materialize into a Pub/Sub structure:

Publisher

There are at least two ways of approaching who should be in charge of event publishing:

Aggregate Roots publish changes in their state directly.
Aggregate Roots register these events for the Application Service to publish.

Subscriber

Event subscribers look a bit like controllers, just limited in scope within our Domain.

They both ingest the primitive types of their respective input and use them to run the relevant use case.

Where controllers receive requests, subscribers receive events, but in essence you can think of their role as equivalent in practice, just with different scopes and implementation.

So for example, a generic Subscriber interface signature might look something like:

public interface DomainEventSubscriber<DomainEvent>

Where the implementation looks like:

public class DoStuffOnCustomEvent implements DomainEventSubscriber<CustomEvent>

CustomEvent might, for example, implement or extend from DomainEvent.

Repositories

They abstract concerns about data storage and other infrastructure.

Ideally, there will be one Repository per Aggregate Root, and it should only be called by the relevant Application Service/s as part of a use case orchestration process.

They usually take the form of a domain leaning interface with concrete implementations based in the specific infrastructure at hand. This is more or less borrowed from Ports and Adapters.

More to come

By now this is probably sounding like a big ball of jargon with not much of an architecture behind it, no real intention or plan.

We'll go over the Strategic side if DDD in a later post.

Pixar Driven Development

Sun, 13 Feb 2022 11:53:06 GMT

Weather you personally like them or not, it's hard to argue the impact that Pixar films have on the vast majority of people and the rest of the industry.

I'm sure you know who Buzz Light Year is, and any 90s kid can recognize the sound of the Pixar lamp from a mile away.

So how do they achieve this? And why on earth should software developers care at all?

Technical Similarities

There are a couple of techniques they follow that sound surprisingly similar to some of the best practices in software development.

See if any of the following sound familiar:

Not over yet

When writing the script, they don't just get the script done first and then continue with the production.

They keep writing and improving the script until the whole film is completely finished (as in actually being released).

So the development of the script is continuous, and the integration with the rest of the production team is constant and changing.

Refactor that scene

The opening of Toy Story 3 was re-written 60 times to get it just right.

No matter how good the first iteration is, you can almost certainly make it better.

Break the script down into Bounded Contexts

Once the script has the minimum basic structure and general shape, they will break it down into sequences: relatively small story arcs that, although all connected, are somewhat independent of one another.

They look for about 25 to 30 sequences per film and assign them to different teams which can work on them (and give each other feedback) concurrently.

What an Agile approach!

Fast Feedback

While all of this is happening, a storyboard is created. Not before the work begins, not after it's done, but while it's taking place.

This allows for a broader, more birds eye view of the project and gives a clear picture of what works and what doesn't.

Communication

Obviously, none of the above are possible without constant communication.

This is not a blanket statement: communication between teams, departments and individuals is absolutely key to making this work.

Digital in Nature

There are clear differences between live action movies and animated ones, mainly due to the digital nature of the latter. Carrying over the limitations of the real world to the digital space makes no sense. Why not take advantage of the differences?

In that spirit, they create a rough draft of the entire film. Everything from fake voice acting done by employees to very rough animations. Doing so allows them to modify animations, adjust the script and the sequences however the like.

After all, it's not like they need to record the whole movie with real actors and only then edit the scenes working with whatever they got.

It's really kind of strange the amount of influence the manufacturing industry has had over software development when you think about it.

Let them write

Another thing Pixar does well is hiring the right talent for the job at hand.

They understand not all animators and/or writers are the same nor are they always good at everything vaguely related to their profession.

Of course, this is easier said than done. In most cases companies have to work with what they can find and/or employ.

There is one thing to keep in mind here: Writers work with total creative freedom, no oversight, no deadlines. You can imagine this sounds like heaven to a writer.

This creates a positive feedback loop where Pixar seeks out the best talent and facilitates a work environment in which pretty much everyone would want to work, which in turn makes finding that talent a lot easier.

Collaboration

None of the movies are the result of one brave employee who knows best and has all the skills.

To set an example, the creation of Toy Story 3 involved all the top level creative people going together to a cabin in the woods. After just 2 days they had the basic premise and ideas for the movie.

These were 6 creative minded people coming together and reaching an agreement (in very little time as well). This trickles down to the rest of the production team.

Every moment of every scene is the fruit of the work and attention to details of hundreds of employees in constant collaboration.

Directors work with writers, writers work with animators, animators with actors, and so on.

It's a network of hard work, trust and harmony that makes the whole deal not only work, but work amazingly well and produces some of the most memorable films of my generation.

The rules

Let me leave you with an extract of Pixar 22 rules of storytelling. See if you can spot the similarities:

You gotta keep in mind what’s interesting to you as an audience, not what’s fun to do as a writer. They can be very different.
Trying for theme is important, but you won’t see what the story is actually about til you’re at the end of it.
Simplify. Focus. Combine characters. Hop over detours. You’ll feel like you’re losing valuable stuff, but it sets you free.
What is your character good at, comfortable with? Throw the polar opposite at them. Challenge them. How do they deal?
Come up with your ending before you figure out your middle. Seriously. Endings are hard, get yours working up front.
Finish your story, let go even if it’s not perfect. In an ideal world you have both, but move on. Do better next time.
Putting it on paper lets you start fixing it. If it stays in your head, a perfect idea, you’ll never share it with anyone.
No work is ever wasted. If it’s not working, let go and move on – it’ll come back around to be useful later.

Docker 102

Sun, 30 Jan 2022 16:14:56 GMT

We'll go over some every-day commands and files you'll use as part of your development workflow with docker.

Let's start by tying together the concepts from the previous post, with the ones we are about to see:

One builds and image (which might share a volume with the host machine) based on the definition found in a Dockerfile, runs it in a container and optionally composes multiple images together.

To make sense of this, let's take a closer look.

Dockerfile

A file that defines a docker Image, a blueprint of sorts. It will look something like this:

FROM alpine

RUN apk update
RUN apk add nginx
RUN echo Image created!

It contains a series of commands of the format INSTRUCTION arguments.

Keep in mind that every line is a new layer in the Image. So the order does matter.

Common Instructions

FROM

Sets the Base Image for subsequent instructions. In its most basic form, you'll see here what OS the Image is based on (Alpine Linux in our example).

A valid Dockerfile must start with a FROM instruction. Most commonly, this will be done by pulling an already existing image from the public repos.

RUN

Executes a command within the Container.

Create a directory? RUN mkdir. Update your system? RUN apk update. Install a dependency? RUN apk add dependency.

Plain and simple.

CMD

Default command to execute when running an image, 'what the image does'.

CMD ["echo", "This will be printed to the host system!"]

Only one is allowed per Dockerfile and whatever command we append to the docker run command will override this instruction. We'll take a closer look at the run command further down.

ENV

Sets environment variables, quite like you would in your .bashrc or .zshrc.

Useful if you need information to be set at build time, for later modification or reference at run time.

COPY

Copy files or directories from the host to the Container.

It works pretty much as you would expect:

COPY /source/host/path/afile /destination/container/path/

ADD

Pretty much like COPY, with the remarkable difference that ADD can also unpack tarballs and fetch files from remote URLs.

So you could say

ADD https://cool-github-repo.git /destination/container/path/

Pretty handy, but if you don't need the added functionality, prefer COPY.

VOLUME

Creates a sort of shared directory between host and Container.

So an instruction like:

VOLUME ["/opt"]

Would make the Container's /opt directory accessible from the host. In fact, it will actually 'mount the volume' somewhere under the host's /var/lib/docker/volumes/ directory.

Docker Build

Used to build an Image, use -f to specify the Image path (optional if it's located in the cwd) and -t to give it a name.

Those are options you can (but don't have to) pass. It does however need to get a context as parameter.

A build’s context is the set of files located outside the Container (local path or URL) that it will be able to refer to at build time.

This is more or less like the COPY instruction we saw before, with the difference being that the COPY command makes the hosts files available at run time, while the context makes them available only at build time.

Example

A build command usually looks something like this:

docker build -t my-docker-image -f src/Dockerfile .

Which is to say: 'Build an Image called my-docker-image based on the file src/Dockerfile with . (or cwd) as its build context'.

Docker Run

Tells Docker to execute the image as defined in the Dockerfile. If a command (or script) is appended, it will override the CMD instruction (if set). Its more or less like spinning up a VM.

It must take an Image as parameter, although its options make it possible to override nearly all the commands specified in the Dockerfile. This allows for a lot of flexibility.

Common options

-it

It allocates a 'pseudo-tty' and keeps STDIN open during execution. Useful if you want to be able to interact with your Container through command line.

--rm

Removes the Container from the hosts file system after execution.

-u

Changes the user and group (both of which are root by default) for the specific execution. Useful if your docker Image outputs files to the hosts file system (which can get a bit unwieldy if done so as root).

One neat thing you can do is make the docker Image run as the current host user (the one executing the command). You would do so by passing "$(id -u "$USER"):$(id -g "$USER")" as the parameter for -u.

--volume

Allows you to bind or mount directories from the hosts file system to the Container, or from one Container to another.

Takes an argument of the structure host-source:container-destination (container-destination must be an absolute path).

Example

docker run -it --rm --volume "$PWD":/data -u "$(id -u "$USER"):$(id -g "$USER")" my-docker-image useful-script.sh

Run the Image tagged as my-docker-image in a Container and execute useful-script.sh at startup. Keep STDIN open with a 'pseudo-tty' while running and remove the Container when done.

Also, mount the cwd ($PWD) of the host into the /data directory in the Container, and operate as the current host user (and group) instead of root.

Docker Compose

Utility for managing the build and run of one or more Images, and the relations (or dependencies) between them.

You might be able to achieve similar results by just running the Images separately from the command line. This is however a really easy and convenient way of building complex systems of interconnected and/or interdependent Containers in a reproducible manner.

So just like you would build an Image from a Dockerfile, you can compose a bunch of Services from a docker-compose.yml like:

services:
    my-cool-app:
        build:
            context: ${PWD}
            dockerfile: ./Dockerfile
        command: python app.py
        ports:
            - "5000:5000"

    mysql:
        image: mysql
    datadog:
        image: datadog
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock

As you might be able to tell by the general structure of the file, it is quite literally a sequence of 'Dockerfile-like' instructions enclosed within a sequence of 'Services' (which for simplicity we'll consider equivalent to Containers).

How it works

Define your app Image with a Dockerfile (and point to it from docker-compose.yml).
Define the Images for the rest of the Containers (Services) you need in docker-compose.yml.
Run docker compose up to start and run the Containers (docker compose down will stop and remove the ongoing processes gracefully).

You can pass it the -d flag to detach the process from the terminal and, just like with the build command, you can use -f to tell it where the docker-compose.yml is located.

Docker 101

Sun, 30 Jan 2022 16:14:52 GMT

Basic docker-related vocabulary (namely Image, Container and Volume) with a brief explanation. There are a lot of nuances I'll be glossing over.

We'll go into a bit more practical detail in the follow-up post.

Not a VM

It is usually described in detail how different Docker (or containerization more broadly) is a from a traditional Virtual Machine.

This is most definitely true: there are a number of low level and practical differences between these two technologies. However, I would argue that pointing out those differences does little to help understand the vocabulary and concepts around Docker.

In fact, understanding VM-related concepts like instance or virtual hard disks and the difference between a VM definition/configuration and a specific run of that VM can go a long way to help understand why docker can be so intricate and useful.

Image

An Image is what defines the default composition and behavior of a Container. Think of how the idea of a table is an abstract representation of a concrete, palpable table.

If we were talking about a VM, this would be more or less analogous to a VM's definition or configuration.

You define an Image in the corresponding Dockerfile, and build it via the command line (or pull it from the web).

In a VM, configuration and behavior would usually go separately. Best you could do are snapshots. An Image on the other hand, not only defines configuration (i.e. OS or installed software) but also behavior (i.e. commands to run, dependencies to install) of what we call Containers.

Container

A Container is a concrete instantiation of what's defined by the corresponding Image. Think of the palpable table from before.

This would be like a specific, concrete instance of your good old VM.

It's an isolated environment in which 'stuff' (might be your app, might be other things) happens. You can think of a Container as the concrete instantiation of an Image It gets created when you run said Image.

The technical differences between VMs and containerization are usually brought up at this point. Just know that Containers are stupidly efficient compared to VMs, and a lot more versatile.

Note

Technically, we say that Images run in Containers.

So Containers only hold that Image run. This is because one Image can be executed multiple times in parallel, so you might have a bunch of Containers running with the same Image but possibly with different processes and/or outputs or results.

However, I reckon it's easier to visualize for a newcomer as explained above.

Volume

Think of it as 'disk space' for a Container (or multiple Containers). It's where docker will operate, its very own file system.

Plain and simple, it's the equivalent of a virtual hard disk for a VM. It can be, and usually is, shared between multiple Containers and can easily communicate with (as in it's mounted to) the hosts file system.

Heuristics for Devs

Sat, 29 Jan 2022 16:17:57 GMT

Some heuristics I find useful at work. Something like this, but dumbed down and more concise.

These are not meant to be dogmas but general rules of thumb that should help you be a better dev. Ditch them as soon as they don't.

Fast Feedback

Both at small (TDD) and large (CD) scale. Don't just think it's OK, actually see if it is in practice.

Expect fuckups to occur, try to know about them fast.

Baby steps

Refactoring

Start small, even if small means apparently irrelevant, peripheral changes (variable names, directory structure, etc.).

New Feature

Find the smallest meaningful piece of the system and build up from there.

Complexity

The root of all evil. It is sometimes necessary, but often accidental. Be skeptical of the former and avoid the latter at all cost.

Divide and conquer

Split everything up as much as possible, even if it seems absurd. As long as it doesn't take more effort to split the task than to actually do it, the smaller, the better.

Decide as late as possible

Chances are, the later you decide, the more knowledge and experience you have. Anything that can be decided further down the road without causing a major setback, should.

Respect the Legacy

We know at least two things about all Legacy Code:

It works.
It makes money.

Respect Legacy Code and the people who wrote it.

Go fast, write great code

The only sustainable way to go fast is writing great code. The only sustainable way to write great code is going fast.

Start anew

Don't build the project of tomorrow with the crap from yesterday.

Listen to your gut feeling

Don't dismiss it with a 'meh, it works'. If you feel something could be better, make it better.

Be water my friend

Flexible with other peoples code, strict with yours.

Commits

Keep 'em coming and keep 'em small. Think of them as checkpoints, safe states you can return to. You can always squash them later.

Boy Scout Rule

Leave the system (code, docs, etc.) better than you found it.

Code

Readable

Code for other people, not for the CPU.

Naming is not relevant

It's extremely important.

Simplistic naming

Complex naming schemes might indicate inadequate modelling.

Expl!c!t language

When in doubt be explicit.

Boring, repetitive, predictable

Boring code is good code. No surprises, no 'WTF'.

Write code like a manual

Show what it does and how to use it. Hide how it works.

Syntax

Nouns for classes. Verbs for functions. Adjectives for interfaces.

Tests

Isolation

Not all layers need to be tested in isolation, or tested at all for that matter.

Test Behavior

Not code.

Coverage

Low test coverage suggests you might want to write more/better tests. High test coverage does not imply you are testing enough/properly.

Don't test someone else's code

Either trust the framework/library or chose another one. If you can't trust it, don't depend on it.

Design

Unix

Do 'one thing', do it well.

Abstract dependencies

Depend on abstractions, not concretions.

Extend existing behavior

Don't modify it.

Coupling and Cohesion

They are the same thing. The latter just has some though into it.

Demeter, don't ask

Units (classes, modules, functions) should talk to one another only if they share the same concern, and in such a way that keeps them ignorant of one another's inner workings.

Avoid changes in abstraction levels

They are hard to follow and indicate that something might need to be in a different layer.

Avoid generalizations

They are easy to build but a pain to remove.

Don'ts

Don't do a perfect job

Perfection is hardly relevant. A bad test is better than no test. Don't waste time and mental space on perfection.

Don't follow the rules

Follow the principles.

Don't be clever

Don't get fancy. Keep it simple.

Don't be an 'architect'

Bug-less code with meh-architecture is better than awesome architecture with buggy code.

Don't fear duplication

It's better than poor abstraction. The DRY principle is about avoiding duplicate logic or knowledge.

Duplicate lines might imply logic duplication, but they might not.

Don't get sentimental

No emotional attachment to code. Not to yours, not to others.

How to find

Wed, 26 Jan 2022 18:50:36 GMT

It not only helps locate files in the file system, it also allows you to manipulate what it finds.

Keep in mind

Not all find implementations are created equal. This post is best on the GNU implementation.

The basics

The find command has the following structure:

find [DIR] [OPTS] [EXP]

Where DIR is the directory in which you wish to search, OPTS are search options, and EXP is an expression by which to search.

The most basic practical use might look something like this:

find . -name 'config'

Which translates to "find anything named exactly config within cwd (.) and its contained directories".

This would print the paths (relative to where find is launched) for all files and directories that match the given pattern.

So for a file named config in a directory named config, it would output:

src/config
src/config/config

The options

For clarity, I've grouped them under three categories: Filters, Operators and Actions.

This separation should make them easier to reason about.

Filters

Technically called tests, these will tell find what 'sort of things' you are after.

-type

Tells find to only consider certain type of files:

-type f -> files
-type d -> directories
-type l -> symlinks

-name / -path

When asking for the name, find will look for a match with the last portion of the path, so after the last /.

When asking for the path, it will look for any path that exactly match the given string.

So if you want to find all files within a something directory, but there are many such directories under cwd, you would tell find to look for files with something as a part of their paths:

find . -type f -path '*something*'

As you can see, the EXP part of the command takes a reduced regex (which is why it only matches the exact string by default).

Here, we include the wild-card *, which will match for cwd/path/something/myFile and/or cwd/something/myOtherFile.

Both the -name and the -path filters have case-insensitive versions: -iname and -ipath.

-regex

Unlock the full potential of regex by using the -regex flag!

-mindepth / -maxdepth

Unless told otherwise, find will always search recursively throughout the directory structure. You can limit the scope of the command by setting its -mindepth and -maxdepth.

These filters take a number as parameter: 1 is the directory passed to find (cwd as . in our examples so far), 2 is its direct children directories, and so on.

So find . -maxdepth 1 -type f -name 'whoami' would look for a file named whoami only within the starting directory (ignoring its child directories).

While find . -mindepth 2 -type f -name 'whoami' would look for that same file in all directories under cwd, excluding cwd itself.

Operators

Mix, match or negate multiple searches:

-not -> negate following pattern
-a -> 'and' following pattern
-o -> 'or' following pattern

So find . -name 'hi' -o -name 'mom' would look for files named hi or mom.

Actions

There are a bunch of actions find can perform. By far the most common and useful one is -exec.

-exec / -execdir

You might need to further manipulate the output of a find command. Usually however, you'll find that the tools you want to use don't read from stdin but rather expect the input as params.

You could use xargs for this, but the find command offers a built-in alternative.

You can use -execdir [COMMAND] "{}" \; (or -exec) at the end of your command to achieve 'pipe like' functionality.

find . -name 'removeMe' -type f -execdir rm "{}" \;

Here, the [COMMAND] is rm, the "{}" is whatever find found (quoted to avoid shell expansions), and \; indicates the end of the -execdir command.

This example means 'remove all files named "removeMe" from cwd and its subdirectories'.

There are a couple of things to keep in mind here:

exec vs execdir

Although most of the examples you'll see around use -exec, this launches the [COMMAND] from wherever you ran find from.

Instead, use -execdir to run the command from the directory used as find's search parameters.

exec vs shell

When we say that "exec runs a given command", what we really mean is that find runs the exec application with the given parameters. exec doesn't really know about shell specific functions, aliases or piping or redirecting outputs.

This is why you'll commonly see something like -exec bash -c "your_cool_cmd 'params' {}"\;. This way, you can make full use of all of, in this case, bash's niceties.

`\;` vs `\+`

You might find some examples ending with a \+ instead of the \; shown above.

Simply put: \; tells -exec to run its command once per result, while using \+ the command will run only one time taking all results from find as a single parameter.

So \+ is more efficient but, depending on the use case, not always a good fit.

Common use cases

Remove empty directories

find . -empty -type d -execdir rm "{}" \+

Detailed results

find . -type f -name '*config*' -ls

Find all config files and print their properties as such:

6454785      4 -rw-r--r--   1 user     user          147 jan 24 12:56 ./tsconfig.json
6454787      4 -rw-r--r--   1 user     user           41 jan 24 12:55 ./config.yml
6427340      4 -rw-r--r--   1 user     user           41 jan 24 12:56 ./node-config.js

Path globs

Say you want the config files under the dotfiles/ directory but you don't know in which subdirectory they are.

find . -type f -path "./dotfiles/*/config"

Will output the config files somewhere within the dotfiles/ directory.

Exclude specific path

find scr/ -name '*.py' -not -path '*/site-packages/*'

Find all files ending in .py, while discarding the ones under site-packages/.

Print only the path to a file

find . -name 'carmen-sandiego' -printf '%h\n'

Prints the relative path (from cwd) to the results excluding their name.

Count stuff

find src/modules/UserLogin/ -type f -execdir wc -l "{}" \+

Will count how many lines are in each file under the UserLogin module, and print out a total as a bonus!

Fancy things you can do

Clean up

You are done 'legally' downloading music and want to clean up the left behind crap from your Music/ directory:

find Music/ -type f  -not -iname "*.mp3" -not -iname "*.ogg" -not -iname "*.wma" -not -iname "*.m4a" -execdir rm -r "{}" \;
# This is just an example, for simple use cases prefer something like rm !(*.mp3|*.ogg|*.wma|*.m4a)

You could be more concise with a well put-together regex, the point is that you can achieve this sort of things without it.

More Execdir

Result-dependent sed

find lady/ -type f -name 'gaga' -execdir sed -i 's:dance:Just Dance:g' "{}" \;

Replace all occurrences of dance for Just Dance in any file named exactly gaga within the lady directory.

Learn more about sed.

Remove trailing spaces from directories

find . -name '* ' -execdir bash -c 'mv "$1" "${1%"${1##*[^[:space:]]}"}" "{}"' \;

Yep.

Redirect output

find a_place/ -execdir bash -c 'do_something_cool_on "{}" > {}_processed' \;

Here we create a new file for each match processed by find.

Pipe output

find . -mindepth 1 -maxdepth 1 -type d -execdir sh -c 'ls -1 "{}" | grep -i -q "list|downloaded"' \;

Translates to: 'List all directories not containing a file called list or download only directly under cwd'.

How to sed

Sat, 11 Dec 2021 18:09:19 GMT

Dive much deeper into sed here and here.

Keep in mind

Not all sed implementations are created equal.

This post is about the GNU version as it has a lot of cool features that OSX, the various BSDs and Busybox variants are missing.

The basics

Sed stands for Stream EDitor, you can edit a stream like this:

echo "searching, seek and destroy" | sed 's/seek/destroy/g'

Or run the program directly on a file like this:

sed 's/seek/destroy/g' lightning.md
 |           |              |
sed      'do_this'       on_this

Let's break down the 'do_this' part: Sed will Substitute seek with destroy Globally[^1] within lightning.md.

[^1]: sed operates on a per-line basis, so when we determine the scope (Global in the example), we are referring to the scope within each line.

As is the case with most terminal utilities, it output to stdout by default, so no changes will be done to our lightning.md file. We can pass it the -i flag to make the changes 'in place', i.e. overwrite the original file.

Of course, we can also redirect its output to a different file with >.

So given a file like:

This line contains the word line twice
This line also contains the word line twice

If we run a sed command like sed 's/line/potato/' test-one-line.md, it would print the following to stdout:

This potato contains the word line twice
This potato also contains the word line twice

Notice how we didn't use the Global[^1] scope, so sed parsed only the first instance of line on both lines.

Using the -i flag it will overwrite the file instead of printing to stdout.

Quality of life

Always quote

Notice the ' in sed 's/seek/destroy/g'. This prevents any regex we might use from leaking out to the shell.

Extended Regex

By default, only basic regex is enabled, which enables you to use some special characters (like . or *) while others will be taken literally (like + or ?).

We can choose to use Extended regex by passing the -E flag to the command. Give this a try if you find your regex to not work as expected.

Learn more about regex here.

Pick a convenient delimiter

Usually, sed examples are shown with the / char as a delimiter.

For this to work, all / within the command need to be escaped.

You might find it useful to switch delimiter, especially when using sed on paths:

sed 's/\/bin\/bash\//\/bin\/sh\//g' -> sed 's:/bin/bash/:/bin/sh/:g' or sed 's_/bin/bash/_/bin/sh/_g'

Sed doesn't really care what you use as long as you are consistent with it.

Simple but useful

Remove all EOL spaces

sed 's/\s$//'

Remove all spaces at the end of all lines in the given file.

The \s is simply a way of representing white spaces. You can learn more about it here.

Delete all instances of word

sed 's/foo//g'

Delete all instances of foo.

You might be tempted to use something like s/.*foo.*//g to delete any line containing foo.

Don't, it will leave an empty line in its place. There is a delete command for this use case.

Only in nth instance

sed 's/lorem/ipsum/2'

Substitute lorem for ipsum only on the 2nd instance of lorem of every line.

Only from nth instance

sed 's/lorem/ipsum/2g'

Substitute lorem for ipsum from the 2nd instance of lorem of every line, until the end of the line.

The not so basics

Only on matching lines

sed '/^foo/ s/hi/mom/' file

Substitute hi for mom only on lines that start with foo.

For example, to migrate CSS classes from snake_case to camelCase, without compromising their properties, you might use something like:

sed -E '/\{$/ s_*(\w+?)_\u\1_g' file.css

Which only does the thing in lines that end with{.

If that looks like a bunch of random symbols to you, check out this post.

Between matching lines

You can apply a command only within a certain (variable) range:

sed '/#region/,/#endregion/s/foo/bar/' file.cs

Re-use the match

You can use & to represent the match:

echo "what a nice example, this is a cool program!" | sed 's/[nice|cool]/VERY&/'

Would output:

what a VERYnice example, this is a VERYcool program!

Case-insensitive

You can add an i at the end to make the match case-insensitive:

sed 's/foo/bar/gi'

Which means:

foo Foo -> bar bar

Negate matches

You can tell sed to do it's magic only on lines not matching a given pattern:

sed '/^foo bar baz.*/! s/foo bar/hi mom/' afile.txt

This would substitute foo bar for hi mom except in lines that start with foo bar baz.

Output replacements to separate file

You can write the lines affected by sed to a separate file with w:

sed 's_foo_bar_w replacementsFile' fileToModify

Substitute multiple lines

By default, sed uses \n chars as line delimiters, so multi-line substitutions are non-trivial.

Thankfully, the GNU version supports the -z flag, which tells sed to use NUL as the line delimiter.

This allows you to get a bit fancy and do things like:

sed -z 's_line one\nline two_merged lines one and two_g'

Consider however, that this means that ^ and $ now refer to the end of the file (NUL) instead of the line, which also affects the g at the end of the command.

Sadly, non GNU implementations of sed require a bit more 'sed-Fu' to achieve this.

Groupings and References

You can leverage the magic of Groupings and References to, for example, switch words around:

sed -E 's:([a-zA-Z]*) ([a-zA-Z]*):\2 \1:' file

Which means:

World Hello -> Hello World

Want a better use case?

sed -E 's_(.+?)\[(.+?)\]\(([^)]+)\)(.+?)_\1\2[^\3]\4\n\n\n[^\3]: \3\n_g' book.md

Let's take it apart:

Search

The 'search' part looks like this: (.+?)\[(.+?)\]$([^)]+)$(.+?).

The first and last groupings are pretty simple: 'whatever goes before/after the mess in between'.

That leaves us with \[(.+?)\]$([^)]+)$, which looks like a mess because we have to escape a lot of regular and squared parenthesis.

There are two distinct zones to this regex: \[(.+?)\] and $([^)]+)$.

The first means 'everything inside [squared parenthesis]', while the second could also be written like $(.+?)$ (which is pretty much the same as the other one, except for the different parenthesis).

Want to know why to use one instead of the other? Check out this post.

So we have four groups:

Everything before
Everything within []
Everything within ()
Everything after

Replace

On the other hand, the 'replace' part reads \1\2[^\3]\4\n\n\n[^\3]: \3\n.

We can see that there are two parts to this mess: \1\2[^\3]\4 and [^\3]: \3, with a bunch of line breaks (\n) here and there.

Notice also how the '[squared parenthesis]' are not escaped here.

The first part simply removes all the parenthesis from the match, while enclosing the third grouping in squared parenthesis and prepending it with a ^.

So text [looks like](a-link) more text becomes text looks like[^a-link] more text.

The second half repeats the previous behavior regarding the third grouping while adding it again after a : and a white space.

Taking into account the line breaks, text [looks like](a-link) more text becomes:

text looks like[^a-link] more text


[^a-link]: a-link

So we successfully turned Markdown links into Markdown references, without breaking the rest of the line.

Keep in mind that this command will hammer through images (![image-text](image-link)) as well. You might want to negate those matches with something like /!.*/!.

Also, this command won't behave nicely on lines with two or more links.

Was it a headache? Yes.

Was it more of a headache than doing it by hand on 400+ pages, heavily referenced book? Hell no!

Change cases

Here are some of the GNU specific goodies mentioned earlier:

\l Turn the next character to lowercase.
\L Apply \l until a \U or \E is found.
\u Turn the next character to uppercase.
\U Apply \u until a \L or \E is found.
\E End case conversion started by \L or \U.

So to give a simple example, you can ensure all headings in a .md file start with upper case letters by running this:

sed -E 's/^(#+) (\w+)/\1 \u\2/' cases.md

Which means:

## all caps -> ## All caps

Concatenate multiple commands

Sometimes doing everything in one go is a bit of a headache or actually impossible.

You can pipe sed commands using the shell (|) or adding the -e flag before them:

sed -Ee 's/(^#+) (\w+)/\1 \u\2/' -e 's/foo/bar/g' cases.md

This way, the file is read once and the commands are run one after the other on each line.

More than substitutions

Sed is a stream editor, so you can do much more than substitutions with it.

Delete

To delete any line containing the word vim you could do:

sed /vim/d file

For a more useful example, you could delete empty lines with:

sed '/^$/d' file

Or delete commented lines (starting with #) like so:

sed '/^#/d' file

Or negate the whole thing and delete everything but commented lines:

sed -E '/^#/!d' file

Print

You can tell sed to print the lines where replacements are made with p:

sed 's/foo/bar/p' file

You can also simulate grep-like behavior with something like sed '/re/p' file (familiar?), which would simply print all instances of re.

Of course, without the -i flag sed prints everything else as well, so you end up with the lines you are interested in printed twice.

Use the -n flag to make it behave as expected (which is to only print matching lines).

For a more practical example, you can print the lines between two matches:

sed -nE '/between-this/,/and-this/p' file

Append, Insert and Change

Append text on a new line after each line containing the given text:

sed '/foo/a\AFTER FOO' file

Insert text on a new line before each line containing the given text:

sed '/foo/i\BEFORE FOO' file

Change line containing the given text:

sed '/bar/c\BAR IS CHANGED' file

How to Regex

Tue, 07 Dec 2021 09:57:58 GMT

There are plenty of super useful CLI utilities, many of which you should already have in your system. To get the most out of them, some basic understanding of common regex patterns is needed.

Keep in mind that not all Regex engines are created equal and their implementations and valid patterns may vary a bit. However, the general concepts should be more or less the same.

The basics

Ranges

Use [] to match whatever falls within the given range.

[abc] ➡️ 'a' or 'b' or 'c'.

[a-z] ➡️ Any char between 'a' and 'z'. It may or may not include diacritics.

[a-zA-Z0-9] ➡️ Any alphanumeric char either lower or upper case.

You can negate them with ^:

[^a-z] ➡️ Any char not between 'a' and 'z'.

The Dot

Use it to match any char, usually except new lines.

. ➡️ Any one char.

.. ➡️ Any two chars (not necessarily the same ones).

Multipliers

Use them to match any number of the previous item.

a+ ➡️ 1 or more instances of a

ab+ ➡️ a followed by 1 or more instances of b (so ab, abb, and so on)

.+ ➡️ Any char 1 or more times

.* ➡️ Any char 0 or more times

.? ➡️ Any char 0 or 1 times

Greedy vs Lazy matches

What would you expect to happen if you pass a string like <body>Banana</body> through a regex like <.*>?

You might be surprised to find that it likely would not match <body> nor </body>. In fact, it would most likely match the whole <body>Banana</body> instead.

By default, most regex engine's + and * multipliers are greedy, which means that they will try to match as much as possible.

A lazy match is probably what you want in most cases, and you usually get that by adding ? to the multiplier: so using <.*?> instead will match <body> and/or </body>.

If you want to get real fancy you could also use <[^>]+> to achieve this, which should be understandable by this point. It's usually more efficient, but be careful, regular expressions get out unreadable real fast.

So remember, if you are having trouble with .* (or .+), try using .*? (or .+?) instead.

Numbered Multipliers

Instead of matching any number, these match a given number or range of numbers of the previous item.

a{5} ➡️ 'aaaaa'.

a{1-5} ➡️ Between 1 and 5 consecutive 'a'.

What's cool about them is that they can behave like a more interesting ? multiplier:

a{3,} ➡️ 3 or more 'a'.

The not so basics

Short-hands

Regex can get hard to write and read, and there are certain structures we often want to match against.

To make our life easier, we can use short-hands (if your regex engine supports them):

\s ➡️ a whitespace.
\S ➡️ anything but a whitespace (opposite of \s).
\d ➡️ a digit (0-9).
\D ➡️ anything but a digit (opposite of \d).
\w ➡️ a 'word' char (shorthand for [a-zA-Z0-9_]).
\W ➡️ anything but a 'word' char (opposite of \w).

Anchors

You might need a regex to only match at the beginning or the end of a line. For this, we use anchors like ^ and $:

^ ➡️ Start of the line.

$ ➡️ End of the line.

\b ➡️ Word boundary (beginning or end of word).

So, for a regex like \bFOO$:

FOO in What a nice line of text BAR FOO would match.

FOO in What a nice line of text BARFOO would not.

Multiple matches

Just like an if statements, you can match for more than one expression:

foo|bar ➡️ Would match either foo or bar.

Escaping special chars

What if we want our regex to match some of the special chars we've seen (like $, [ or +) literally?

We would need to escape them by putting a \ in front of them.

If we take our previous example and escape the $: \bFOO\$:

FOO in What a nice line of text BAR FOO$ something else would match.

FOO in What a nice line of text BAR FOO would not.

If you come across a scary looking, unreadable regex this is probably the main culprit. Don't let the \ scare you!

Grouping and References

One neat trick that most regex engines will allow you to do is grouping parts of the match and referencing them later in the regex.

One regex can have multiple groups and these get referenced by their number (starting with 1).

You surround the group in () and reference it with \ followed by the group's number:

(foo)-(bar) \2\1 ➡️ Will match foo-bar barfoo (notice the spaces).

If you know how sed works, you can probably imagine this can save a lot of headaches.

Negations

You can negate parts of your regex using lookarounds.

Say you want to match all instances of foo followed by anything but bar, followed by baz. So for example, we want foowhateverbaz to match but not foobarbaz.

A lookahead like foo(?!bar).+?baz would do just that: It negates the part of the regex between parenthesis and preceded by ?!.

It simply means 'not followed by (?!this)'.

Similarly, you might want to go about this the other way around.

If you want to match all instances of foo except when it is preceded by bar, you could use a lookbehind like (?<!bar)foo.

So This whateverfoo is weird would match.

But This barfoo is weird would not.

It simply means 'not preceded by (?<!this)'.

Both lookaheads and lookbehinds can be used to match a pattern while negating another one. Which one to use just depends on whether you want to negate something before or after something else.

Software Architecture for Noobs

Sat, 27 Nov 2021 12:14:39 GMT

Some posts overviewing some code architecture patterns I've seen being used in one way or another either directly or indirectly through the ideas they bring to the table.

Best to read them in order, since newer concepts are often better understood with the previous ones in mind.

These posts are not about low level architecture nor whiteboard-like systems design. Rather we go over different ways to structure code to make it (hopefully) more maintainable and easier to work with in the long run.

Apart from reading the source material for each architecture, you can go in much more detail in this great series of posts.

Seriously go check that out, it's great. In the meanwhile:

Have fun!

Clean Architecture

Tue, 23 Nov 2021 17:20:20 GMT

This is part of a series, start here!

This is Uncle Bob's attempt at synthesize previous architectural patterns and concepts.

Based on a common thread between Ports & Adapters, Onion, EBI, he wrote an article as 'an attempt at integrating all these architectures into a single actionable idea'.[^1]

[^1]: From the original article

High level

The original article starts with a diagram similar to this:

Project structure

At face value, the diagram seems to suggest a project structure as such:

Entities
├── Tenant
├── Landlord
├── Rent
Repositories
├── TenantRepo
├── LandlordRepo
├── RentRepo
UseCases
├── PayRent
├── RequestRent
├── RequestRepair
├── CalculateRent
...

There are however a couple of shortcomings here:

Low cohesion: Modify one Use Case, and you'll have to change code in three different modules
No clear purpose: A newcomer would have to dig through the directory structure to know what the application is for

Borrowing some key concepts like Bounded Context from DDD, the previous system can be represented as such:

Tenant
├── Tenant
├── TenantRepo
├── PayRent
├── RequestRepair
Landlord
├── Landlord
├── LandlordRepo
├── RequestRent
Rent
├── Rent
├── RentRepo
├── CalculateRent
...

Low level

There's a pretty useful diagram in the slides Bob uses in his conferences.

Cool, but a bit overwhelming. Let's start with Entity and Interactor[^2] and build the diagram with concepts from other notable architectures.[^3]

[^2]: From the EBI architecture [^3]: Start here

Think of the Interactor as the implementation of a use case of the application.

Thanks to Ports & Adapters, we know that a use case should be defined as an abstraction to ensure inwards dependency.

So if a use case is an abstract concept of 'what needs to be done', the Interactor is the concrete implementation of 'how exactly it will happen'.

Entities usually require some sort of persistence, we can use a Driven Port/Adapter pair for that.

Let's also represent the actor that will use the Driver Port from before, as well as the data structure (DTO) that will be shared between it and the Interactor.

The controller could just as well be a CLI or a GUI.

Using a DTO here allows us to avoid exposing the Domain Entities outside the boundaries of the application. Speaking of boundaries.

The red line marks the limit of our application logic and separates it from the pieces of the system that are necessarily coupled to the Infrastructure.

Now suppose that, when something happens in the system, we want to notify the user or update a UI element.

Or using Uncle Bob's terminology:

With this, we are back to his original diagram.

Let's review how the flow of execution would go for an incoming HTTP request:

The Request reaches the Controller
The Controller:
1. Dismantles the Request and creates a Request Model with the relevant data
2. Through the Boundary, triggers the Interactor
The Interactor:
1. Finds the relevant Entities through the Entity Gateway
2. Orchestrates interactions between entities
3. Creates a Response Model with the relevant data and sends it to the Presenter through the Boundary

The diagram in the lower right corner of the first image on the original post might help visualize what's going on.

Swap 'Use Case Output/Input Port' for 'Boundary'.

Onion Architecture

Sun, 10 Oct 2021 14:02:46 GMT

This is part of a series, start here!

It helps to think of this as an update to Ports & Adapters that brings more fine-grained control over the Application Core.

Bringing back the layers

When going over Ports & Adapters we saw that it sort of got rid of the layers. More specifically, it only (implicitly) left two of them:

An external layer (with Adapters and the relevant infrastructure).
An internal one (with pretty much everything else).

As you might imagine, especially in bigger applications, having most of the code bundled in a single layer can get... Complicated.

That's where Jeffrey Palermo's Onion Architecture shines.

In a nutshell, it's a more detailed specification regarding how to organize what remains of our code after defining where the boundaries (Ports & Adapters) are.

Let's have a look at these layers, starting from the core of the Onion.

Domain Model

This would be our core business Entities, enriched with their corresponding rules and logic.

[...] the state and behavior combination that models truth for the organization. [^1]

[^1]: From Palermo's original post.

This part of the system should change only if the most essential business rules change, which doesn't usually happen (if ever).[^2]

[^2]: When these changes occur, they are usually accompanied by structural, organizational changes.

Domain Services

Similar to how Interactors managed the interactions between Entities in the EBI Architecture, all Domain logic involving multiple Models will go here.

Whatever business logic doesn't fit the scope of a single Entity (or value object for that matter) belongs here.

Application Services

Like Domain Services orchestrate the interactions between multiple Models, Application Services orchestrate the interactions between multiple Domain Services.

Here we'll also find the Ports and use cases definition from Ports & Adapters, right at the boundary of our application.

Inwards dependency

As you can see, this design enforces the same 'direction of dependency' as Ports & Adapters, only this time it is also applied to our own.

Not only does our infrastructure depend on our Domain (and not the other way around), but the layers within our domain also depend on whatever layers lay beneath them.

This way, we end up with an independent core that can (and should) be compiled, executed and tested independent of its outer layers.

We couple towards the center.

Pair programming doesn't have to suck

Fri, 08 Oct 2021 18:09:55 GMT

A while ago an interesting debate occurred between my coworkers regarding pair programming. We all follow this practice at LeanMind as much as possible, but some of us do so pretty much 24/7 while others have a more selective approach. This led to an interesting conversation that made each other more aware of the way others view this practice.

When is pairing not fit for purpose

It's only fair for me to point out some valid considerations regarding what can go wrong with pairing, before arguing why you should still do it.

Productivity

One might argue that, if the task is simple enough, having two developers tackling it is a bit wasteful. Sure, complicated o critical problems might require group collaboration but maybe if we could get more done on our own, we should.

This will probably resonate more with business people than with developers, we'll see why in a bit.

It aint' easy

Some people are more social than others. Even the same person can be more social one day than the next, and that's okay. There's an argument to be made for not feeling pressured to pair. It shouldn't be a mandatory thing, but rather a scenario to strive for.

Take a break from time to time, change partner, ease yourself into it. Computer people are not really known for their social skills, nobody expects an introvert to want to spend 8 hours straight with a co-worker.

It might be healthier to think of pair programming as an objective in some cases, rather than 'a thing you just do'.

Everybody is different

Just like with any other human interaction or relation, some work and some don't. This can be harsh and hard to deal with (for some more so than for others).

Some get distracted while being the 'navigator' and wander around not really helping a lot. Some want things done their way and can't seem to listen while being the 'driver'. Some may not be willing to swap out the role at all.

Dealing with these situations is quite simply not developing software. And you can bet that your average developer has zero interest in resolving issues like these (although one could argue that they probably should).

Juniors might hamper Seniors

No matter how you define 'Juniors' and 'Seniors' (or if you just don't care much for the terms at all), people with less knowledge and/or experience in any particular subject will inevitably take longer to understand things and have a lot of questions.

It's easy to see yourself pairing with someone more or less at the same level you are, but it can be annoying to have someone stopping you at every step to ask you basic (although very valid) questions.

One could consider pairing partners based on experience, although there are very valid reasons to mix and match experience levels.

Pair as much as you can

Now for the good stuff.

Beware the bias

I've been very lucky in this regard: I've written some code on my own, but I have never worked alone.

I started a while back in a team that felt more like a bunch of friends than a work group. Not that we weren't getting things done (we were even helping other teams on a regular basis), we just worked in a very friendly and enjoyable environment.

We laughed a lot and were there for one another, it really felt like hanging out with some friends.

Just keep in mind that someone less fortunate might not feel the same as I do about pairing, which is totally fair!

Don't care much about productivity

It might be productive in the short term to have two developers doing two different tasks, but is it also in the long run?

When both codes need to be merged, but they seem like they come from two different worlds, is pairing still less productive? What about when one of them spends two days looking for a bug that could have been easily found in half an hour by just having double the eyes looking at the code? Don't you think you would benefit from a helping hand when you hit a dead end? Wouldn't you always prefer to know early if there is a better way? Do you really think that having some help or a different opinion makes you less productive?

Sure you could just ask for help when needed but that assumes everybody is mature enough to explicitly and openly admit they need help.

This is why I mentioned that business people might focus more on this than developers: The usually just don't see or understand the present and long term differences between clean code and spaghetti code. And that's not a criticism, it just isn't really their job to tell one apart from the other. They shouldn't be expected to. If anything it's on developers for not knowing how to present these issues properly to the business.

Majority of us are noobs (and that's fine)

Uncle Bob has repeated this fact time and time again: Developers more or less double every five years or so. This means there are always a lot more inexperienced developers than experienced ones. There is an obvious need to take Juniors and their learning process into account when thinking about anything related to software development.

It's kind of strange that 'shadowing' isn't just standard practice in the industry, but if it were, it would benefit greatly from pairing. I can't think of a faster way for someone to get the hang of a particular language, framework, methodology or project than for him to pair with an experienced developer.

Sure you can do test projects on your own (and it actually helps a lot), but the level of insight and 'know how' you get from watching a Senior hands on with the code, asking questions and receiving corrections and guidance in real time is just on a completely different league.

Of course, that will hamper his speed (quite a lot at the beginning), especially if the more experienced one actually cares about his pairing partner getting better on the job at hand. To this I would say...

Developer 'flow' is no good

It sure feels good: Being on you own, doing your thing at light speed. Just ripping through the code and solving tickets left and right. You look at your code after a week and have no clue what you wrote, why you wrote it that way or if there was a better way to do it. You overengineered half of it and didn't stop for a moment to see what the rest of the codebase looks like.

I would argue that being in this state of 'Flow' is only useful if you work on your own in a small and/or unimportant project. It seems much, much better in the long run (and for the rest of the team) to have someone stop and question you every once in a while. This will make you consider why you are doing what you are doing (and if you are making it unintelligible for the rest of us). If you can explain it simply you can be sure that you know what you are doing (and you will have taught something new to your peer, making both of you better off).

Nobody wants to be distracted while solving a problem, but when doing things out of inertia we can all benefit from an external voice of reason.

Good for the mind

Especially with remote work becoming the norm, it's not really healthy to spend 8 hours a day solving problems under stress on your own. I know, it sounds strange, but we are supposed to be around each other. We are not built to be alone.

You might think you are better off on you own, but I can guarantee you will feel better with good company. Pairing can be fun, relaxing and morally helpful. Don't isolate yourself for no reason.

Enforces soft skills

I've been pleasantly surprised with how important soft skills are in software development and I can see a couple of clear benefits of pair programming in this regard:

Leave your ego at the door

If done well, pairing will force you to listen to others, overcome your ego and accept what's best for the team. You can never have too much of that.

Leave your personal problems at the door

Not only will this help you compartmentalize trouble (which is always good), you will have to decouple your mind from your personal issues. That's just free counseling right there.

Forces professional behavior

A doctor doesn't just behave badly or work alone when he is 'not feeling it'. He picks himself up and gives his best. Why should we not do the same?

Soft skills Hard truth

The reality of soft skills is that there is no method to it. You can watch what you say and how you say it. You can be more assertive and sensitive. But at the end of the day, you learn soft skills by throwing yourself into the fire.

You can't 'think your way' through human interactions. You deal with them as they come, in the context you struggle the most, screw up as many times as you need in order to get a feel for it and acquire that new skill (just like you would when learning to ride a bike). Working alone won't help much with this.

When in doubt, program in pairs. It's good for everybody

Ports & Adapters

Sun, 03 Oct 2021 18:11:44 GMT

This is part of a series, start here!

This architecture, created by Alistair Cockburn in 2005, builds on the layer-based designs that came before it, putting a much bigger emphasis on the outer boundaries of the system.

Everything is I/O

Focusing on I/O to and from our application will help understand this architecture.

Try to see everything but the most core business logic as an I/O device.

Database? I/O device.
Messaging service? I/O device.
GUI? I/O device.
The web? I/O device.

We ditch the layers as we used them before and swap them for puzzle pieces of sorts, where our core business logic communicates with different types of I/O devices.

Keep this image in mind going forward.

Port

There is a really nice definition in the post I'm leaning on to write this:

A port is a consumer agnostic entry/exit point to/from the application.

It's a piece of our core application that vaguely defines how we communicate with any given 'I/O device', without caring about the implementation detail.

If we use a persistence port as an example, it should stay unchanged no matter if we use MySQL, MongoDB, or whatever else.

It just knows we need to persist and what parts of our domain need persistence.

In this example, it will most likely take the form of an Interface or a Trait: It defines that we need a persist(user) functionality, but not how to implement it.

Adapter

A module that adapts to or from a specific port.

So if we are using an SQL database, its adapter will implement our persistence port and translate our domain to whatever SQL commands needed to read from or write to the database.

This, unlike its port, will not stay if we swap SQL for MongoDB. Our domain ends where our adapter starts.

This way, we decouple our domain from any particular technology we might be using.

This thought process should sound familiar, as it is, in practice, how you often implement a Repository from DDD.

Far Beyond Driven

You might have noticed that the example of a persistence port/adapter doesn't quite fit all types of infrastructure.

There is a difference between our previous SQL example and an REST API port/adapter: One is driven by our application while the other decides or drives what our application does.

source

Think about your typical REST API controller function, for example a GET controller that calls a GetUser use case.

If you follow the D in SOLID, your controller should depend on that use case abstraction (interface), which will be implemented somewhere else inside the hexagon.

So the REST controller (Adapter) would use the use case (Port) which will be implemented within the boundaries of the application as part of the Business Logic.

Inwards dependency

By following these guidelines, we end up with the infrastructure related code depending on the Domain/Business Logic and not the other way around.

The Domain doesn't know a thing about a web UI, API calls, CLI, SQL, Redis, etc.

It knows its own rules and logic, it knows what it can do (driver ports) and what it needs to function (driven ports).

This allows us to focus on what actually matters, disregarding external libraries and other infrastructure as nothing more than 'a thing that fits (or uses) the ports of the application'.

It also has the (quite nice) added bonus of making the system really easy to test. Just swap the real adapter for a test one and you are good.

Considerations

This enforced isolation between our application and its dependencies is the most important idea Ports & Adapters has to offer.

By structuring an 'us vs. them' approach, it materializes the idea of 'core business rules', which while present in previous architectures were not so clearly delineated.

Of course this only considers two concentric 'layers': Our code and the rest of the world. This will be expanded upon by Jeffrey Palermo three years later.

Hexagonal?

So what's the deal with the hexagon? Why is it called Hexagonal Architecture? What piece is repeated six times?

The shape should evoke the inside/outside asymmetry rather than top/down or let/right. Then Square is not suitable. Pentagon, Heptagon, Octagon, ... Too hard to draw.

So Hexagon is the winner.

Well, that's a bit lame. Don't get me wrong, hexagon is bestagon, but it might as well have been a circle.

'Ports & Adapters' is just a better name: it's accurate and focuses your attention to what really matters and what defines this architectural design.

EBI Architecture

Sat, 02 Oct 2021 19:26:16 GMT

This is part of a series, start here!

Created by Ivar Jacobson in 1992 as EIC (Entity-Interface-Control) and renamed by Uncle Bob to Entity-Boundary-Interactor (EBI), It's a more back-end-focused version of the MVC architecture that came before.

Entity

It consists both of the 'domain' entity and all behavior strictly related to it.

So in a very simple example, the 'Dog' Entity would hold the data regarding its breed, fur color, health, etc. as well as the logic required for it to behave as expected: a walk function, a bark function, etc.

Already back in '92, Jacobson was warning about anemic entities and god objects.

Boundary

The I/O interface of the system.

Think of it as the 'fence' of the domain (It's in the name).

All interaction between your code and the user on one side (GUI, CLI, etc.) and the infrastructure on the other (persistence, event queue, messaging, etc.) should be handled by this guy.

You might want to make it an interface and call it a Port, but that's still 13 years away.

Interactor

The ones in charge of validating I/O between Boundaries and Entities.

More important than this, they will be managing the interactions between Entities. In practice, this means that all logic not belonging to or fitting in the Entities will end up here.

In our previous dog example, this role would be taken by the owner. Dogs don't play dead on their own, the owner (or trainer) needs to give the order.

Stretching the example a bit, dog to dog interaction is usually mediated by one or more humans (assuming they are pets).

The same applies here.

Of course, one interactor will often not be enough. You should end up with about one interactor per use case.

For every abstract operation a user could perform on your system, there should be an interactor ready to handle the use case.

MVC & MVVM

Sat, 02 Oct 2021 17:40:28 GMT

This is part of a series, start here!

After suffering the consequences of mashing everything together, someone got tired and gave us MVC.

Model-View-Controller

In 1979, Trygve Reenskaug came up with this architecture as a way to solve some of the issues related with writing code for the machine and not for the human.

This was our first attempt at 'separation of concerns' and was guided by the following logic:

Separate data, logic and presentation.

More accurately, it contemplated three basic units (or layers):

Model: The business entities/logic.
View: The UI.
Controller: The 'procedural' or 'application' logic. It would guide all interactions between the previous two.

Would look something like this:

If done right you would end up with multiple View-Controller pairs per screen, since each view should ideally only be responsible for a single piece of the UI (widget, button, text field...) and talk to a Controller if data was needed.

This also means that if multiple Views needed the same data, they would communicate with the same Controller.

Notice here that the Controller reacts to the View, and manipulates the Model as a consequence.

The View would then react directly to the events triggered by the Model, updating the UI accordingly.

This forces a one directional flow in which the user interaction with the View determines what a specific Controller should do with the Models, which in turn updates the View directly.

This design is often still used in the front end (which is no surprise since it was created in the context of GUI reliant desktop applications).

All things considered, this approach leaves us with a couple of issues:

The View-Controller relation can get messy fast.
Having multiple Views per Controller can get even messier, since each View ideally corresponds to a piece of the UI.
The View is coupled directly to the Model.

Model-View-ViewModel

That's what John Gossman tried to solve around 2005.

Basically, he called the old Controller ViewModel and made it also responsible for the events fired by the Model.

So now, the flow of execution would stop in the ViewModel both on its way to the Model and on its way back to the View.

This new and improved 'Controller' now had the power to manipulate Models as well as to implement View specific logic (which made the View much simpler, so designers to focus on... Design).

Now View and ViewModel must have a 1:1 relation.

As you might imagine, once the Model operations get complicated and/or there's a bunch of different data to manage or present together, it's easy to still wrangle things together.

The EBI architecture attempts to solve this.

Blockchain with Node

Sun, 26 Sep 2021 14:03:46 GMT

We'll build a very simple POC blockchain with four distinct building blocks. Check the repo for reference!

Transaction

The only things really needed for a transaction to take place are:

Who sends the money?
Who receives the money?
How much money?

export class Transaction {
    constructor(
        public amount: number, 
        public sender: string, 
        public reciever: string
    ) {}
}

Block

Think of a block as a group of transactions with a time stamp and a reference to the previous block:

import {Transaction} from "./Transaction";

export class Block {
    constructor(
        public previousHash: string,
        public transaction: Transaction,
        public timeStamp = Date.now()
    ) {}
}

As discussed before, we need to hash our blocks. To do this, we'll use the crypto library:

get hash() {
    const hash = crypto.createHash("SHA256");
    const block= JSON.stringify(this);
    hash.update(block).end();
    return hash.digest("hex");
}

We create a SHA256 hash, add our stringified block to it, and output a hexadecimal representation of it. This is what will link multiple blocks together.

Chain

Since there should only ever be one chain, we can represent it as a Singleton. We also know that it will host our blocks and that these need to know about their predecessor, so we can start by:

import {Block} from "./Block";

export class Chain {
    public static instance = new Chain();
    chain: Block[];

    get lastBlock() {
        return this.chain[this.chain.length - 1];
    }

Let us also initiate the chain with a first block:

constructor() {
    this.chain = [new Block("", new Transaction(10, "God", "Satoshi"))];
}

We'll also need a way to add blocks to the chain. Something like:

addBlock(transaction: Transaction) {
    const newBlock = new Block(this.lastBlock.hash, transaction);
    this.chain.push(newBlock);
}

This will work, but it allows anybody to send anything anywhere. We need verification, signatures and keys. We need a wallet.

Wallet

At a basic level a wallet is just a wrapper for a key pair, not unlike what you use to secure an SSH connection.

export class Wallet {
    public pubKey: string;
    public privKey: string;

    get address(){
        return this.pubKey;
    }
}

Again, we'll use the crypto library to generate the key-pair. Since we want two ways encryption, we'll use RSA.

import crypto from "crypto";

constructor() {
    const keyPair = crypto.generateKeyPairSync("rsa", {
        // standard rsa settings
        modulusLength: 4096,
        publicKeyEncoding: {type: 'spki', format: 'pem'},
        privateKeyEncoding: {type: 'pkcs8', format: 'pem'},
    })

    this.pubKey = keyPair.publicKey;
    this.privKey = keyPair.privateKey;
}

Payment

As explained before, we don't want to expose the private key nor the encrypted data. Rather, we use the private key to sign the data.

This way, the public key can be used to verify the data's integrity without exposing the private key.

pay(amount: number,senderPubKey: string) {
    const transaction = new Transaction(amount, this.pubKey, senderPubKey);

    const sign = crypto.createSign("SHA256");
    sign.update(transaction.toString()).end();
    const signature = sign.sign(this.privKey);

    Chain.instance.addBlock(transaction, this.pubKey, signature);
}

Here we create a transaction and use crypto to sign it with the payer's private key. In a real world scenario we would verify the signature sending the block to various miners (nodes) over the web, but it's good enough for our purpose.

We'll have to update our addBlock function to ensure the block is verified before being added to the chain:

addBlock(transaction: Transaction, senderPubKey: string, signature: Buffer) {
    const verifier = crypto.createVerify("SHA256");
    verifier.update(transaction.toString());
    const transactionIsValid = verifier.verify(senderPubKey, signature)

    if (transactionIsValid) {
        const newBlock = new Block(this.lastBlock.hash, transaction);
        // Proof of work!
        this.mine(newBlock.proofOfWorkSeed);
        this.chain.push(newBlock);
    }
}

Mining

As mentioned before, we'll use of the concept of "Proof of Work" to our advantage.

Let's add a POW seed to our Block class:

proofOfWorkSeed = Math.round(Math.random() * 42069666);

Now we can add a mine function to our Chain such as:

mine(proofOfWorkSeed: number) {
    let solution = 1;
    console.log("⛏️ ... ⛏️ ... ⛏️");

    while (true) {

        const hash = crypto.createHash("MD5");
        hash.update((proofOfWorkSeed + solution).toString()).end();

        const attempt = hash.digest("hex");

        if (attempt.substr(0, 4) === "9999") {
            console.log(`Done! Solution: ${solution}`)
            return solution;
        }

        solution += 1;
    }
}

Here we look for a number that added to the seed will produce a hash starting with four consecutive nines. The specific implementation here doesn't really matter, as long as it is somewhat costly to compute.

After finding the correct answer (which will be different for each block) it will return it for the other nodes to verify (which is much easier than to solve), or at least that is what would happen in a real world application.

Run it

Hopefully you've ended up with something like this.

You can add your own finishing touches and run npm run start to see it in practice.

There you go, your very own blockchain!

Blockchain 101

Sun, 26 Sep 2021 14:02:46 GMT

General structure

As the name implies, a blockchain is nothing more than a chain of blocks, where each block contains a collection of transactions and is connected (chained) to the previous block. Kinda like a Git repo is made up of a bunch of commits linked to one another, each containing a bunch of changes. Only in this case you could only commit new code to the repo, no rebasing or amending.

The chain can be (and usually is) distributed among multiple machines, thus it's decentralized nature. This begs the question, what if I mess around with my copy of the chain? Can I give myself a million coins just like that?

Proof of work

To ensure the validity of the transactions (and that the blockchain has not been tampered with), all parties involved in the network need to agree on one 'correct' blockchain.

A common way to face this problem, although not the only one, is by using a proof of work system.

Simply put, the transaction finds its way into a block, which is then validated by solving an algorithm.

The lucky first will receive a portion of the transaction as a payment while the others will verify that the solution is correct (this is why you can't just create coins as you wish).

The algorithm needs to be quite expensive to compute for this to make sense, but we also want to verify and compare the results easily. For this, we use hashes.

Hashing

A Hash is a "one way cryptographic function". Which means we can use it to encrypt data, but getting the original data from it's encrypted form is impossible.

This is useful because it makes it impossible to tamper with a computed block once it's hashed, while making it easy to compare it to others.

Plus, hashing has the added benefit of producing standard sized blocks.

Transacting & Signatures

The last piece of the puzzle are transactions.

To send and/or receive coins, each user needs a Wallet. This is little more than a key pair (a public key and a private key).

A transaction in this context is basically composed of the amount to send/receive, plus the sender and receiver public keys.

When a transaction is sent, it is singed with the sender's private key. This means that anybody can confirm who the sender was by checking if the public key matches the signature of the transaction.

This way, we can trace the transactions as far back as we want, without exposing the private keys of the wallets involved.

Operate and maintain your VPS

Mon, 12 Jul 2021 16:15:12 GMT

SSH

To ensure a secure SSH connection, it is best to not rely on password authentication. Instead, we should use a key pair.

The idea is, we generate an ssh key for our machine and make our server trust it.

By doing this, we ensure that only the holder of the key can connect through SSH with the VPS, making access quicker, easily scriptable and brute-force proof.

Generate the pair

ssh-keygen -t ed25519 -a 100 -C "[email protected]"

This command should prompt you for a path in which to store the keys (usually ~/.ssh/) as well as a passphrase.

You may want to leave the passphrase blank if you plan on scripting on top of this connection. Although more convenient, it is less secure. In any case you can later run ssh-keygen -p to change or remove the passphrase.

You will now see a key pair in the path you selected. Let's get one of them on your server.

Get the public key on your server

ssh-copy-id [email protected]

You'll have to enter your VPS's root user's password, after which your server will authorize access to the machine holding the SSH private key.

Log out and back in. If it bypasses the password prompt it worked! If it didn't, check the permissions for both the keys and the .ssh directory.

Disable password logins

Not a necessary step but if you want a reason to do it just check the output of journalctl -xe on your VPS. To avoid brute force attacks, let's make sure logins are only allowed for the private keyholder.

Open /etc/ssh/sshd_config in your VPS and find/set the following lines as shown.

PasswordAuthentication no
PermitEmptyPasswords no
MaxAuthTries 3

This will harden your connection quite a bit, you can go the extra mile by setting up a non-root, sudo user and only connect to the VPS with that user. If you go that route, make sure you set PermitRootLogin no to remove the possibility of a root login completely.

Of course, reload ssh: systemctl reload sshd

What if I lose the key?

Don't. Backups are your friends.

But in case you do, your VPS provider will likely offer some sort of local prompt emulated in a browser window.

Being local to the machine (or at least functioning as such), this will only be accessible to you after you log in to your VPS provider online account and won't prompt you for authentication Then, simply set PasswordAuthentication yes in /etc/ssh/sshd_config.

Rsync

You'll often find yourself needing to transfer files to and from your server. Rsync is probably the easiest way to do it: it's fast, reliable and simple.

Make sure it's installed both on your local machine and on the server, then write (or make an alias for):

rsync -rtvzP /path/to/localFile [email protected]:/path/on/the/server

This command will run recursively (including directories), transfer modification times (skips non modified files), visualize the files being uploaded, compress (z?) files for upload and Pick up where it stopped in case of lost connection.

Of course, to download something from the VPS, just reverse the arguments.

Cronjobs

There are certain routine tasks that are better left to Cron. It will take care of running any command with a given frequency or repetition pattern.

Say, for example, that you want to automate updates for your server. You could run crontab -e and insert something like 30 2 * * 0 apt -y update && apt -y upgrade into the file (:wq to save and quit).

Let's break it down:

 .------------------ minute (0 - 59)
 | .--------------- hour (0 - 23)
 | | .------------ day of month (1 - 31)
 | | | .--------- month (1 - 12)
 | | | | .------ day of week (0 - 6)
 | | | | |
 * * * * *
30 2 * * 0 apt -y update && apt -y upgrade

Basically a machine friendly version of 'please update the system every Sunday at 2:30AM'.

There's plenty you can do with cronjobs, this website is a great tool to set specific patterns.

Finding Things

You might use whereis or which to look for executables, but that can be insufficient depending on the task at hand.

updatedb can quickly index the whole system, which is neat since there's a tool called locate that can easily find files and directories whose paths contain any given string.

So if you run updatedb and then locate cron, you will see a list of files and directories containing cron in their paths. Pretty cool!

Ports

Often you'll want to know which ports are in use. Assuming ss is installed, you could use something like:

sudo ss -tulpn | grep LISTEN

In some cases, you'll want to free a given port no matter who is occupying it. Assuming lsof is installed:

lsof -i tcp:[PORT_TO_FREE] | awk 'NR!=1 {print $2}' | xargs kill

Keep in mind that this is a bit of a nuclear option, you might want to double-check what you are getting rid of!

Troubleshooting

Troubleshooting issues is quite common when working on a VPS, and you'll likely find the root cause in the logs.

System-wide logs can be seen by running journalctl -xe (-xe is to make the results a bit more useful). Of course, you can make your life easier if you know what you are after:

journalctl -xeu brokenApp

This will only show entries relevant to your brokenApp.

Of course, not all apps use the system logs. These will usually have their own under /var/log/. For example, NGINX has its access logs under /var/log/nginx/access.log, and the error logs in /var/log/nginx/error.log. Have a look around, you'll definitely find what you're after.

Another useful troubleshooting utility is systemctl. It won't give you any logs, but you can use it to stop, restart, reload and start services manually and/or check their status:

systemctl status appNotWorking.service

This command will give you plenty of information regarding that specific service.

Set up a firewall with UFW

Mon, 12 Jul 2021 16:14:37 GMT

What it is

The Uncomplicated Firewall is just an easy way to interact with iptables, the default way for Linux based systems to control connections to and from the web.

You'll usually find it in web servers, although it can --and arguably should-- be installed on your main machine.

Basic Setup

Let's take a look at a basic setup for a web server:

Status: active
Default: deny (incoming), allow (outgoing)

To                           Action      From
--                           ------      ----
22                         LIMIT IN    Anywhere                  
80                         ALLOW IN    Anywhere                  
443                        ALLOW IN    Anywhere

The second line tells us the default policies for all non specified ports. In this case it denies all incoming traffic while allowing all outgoing. Specific port policies are listed below (SSH, HTTP, HTTPS, etc.).

Config

If we run ufw status right after installing it, we'll get an underwhelming Status: inactive as a response.

Makes sense, now let's configure a basic server-ready setup like the one above:

ufw default deny incoming # Block everything from the web
ufw limit in 22 # Limit incoming SSH connections
ufw allow in 80 # Allow incoming HTTP connections
ufw allow in 443 # Allow incoming HTTPS connections
ufw enable

Important: Make sure to not block ssh communication! That might lock yourself out of your VPS/Server completely!

Now if you run ufw status verbose you should see pretty much the same information as we saw in the example above.

Deleting rules

For example, if you want to delete the previous HTTPS rule:

ufw delete allow in 443
ufw reload

Fine-Tuning

Of course, you can easily change the default behavior as well as fine tune the policy on a per port basis. You can deny, reject, limit or allow either in or out going traffic for which ever port you might need, as well as use the same parameters to define default behaviors.

Serve your site on Tor

Mon, 12 Jul 2021 16:14:24 GMT

Why?

Isn't Tor used by criminals to do bad stuff? Kinda. It's also used by people that cannot safely browse the clear web and/or express their opinion due to tough restrictions or straight systematic oppression. People that don't want certain queries to be public, journalists, whistleblowers and privacy minded people regularly browse the web through Tor.

It's really easy, so why not make your site accessible for Tor users? Consider that even if you are not using Tor today, you might in the future.

Install & Enable

For any serious use, you should have a look at their install instructions. To keep it simple, you can probably find reasonable up-to-date packages in your preferred distro's repositories.

Once installed, open /etc/tor/torrc in your favorite editor, search for the lines HiddenServiceDir /var/lib/tor/hidden_service/ and HiddenServicePort 80 127.0.0.1:80 and uncomment them.

Start the service with systemctl enable --now tor.

Get your onion address with cat /var/lib/tor/hidden_service/hostname.

Serve

If you know how to set up nginx this won't be anything new to you.

Simply create your nginx config file for the onion site by opening /etc/nginx/sites-available/your-onion-website with your favorite text editor. Then, paste and adjust these lines:

server {
   listen 80 ;
   root /var/www/onion-site ;
   index index.html ;
   server_name onion-address.onion ;
}

Ensure you point to the right path and onion address, and that it!

Make sure to symlink the config file to /etc/nginx/sites-enabled and reload nginx.

You should now be able to open your Tor browser, paste your onion address, and visit your site!

Serve your website with Nginx

Mon, 12 Jul 2021 10:50:30 GMT

You are going to need a server or a VPS, as well as a registered domain name.

Basics

Assuming you already have a properly configured shh connection, connect to your VPS and install NGINX.

Broadly speaking, NGINX will look for instructions on how to serve a given site in the sites-enabled directory. These are usually symlinks to config files located under sites-available.

Let's learn by building something.

Site level config

Say you have your website under /path/to/your/website/, just as an example. We'll begin by creating a config for your website:

nano /etc/nginx/sites-available/yourwebsite

Of course, name yourwebsite whatever you like.

Just copy the config below, swapping yourdomain.org for whatever your domain is.

server {
    listen 80 ;
    listen [::]:80 ;
    server_name yourdomain.org ;
    root /path/to/your/website ;
    index index.html ;
    location / {
        try_files $uri $uri/ =404 ;
    }
}

It's pretty self-explanatory: "Listen on port 80 (HTTP) for requests to yourdomain.org, and serve them whatever is under /path/to/your/website, starting with the index.html file. If nothing is there, respond with code 404."

Enable the site

Use ln -s to symlink the config file as explained above and restart nginx to make it load the new configurations:

ln -s /etc/nginx/sites-available/yourwebsite /etc/nginx/sites-enabled
systemctl restart nginx

We are basically done at this point! This will serve your website under your domain name. It's really that easy.

That said, there are some other things you might want to consider.

NGINX config

We went over how to configure NGINX at the site level. There is also a more general config file: /etc/nginx/nginx.conf Here you can tinker with more generic settings regarding NGINX itself.

For example, if you are hosting a file server you might want to play around with client_max_body_size. This will change the maximum allowed size of the client request bodies.

You can change where the access or error logs are stored with access_log and error_log.

Or you could please the SEO gods with these Gzip settings:

gzip on;
gzip_vary on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml application/json;
gzip_disable "MSIE [1-6]\.";

Security

As a general rule, you want to at least have a properly set up firewall.

Moreover, your browser will try to prevent you from visiting your site. This is due to the lack of certificates, modern browsers really prefer HTTPS (and for good reasons).

Let's see if we can get that lock icon that browsers like so much.

Be the S in HTTPS

There are multiple ways to obtain certificates for your site, but by far the easiest is to use certbot.

apt install python-certbot-nginx
certbot --nginx

Just follow the instructions, it's dead easy. It'll ask for:

Your email (to notify about expiring certs)
Which domain/s to certify
Whether to redirect traffic from HTTP to HTTPS. This is definitely the way to go, so feel free to select option 2 here.

Certificates created with this method need to be renewed every three months. We can automate this using cron.

crontab -e

Paste 0 0 2 * * certbot --nginx renew into the file to create a cronjob that automatically asks certbot to renew all certs, and does so every two months.

Now we have a decent NGINX setup as a starting point for your website or for whatever else you might want to host on your VPS

Map & Reduce in Java

Fri, 28 May 2021 15:01:39 GMT

Streams rundown

Since Java 8, you can use the Stream API to manipulate Collections. They are a fairly semantic way to apply a series of changes to a Collection, producing another different Collection (or object, or variable) as a result. No mutation occurs.

Generally speaking, you turn a Collection into a Stream with .stream() and pipe a series of operators to produce the desired result.

Each intermediate operator (map, filter, sorted, etc.) takes a Stream as input, while outputting another Stream.

You close the Stream by calling one of the terminal operators: collect will return a Collection (no surprises here), forEach will return Void, while reduce and find will be covered below.

Map

Runs a given function on each element of the Stream.

Something like inputStream > Map(myFunction(element)) > outputStream.

For example:

List<String> list = Arrays.asList("this", "is", "a", "test");

List<String> answer = list.stream()
  .map(String::toUpperCase)
  .map(str -> str + ".txt")
  .map(str -> str.length())
  .collect(Collectors.toList());

System.out.println(answer);

Output: [8, 6, 5, 8]

Filter

Similar to map, but in this case, the function it receives needs to return a boolean. This is because filter will only output the elements of the incoming Stream that return true after being evaluated with the function it received.

Something like inputStream > Filter(myFilter()) > outputStream, where myFilter() returns a boolean.

List<String> list = Arrays.asList("this", "is", "another", "test");

List<String> answer = list.stream()
 .filter(str -> str.length() > 3 && str.startsWith("a"))
 .collect(Collectors.toList());

System.out.println(answer);

Output: [another]

Reduce

Produces a single result from a Stream, by applying a given combining operation to the incoming elements.

There are three possible components in this operation:

Identity (optional): initial or default value, if the Stream is empty.
Accumulator: function that takes two parameters:
- Partial result of the operation.
- Next element of the Stream.
Combiner (optional): function used to combine the partial results when under parallel execution or mismatch between the types of the accumulator.

Something like inputStream > Reduce(myIdentity, myAccumulator, myCombiner) > result.

Accumulator

Unless the accumulator has some complexity to it, you'll usually see it as a Lambda:

String[] array = { "Java", "Streams", "Rule" };

Optional<String> combined = Arrays.stream(array).reduce((str1, str2) -> str1 + "-" + str2);

if (combined.isPresent())
  System.out.println(combined.get());

Output: Java-Streams-Rule

By default, reduce will return an Optional of the type it finds in the incoming Stream, hence the if statement at the end.

You can avoid that part by closing the Stream with an orElse().

Identity

Useful for avoiding NullPointerExceptions, especially when reducing complex objects.

int product = IntStream.range(2, 8)
   .reduce(0, (num1, num2) -> num1 * num2);

System.out.println("The product is: " + product);

Output: The product is: 5040

Combiner

Due to some quirks of the JVM when under parallel execution, we'll need a way to combine the results of each sub-stream in one.

A simple example with the three reduce components explicitly set might look something like:

int sumAges = Arrays.asList(25, 30, 45, 28, 32)
   .parallelStream()
   .reduce(0, (a, b) -> (a + b), Integer::sum);

System.out.println(sumAges);

Output: 160

The Combiner will also be necessary if different types are managed in the Accumulator. In the example, the Accumulator has an int as partial result, but a User as next element:

List<User> users = Arrays.asList( new User("Dacil", 30), new User("Gabriel", 35));

int result = users.stream()
  .reduce(0, (partialAge, user) -> (partialAge + user.getAge()), Integer::sum);

Find

There are two variants of the find function in Java:

findFirst: Deterministically find the first element in the Stream.
findAny: Return any single element of the Stream, disregarding order.

One always gets the same element (given the same input Stream), while the other does not guarantee it. Bear in mind, that in simple single-threaded examples like these, both are likely to behave in the same way.

String[] array = { "Stream", "Java", "Rule" };

Optional<String> combined = Arrays.stream(array).sorted().findFirst();

if (combined.isPresent())
 System.out.println(combined.get());

Output: Java

By default, findFirst will return an Optional of the type it finds in the incoming Stream. Just like we did with our reduce example, you can avoid handling the Optional by closing the Stream with an orElse().

Back in time with git

Sat, 22 May 2021 16:46:36 GMT

One useful feature of VCS (git or otherwise) is the ability to restore the state of a project to a previous point in time.

Here are some common mistakes and how to fix them.

Local changes

Changes you might want to undo before being pushed to a remote.

Commit

Commits can be undone using the git reset command. There are multiple ways to undo a commit, depending on what you want to do with the changes in it.

--hard: Removes all changes from the removed commit
--soft: Puts all changes in the staging area
--mixed(default): Puts all changes in the working dir (unstaged)

We also need to tell git which commit we are resetting to. This is done by passing the hash of the commit prior to the one to be undone:

git reset --soft <hash_of_good_commit>

You can get the last 10 hashes with this command:

git log -10 --abbrev-commit --pretty=oneline

If only the last commit needs to be reset, HEAD~1 can be used instead of the hash to tell git to go to the commit before the current one:

git reset HEAD~1

Of course this allows for any number of commits to be undone, not just the last one.

Change

Let's suppose, to keep things simple, that all changes to a file need to be undone. If these changes have not yet been added to the staging area, git restore <file> will remove those changes.

If instead they are already in the staging area, git restore --staged <file> will unstage them, so that they can be either modified and restaged, or removed altogether using the previous command.

Of course if multiple files need to be handled a . can be used instead of a list of file names. Consider that this will apply to all files.

Similarly, if all uncommitted changes need to be fully discarded, running git reset --hard with no commit hash will reset the state of the project to whatever is in the current commit, removing all other changes.

Merge

Undoing an in-progress merge is as simple as running git merge --abort.

If however the merge has already been committed, the previously mentioned git reset --hard HEAD~1 will also work here. Of course, using the hash instead of HEAD~1 would work as expected. Merges are ultimately just fancy commits.

The Nuclear option

Sometimes, the local work tree gets mangled by a combination of odd git abstractions and user error.

It might be easier to fully reset the local env to whatever is currently on the remote repo. To do this, run these commands:

git fetch origin
git reset --hard origin
git clean -xdf

Here, the state of the remote repo is fetched, the state of the local repo is reset to the remote one, and all untracked files are cleaned recursively, leaving the working area with no changes.

Indeed, at this point one might consider rm -rf ./the_whole_project/ && git clone the_thing_again. This works, but also removes all branches and ignored files. Plus, big repos might take a while to fully clone. The commands described here should be more time-efficient.

Pushed changes

If the changes have already been pushed, using reset like before will require a git push --force, which will overwrite the remote repo with your current one (or more specifically, overwrite the conflicting changes).

This might not be an issue in a personal project but when working with other people it's a bit no-no. In fact, force pushes might be disabled altogether.

This makes sense, since changing the state of the remote while another person's work depends on that (now overwritten) state can render their work useless or take a while to merge back together.

Revert

Apart from resetting a branch to a given commit, we can also revert a specific commit (or set of commits).

This way, instead of removing commits, we add new ones with the changes required to reset the state of the project to how it was before the commit to be reverted.

So given git log like this:

621d866 (HEAD -> master, origin/master) oh fuck
07ef6b4 another goot commit
3dbbc2b good commit

Resetting the last commit would require a force push, but git revert HEAD will simply add a new commit that can be safely pushed:

5c07fa2 (HEAD -> master) Revert "oh fuck"
621d866 (origin/master) oh fuck
07ef6b4 another goot commit
3dbbc2b good commit

Beware however, that if a revert is done on a commit previous to the last one, and the reverted changes are needed for the changes in newer commits to work, those commits might break (as in, the build might break, or the tests might fail). In those cases you might need to revert multiple commits or introduce further ones to fix the issues.

This has no good solution, so consider reverting a commit as soon as possible and committing small changes at a time. A commit that changes 200 files is bound to cause issues when reverted, while one that only modifies a function likely will not.

Merges

Perhaps surprisingly, using the previous revert command on a merge commit will fail:

error: commit <HASH> is a merge but no -m option was given.
fatal: revert failed

That -m flag takes a number that corresponds to the Main parent.

This makes sense, a merge by definition has two parent commits: the one you are on when running git merge (1) and the one you are merging into it (2).

So if merging feature-branch into master, the former would be 2 and the latter would be 1.

The command:

git revert -m 1 <merge-commit-hash>

Would create a revert commit restoring the state of master.

Since more than two commits can be merged, the -m flag takes an indefinite number. In most cases, the expected behavior will be achieved passing 1 to it.

Task scheduling with Spring

Thu, 29 Apr 2021 16:19:52 GMT

Enable the resource

There are many ways to manage repeating tasks in Spring, but by far the easiest one is using the built-in Scheduler.

To enable it you just have to annotate the main class with @EnableScheduling.

It's worth pointing out that the default behavior doesn't allow for parallel execution of tasks. To do this you'll also use @EnableAsync on the main class and @Async on the desired function.

Types of Scheduling

Spring offers three ways of managing recurrent jobs:

Fixed Rate

Runs the method every 'X' milliseconds. Enable it with @Scheduled(fixedRate = timeInMilliseconds).

@Scheduled(fixedRate = 2000)
public void repeatEveryTwoSeconds() {
 System.out.println("I run every two seconds, no matter the previous run!");
}

Fixed Delay

Runs the method 'X' milliseconds after the previous execution is done. Enable it with @Scheduled(fixedDelay = timeInMilliseconds).

@Scheduled(fixedRate = 2000)
public void repeatAfterTwoSeconds() {
 System.out.println("I run two seconds after the previous run is over!");
}

You can also adjust the initial execution delay adding initialDelay, as such: @Scheduled(fixedDelay = 2000, initialDelay = 3000).

Cron

For greater flexibility, Spring allows us to adjust the repetition pattern with Cron. Enable it with @Scheduled(cron = "* * * * * *").

@Scheduled(cron = "0 0 0 * * *")
public void repeatEveryMidnight() {
 System.out.println("I run every day at midnight");
}

Unix cron vs Spring cron

There are some subtle differences between the cron schedules you'll set up in Spring applications and the ones you'll find in your typical Linux machine.

Unix Cron

┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of month(1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
│ │ │ │ │
* * * * *

Spring Cron

 ┌───────────── second (0-59)
 │ ┌───────────── minute (0 - 59)
 │ │ ┌───────────── hour (0 - 23)
 │ │ │ ┌───────────── day of month (1 - 31)
 │ │ │ │ ┌───────────── month (1 - 12)
 │ │ │ │ │ ┌───────────── day of week (0 - 7) (Saturday to Saturday)
 │ │ │ │ │ │
 │ │ │ │ │ │
 * * * * * *

As you can see, where Unix-like Cron has only 5 fields (some systems have 6, but that's used for user permissions), Spring-like Cron has 6; adding the ability do manage tasks my the second.

Moreover, while traditional Cron only supports macros in some systems, Springs version does so by default:

Macro	Description	Cron
`@yearly`	Once a year	`0 0 0 1 1 *`
`@monthly`	Once a month	`0 0 0 1 * *`
`@weekly`	Once a week	`0 0 0 * * 0`
`@daily`	Once a day	`0 0 0 * * *`
`@hourly`	Once evey hour	`0 0 * * * *`

Life beyond Google Search

Sun, 25 Apr 2021 10:42:43 GMT

Even setting aside all the privacy concerns that come with using any Google product, some find the almighty search engine to be pretty lack luster.

Its results are filled with ads, spam and irrelevant or even auto-generated results. Plus, it's nearly impossible to sift through the noise when investigating any sort of vaguely controversial topic.

To be fair, there is one thing that it does pretty well: help you out when you don't quite know what you are looking for. I wouldn't say it does so to your best interest but hey, it's something.

Search Engines

There are quite a few search engines available to you. Some of them (Bing) are widely considered meme-engines.

I would disagree: each of them has its own use case. We are just used to the (supposed) omnipotence of Google. Here is a quick overview:

Duckduckgo

Probably the most popular of the bunch. Privacy minded, kind of bare bones. No tracking, no profiling and much fewer ads than Google. Pretty good general purpose alternative, except maybe for image search.

Startpage

A different front-end to Google's back end. The idea is that you still want Google's results (for some reason) but would rather not have the NSA over for dinner. Most of the targeted advertisement should not spam your results. They are based in Europe which might give you some peace of mind.

Swisscows

Super family friendly SE. Built-in blockage of porn, violence and the likes. It mixes its own indexing with Bing's.

Bing

Indeed, trusting Microsoft instead of Google is hardly any better. However, it's preferable to have 5 different companies partially tracking you than to have one knowing you better than you know yourself. They do their own indexing, so results should be more or less independent of Google. Plus, it's actually pretty good for image searches.

Yahoo

The same reasoning as above more or less applies here as well. Still not a great service privacy wise, but useful in its own right. If you are into crypto or finance in general, it has some pretty useful tools and is well respected in that regard.

Qwant

Based in France, it recently started doing their own indexing. Easy to use, simple UI, user-friendly design.

Wolframalpha

Mainly used in academia. Rather different from what we usually understand by 'Search Engine', but rather useful with technical searches.

Yandex

For those who hate the NSA but would love to meet the ~~KGB~~ SVR. Jokes aside, it's very widely used along Russia's sphere of influence. As such, it's very useful for learning different perspectives on sensitive issues or topics.

If you want to 'legally' download content, look no further.

You.com

Pretty UI, basically no ads, and the ability to customize sources and searches to your heart's content. What's not to like?

Brave search

Another one of the few that do their own indexing. Pretty fast and reliable. Their whole marketing revolves around privacy on the web, so you can expect a decent level of privacy. It even offers to add results from other SE in case you aren't satisfied.

It also works on Tor!!

Search with Searx

So what now? Am I supposed to use twelve search engines instead of one?

You can just use a meta SE! Simply put, it queries a bunch of different SE for you and presents all the results in a single page.

The main one that comes to mind is Searx (or more accurately, it's fork SearxNG).

It doesn't offer personalized results, because it doesn't generate a profile about you.
It doesn't save or share what you search for.
It's fully open source (code here) so you can actually host your own instance (I actually do so and use it daily) or just chose one you trust from this list and use it.

If you want you can set exactly how and which SE it queries. If you don't, you can just go ahead and use a public instance as is. It won't work with every SE available, and it might be a bit fiddly on occasion, but it probably offers you more than you need, and it's just so convenient there's no getting around it.

Please use a password manager

Wed, 31 Mar 2021 18:49:26 GMT

Why you need it

Repeated passwords

Even if you think its new and clever, you might very well have used the password you just 'invented' for a long forgotten account (which may or may not have leaked). Repeating passwords is nearly as bad as setting them to 'admin' or 'password1'.

Plain text

If you don't use a password manager and don't repeat passwords, chances are you are storing them in an unencrypted, plain text file.

We all manage a huge amount of accounts, no way you can remember all those passwords.

Anybody with (even remote) access to your machine can read an unencrypted file. Plus you need to be in that specific machine to access your passwords or copy that file around.

TOTP

Nowadays, it's often required to have some sort of MFA set up. One Time Passwords are by far the mos convenient and secure way to achieve this.

Plain and simple, this is not possible without a password manager.

Work passwords

You might not care about your personal stuff, but please do care about your work related accounts and credentials.

You put your whole company, coworkers and clients/users at risk when you neglect your online security at work.

One of the main ways attackers get access to user's sensitive information is by taking advantage of bad practices used by the people who are supposed to be trusted with that information.

Why you want it

It's more comfy than your solution

I can bet that the way you currently manage your passwords is either uncomfortable or insecure. You either have them written in plain text in a file you have to fetch every time you log in (or even worse, written in a physical paper like a caveman), or you let your browser manage them for you (good luck using a different browser or needing any kind of advanced management).

Good password managers, especially if they have a companion browser extension, are literally a one click solution to both creating good passwords and filling them into the login forms.

Good passwords are hard

Just look at the requirements for any account password and be honest: Can you really come up with a good one without using personal information like name or DOB? Yeah, me neither.

Typing huge passwords sucks

And you always get something wrong.

Lost the file? Lost all passwords

If store them in a file, your passwords are gone forever as soon as that file gets deleted.

That's just a bummer.

What to use

Well... a password manager 😀. Here is what to avoid and a personal suggestion.

Avoid non FOS software. Here is why

Nobody knows what the code actually does or how secure it is. You are 100% just trusting the company offering the service.
FOSS is always more secure. It can be publicly audited and people will pick it apart and patch it.
If the company decides to make you pay for features that where once free you might have no choice, except maybe to export a JSON or CSV file and move away.
If the company goes six feet under, you're on a ticking time bomb to find an alternative.
You are in charge. You don't have to, but often can go and host the service yourself.

The dynamic duo

BitWarden

The more user friendly alternative. They are widely used and known, are repeatedly audited by third parties, have a free and a paid business service, and have pretty much anything you might need:

Desktop GUI
Desktop CLI
Mobile GUI
Browser Plugin
Web Vault

You have the option to make an account with them and host your passwords in their servers (just like with any other password manager) or you can host your own instance on your own server.

If you plan to go that route, check out VaultWarden for a super lightweight alternative!

KeePassXC

Minimal solution (although not as minimalist as just using pass). It's a cross-platform implementation of the KeePass standard with added plugin support.

You have a local encrypted vault which you connect to the plugin and that's it. You are in charge of backups and security and can access the vault only locally, but there is literally no one else involved. Not even a connection the web.

Conclusion

Personally I have used a lot of different password managers. Nowadays, I run a VaultWarden instance on a VPS but still have a local copy available from KeePassXC, just in case.

No solution is perfect for everyone and each have valid use cases.

Except for not using one. That's just silly 🙃.

Clean Your Code

Wed, 24 Mar 2021 14:52:21 GMT

Got the idea from here, but concepts below come mostly from horizontal reading and Uncle Bob's speeches.

Overview

As a general rule of thumb, try to make your code 'pretty'.

Look at the indentations, do they make sense? Are you like five indentation levels deep? Is the naming descriptive? Is there a more efficient way to achieve the same thing? Is your code elegant? Do you actually feel proud about it and have an urge to show it to your peers?

Made to be read

You got your code to work? Great! That means it's machine friendly. Now you need to go back and make it human friendly.

When coming up with new ideas or solutions to existing problems, our minds tend to become a bit messy. We forget stuff, rush through ideas and generally don't care much for the 'proper solution' but focus on 'a solution'.

This is good, we should focus on getting the job done first. But we also need to take some time after the fact to clean after ourselves.

You are most likely not the only one working on that peace of code, treat it like a common space. First make it work, then clean it up.

You're not done when it works, You're done when it's right. — <cite>Robert C. Martin<cite>

Code is clean if it can be understood easily by a competent reader. Another developer should be able to enhance it (or fix it for that matter).

Surprises might be nice irl but I'm not looking forward to going "WTF?" when reading your code. The more predictable the better.

Clean code does one thing well. — <cite>Bjarne Stroustrup <cite>

Some Rules

General

Conventions are there for a reason. It's easier for me to understand you if we both follow a set of common rules.
KISS (Keep it simple stupid). Reduce complexity as much as possible, keep it minimalistic.
Always find the root cause. Don't just fix the problem, actually fully resolve the issue.
Boy Scout Rule. Leave the code better than you found it.
Unix philosophy. Do one thing and do it well, don't write code with excessive/diverse responsibilities.

Understandability

Be consistent. If class A has a var name and a var surname, class B shouldn't have a var firsName and a var lastName in their place.
Use explanatory variables names. They help with semantics.
Prefer value objects to primitive type. They also help with semantics.
Avoid negative conditionals and be generally careful with conditionals, they get out of hand fast.

Naming

They should be descriptive, unambiguous, meaningful, pronounceable and searchable.
Don't use data or object in the name. We already know.
No Magic Numbers.

Functions

They should be small and single scoped.
Their names should describe their intent. Default to using verbs if possible.
No more than 3 arguments, the fewer, the better.
Side effects are usually bad, hard to track and easy to avoid.
Don't use flag arguments. If a function does one thing or another based on the input, you should have two functions, not one.

Comments

A necessary evil. To be avoided if possible.
Trust your VCS, remove the code.
Use them to: explain the intent, clarify the code or warn of consequences.

Structure

Prefer vertical coupling/cohesion rather than horizontal.
Group variables, objects and functions if they relate in usage.
Respect indentation.
Keep lines short.
Blank lines can be used to separate weakly related elements. Consider separating them further.

Objects

Expose an interface (API), hide internal structure.
Keep them small and single scoped.
Few imports and few instance variables.

Avoid

Rigid software is hard to change, and there are usually cascading consequences for each change.
Fragile code breaks in many places due to a single change.
Complexity is sometimes necessary, but often accidental. Avoid the latter.
Code Repetition is a pain to work with.
Opacity is not a sign of a clever mind, but of an uncaring personality.