alecgerona.io RSS Feed

PyConPH 2024 Speaker Slides

Sun, 25 Feb 2024 00:00:00 GMT

I’m proud to announce that I delivered a talk for PyConPH 2024! I talked about Developer Experience and how we approach it in my company, BillEase.

If you want to see the slides I used, please click here.

Setting up media automation on my NAS

Sat, 12 Mar 2022 00:00:00 GMT

So what exactly do I mean by media automation?

So the plan is simple: have my NAS serve all my media. But where will the media come from? How will they be managed internally in its folders? Drives? What can I use to manage this and most importantly automate it?

What to do

So if you haven’t seen my previous post, I’m using Unraid as my NAS OS. Just a way to keep management overhead to a minimum. This also means that it’ll also manage my Docker installation for me which I intend to use to run all the parts that constitutes my media automation strategy.

So without further ado, let’s go step by step.

0. Prerequisites

Before proceeding with this guide be aware that you should have a working knowledge of Linux filesystems, Docker, media servers, torrents, and usenet to maximize the features. If you’re at all confused by any of this (as was I when I started) feel free to leave a comment and I’ll help out when I can.

1. qBittorrent

Of course the start is qBittorrent ( ͡° ͜ʖ ͡°). I chose to go with hotio’s qBittorrent image since it has more features (such as having Vuetorrent pre-installed) than say, Linuxserver.io’s which I have used in the past. Unraid allows this via a simple install command in the apps tab assuming you’ve installed Unraid’s Community Applications OS plugin.

Remember to toggle autostart to start the containers on reboot.

And here are the settings that I used. The /data/torrents in the container path is where we will have qBittorrent direct its downloads.

Also, very important, make sure to use a custom docker network for this one (and all the succeeding containers following except otherwise noted). This serves a lot of purposes but the most important part is that our containers will be limited to their own network and that they can connect to each other using their own names instead of IP. This is critical for when you’re moving servers, restoring a backup, or changing your server’s static IP.

In case you forgot how to create a custom docker network it’s as simple as going into the terminal of your Unraid server and:

docker network create <your-network-name>

Then once in the web UI, we have qBittorrent use the path we’ve specified in the step above.

Then we change our default admin password and enable VueTorrent web UI.

Now before we hit save, restart the UI, and see that sexy VueTorrent skin, a quick reminder to disable these two settings right here. This will be a needed way down the line. Of course this is optional as is the later step but if you’re not planning to open up your NAS to the internet then this really isn’t a security issue.

That should be it for our downloader.

2. Sonarr

Next up is Sonarr or the TV show downloader. I’m also using Hotio’s just by virtue of already using the previous one with it.

We pretty much set up the same settings on its mapping. Except that instead of mapping the container path to a specific host directory or share, we’re giving it access to the entire Unraid /data share. This is so that Sonarr has access to all the subdirectories within that share and can move files between torrents/usenets to media and other subdirectories we may have via hardlinking or instant-move.

Now in Sonarr’s web UI, go to Settings > Media Management and map our /data/media/tv as a root folder.

Some things to note here is that the disk space that Sonarr shows might be incorrect since it may be showing the container’s disk space instead. So don’t worry about it. Also, you might notice that there are a few issues listed in the Systems tab. This is normal. This is just telling us that we need to do to make Sonarr fully operational.

3. Jackett

Which brings us to our next requirement, Jackett. Now, Jackett is what is called an indexer or basically the tool that searches in behalf Sonarr and the other Arrs on where they might find their files to download. No need to change anything in the mapping so this is a straight install.

Now onto the web UI, click Add indexers and add the indexers that you might want to use. For security’s sake, I won’t put up a screenshot here. You should know it when you see it.

Back to Sonarr, go to Settings > Indexers and add a Torznab feed. Here you just add the name of the indexer itself, the Torznab feed URL and the API key from Jackett.

I also added a few more settings to my preferences in the categories.

You need to repeat this for each indexer you’ve added in Jackett.

Next is to add qBittorrent to Sonarr. Go to Settings > Download Clients. There should be a qBittorrent preset. Note here that the host is now the qBittorrent container’s name instead of its ip/port! This is because they’re all in the same Docker network.

Then make sure the prefilled category, tv-sonarr, is in qBittorrent itself.

Progress Check

Okay, okay. I know that was long. But those were mostly images. So not really. Just a quick recap, but at this point we’ve actually automated everything from looking up a series to having it downloaded to our array. Really. In Sonarr, try going to Series > Add New, look up any series you want, and hit Add. If you tick the Start search for missing episodes box, it’ll actually search and then download the episodes for you. Yes. Every episode of every season.

And we’re just getting started.

4. TV Show Download Preferences

Now this part is mostly for my benefit (as this guide really is), but feel free to copy them.

4.1. Set seeding limit time

First, head to qBittorrent to set the seeding limit time to 120 minutes. This will remove the torrent from qBittorrent after the specified time.

4.2. Create a Sonarr HD+ Quality Profile

Sonarr by default comes with a lot of Quality profiles. These work well the default qualities you would find out there such as 720p, 1080p, 4k, and so on. This profile is just a quick QoL way to tell Sonarr to prefer the latest and greatest quality (4k with niceties such as Dolby or Remux) all the way down to 1080p. Sonarr will then flag an error if it can’t find beyond that.

You should then select a profile per series accordingly.

5. Radarr

The next Arr for movies.

We know the drill by now, so I’ll just post screenshots for posterity.

Set the root folder.

Set the indexers.

Add your qBittorrent download client.

Make sure qBittorrent has the required category.

5.1. (Optional) Radarr Profiles and Custom Formats

Now we’re gonna pull off a pro-gamer move and make Radarr picky. Now as you may know, there’s a lot possible group/encoding/quality for releases. Some are better than others. Some should never be downloaded at all. Now here’s my preferences. I mostly prefer 4k releases packaged by Tigole/QxR with x265 or HEVC encoding and AAC audio. This ensures a very reasonable file size with great picture and audio quality. Perfect type to hoard.

Of course not every release will be like this, so we need to tell Radarr to “score” releases according to our specifications and just grab whichever scores the highest (the one that fits our criteria the most).

First, go to Profiles and again create an HD+ profile.

As you can see, there’s a section below called Custom Formats along with the criteria I listed above. Next to the labels are the points assigned to them. Let’s create them now.

Go to Settings > Custom Formats > Add Custom Format and hit Import.

Paste the following.

{
  "name": "x265",
  "includeCustomFormatWhenRenaming": false,
  "specifications": [
    {
      "name": "x265",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": true,
      "fields": {
        "value": "[xh][ .]?265|\\bHEVC(\\b|\\d)"
      }
    },
    {
      "name": "Remux",
      "implementation": "QualityModifierSpecification",
      "negate": true,
      "required": true,
      "fields": {
        "value": 5
      }
    }
  ]
}

Hit save and add one more.

{
  "name": "Preferred Groups",
  "includeCustomFormatWhenRenaming": false,
  "specifications": [
    {
      "name": "Preferred Groups",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": true,
      "fields": {
        "value": "\\b(Tigole|qxr|QxR|tigole)\\b"
      }
    }
  ]
}

Now these may look complicated but they’re simply just JSON text that has a regex as their value to provide the rules to Radarr.

That’s it, actually. Now when we try to add a movie and hit search files manually, we can see the scores that Radarr has given to each release and prioritized them accordingly.

Now just as a footnote if you’re gonna play with this around some more: Radarr will always prioritize quality over whatever score a release may have been given. Meaning, if you have a release that has a higher quality than what matched your criteria, Radarr will choose that. Check out Radarr’s complete priority list.

6. (Optional) Bazarr

Note: This is only required if you’re going to use a media server that does not already fetch subtitles. Since I’ll be using Plex, I can skip this.

Since we’re done with our optical media, it’s time to subtitle them.

Now go open Bazarr’s web ui, go to Settings > Sonarr and tick Enabled. Input these settings. You can get Sonarr’s API key in Sonarr’s Settings > General > Security. Hit Test to validate.

Now we add a Language Profile. We will set this profile to be applied to all incoming TV Shows and Movies.

Next we add our subtitle provider. Go to Settings > Providers. Do create an account first.

Now Bazarr will automatically download English subtitles to our media and place it alongside the files.

7. Lidarr

Our final Arr, for our aural media.

Much like Sonarr and Radarr, let’s use the same settings.

Set the root folder.

Set the indexers.

Add your qBittorrent download client.

Make sure qBittorrent has the required category.

Progress Check

Time for another one. Now I know we’ve really done a lot now. We’re now able to automatically procure media of all types. What’s left is a way to consume them.

8. Plex

Arguably the reason for most NASs, Plex turns your collection into Netflix.

Now I know we’ve been using Hotio’s images for a while but for this one, since I’m planning to buy Plex Pass, which is the pro version of this, and I was an idiot that didn’t use the correct network type and incorrectly blamed Hotio’s image for Plex not working, I went with the official Plex Media Server image.

The config now has more fields. Primarily because of more volumes to map. Nothing complicated, however. One thing we need to do though is to generate our claim key. This assigns our Plex account to our NAS. Go to https://plex.tv/claim to log in and generate one.

Note: Set the Network Type to Host. This container is the only exception.

Now let’s go to Plex web UI and configure our server. Add a name and all our media libraries.

Et voila. We now have a fully functioning media aggregator in our server. This is accessible by any native Plex apps your devices can install (Android, iOS, Smart TV, etc.).

Now let’s set a few more details for Plex. Starting with subtitles. Go to Settings > Your Server > Agents. Create an account in opensubtitles.org and use your credentials in this field. Rearrange the Local Media Assets tab to be the highest. This is so that if you ever use Bazarr or would like to use your own metadata for a specific entry then Plex will use that over whatever it has downloaded on its own.

Do the same for TV shows.

Next, go to Settings > Your Server > Library. Check the following boxes pertaining to Library sync. This makes sure that Plex’s libraries will reflect whatever you’ve downloaded or moved to your media folders.

9. Organizr

At this point you may be tired of either memorizing ports, bookmarking the different web UIs or just plain typing them out all the time in your browser to visit each. I know I was. So there’s a better way. There always is.

After installation, just add tabs like below:

This is where the settings for disabling those things such as clickjacking come to play. This enables us to view our web UIs in iframes, so we never have to open another tab to view another web UI. Well, except Jackett. I haven’t been able to view that as an iframe yet. I didn’t bother because you never have to view it anyway.

10. Spring Cleaning

Now all that’s left is to make sure that our server is all tidy and properly backed up. Luckily, Unraid already has plugins for them.

10.1. Appdata Backup

First up is backing up our containers’ data. A lot of people gloss over this but with all the effort we spent painstakingly setting up our containers we should get this backed up properly, so we won’t have to do it again should our cache drive (where our appdata lives) fail.

Let’s start configuring by creating the share we want our backups to go.

Then schedule the backup. Since I won’t be adding/configuring docker containers all that much, weekly is fine. Remember that this backup will turn off all your containers when it’s running. This may or may not affect the daily mover task. I prefer to keep just one daily scheduled task in my headspace.

10.2 Appdata Auto Update

Next is to schedule the updates of our plugins and Docker containers. Luckily, this works hand in hand with Appdata Backup.

Simply install:

Configure (do the same for Docker):

And toggle the Update Applications On Restart? checkbox in Appdata Backup.

11. Trakt

Last but not least, we need a way to discover movies and shows to add to our collection. Plex does not offer this and Radarr and Sonarr are quite primitive at this, too. This is where Trakt shines.

Trakt is a service to track your media consumption not just in Plex, but in every streaming service out there such as Netflix and Disney+.

Note: The following instructions are just a unidirectional sync of Trakt and Plex as you need a Plex Pass for bidirectional syncing. The sync is not even for Plex per se but more for Radarr and Sonarr to tell it what to add and download.

Begin by creating an account in https://trakt.tv

Then head over to Radarr > Settings > Lists > Add List > Trakt List > Trakt User. In here I’ll be tracking my Watchlist public list. This is a list that every Trakt user gets by default. You can copy this or create your own list.

Same with Sonarr. Settings > Import Lists > Trakt User.

And that’s it. Soon as you add media into your Trakt watchlist be it using the web app or the iOS companion app, Sonarr and Radarr will add it.

Conclusion

And we’re finally done. I’m planning to add more posts regarding my Unraid journey and possibly trying my hand in making automation tools of my own. Again, if you have any questions don’t hesitate to drop me a line or a comment down below, and I’ll do my best to respond and help.

Here’s to hoping future me can read this and find this useful.

Building my own NAS

Sun, 06 Mar 2022 00:00:00 GMT

What for?

You know what. I don’t know. I just want to hoard data and have my own media server.

Okay, what do?

Well, this is actually not simple. I’ve decided right out of the gate that I wanted to build it myself. I’m not a stranger to a media server or a NAS before. If you could call a Raspberry Pi with a 16gB SD card a NAS. I also want something that for the most part I can plug and play, have high availability, flexible for updates, and can generally take care of itself.

In the NAS world these two things are generally antithetical of each other. Good rules of thumb here are if you want to build it yourself, you can save money but you’d have to do a lot of legwork. Likewise, if you buy pre-built, you get access to their ecosystem (OS, parts) and is generally plug and play. Furthermore, dollar for dollar you get more with building it yourself (as it is with PC building in general).

So, since I decided that I will be building it myself, Synology and Qnap (two of the most popular pre-built NAS vendors) are out.

Which means I need to decide on two things: the OS and parts.

Unraid OS

When I first heard of Unraid I was immediately dismayed. It’s not open source and it’s not cheap. Given that the servers I manage professionally at work are all open source, this is a pill that’s very hard to swallow.

However, Unraid offers a lot for what it is that is simply a dealbreaker in other OSs for me.

It’s on Debian

This is a no-brainer. Debian-based OS are some of the easiest servers to use and maintain. This is also what I’m most familiar with.

Docker as a first-class citizen

At this point I’m nearly sold to be honest. When I first experimented with the Raspberry Pi NAS, I’ve been using Docker containers to manage various services. The ease-of-use just trumps every other possible application hosting there is. I didn’t want to deal with having to set up Docker via a hack in other OSs. Especially if it’s FreeBSD like TrueNAS.

You can use mismatched drives!

And here it is. The most important part. Unraid uses a parity system to protect our drives. This is different from using traditional RAID systems that require you to have all your drives at once or all in the same size upfront. For someone just starting out using a proper NAS, the cost-entry to get say, six 20 tB drives at once is just not feasible.

And so, Unraid it is.

Parts

Now on to parts. For a TL;DR here’s my pcpartpicker link.

Due to availability issues, I had to make tradeoffs for this build. Let’s go one by one.

CPU - Intel Core i3 10100

The only reason I went with Intel is that the 10100 is known as the king of energy efficient CPUs for NAS (as far as I read. YMMV.) and that it supports Quick Sync aka Intel’s tech for CPU transcoding that supposedly makes it better. I wouldn’t know since I don’t plan to transcode, but it’s nice to have.

Motherboard - Asus ROG Strix B60-I Gaming WIFI

Now this is the tradeoff. Unfortunately ITX motherboards are unicorns in this day and age of gamers’ market that craves the biggest and most obnoxious cases out there. I had no choice but to get this expensive (and gaming) motherboard that is compatible with my CPU. To be fair, I could’ve gotten an mATX one, but I actually bought the CPU case first, which is amazing by the way, so there you go.

CPU Cooler - Noctua NH-L12S

One more goal for this build is to have an ultra-quiet NAS. We’re talking drives spinning and files copying quiet. What better way to achieve that than with a Noctua cooler? I actually tried putting my ear next to it while it’s running, and I swear I couldn’t tell it was on. Don’t do this at home though. My finger got caught in the spinning blade.

RAM - G.Skill Ripjaws V Series 16 GB

Unraid actually boots off of a flash drive. Yes, you read that right. It doesn’t boot off of an SSD. After it boots, all data needed to maintain the OS’s running state is transferred in RAM. Even with that though, Unraid isn’t as memory hungry as TrueNAS. My utilization sits at 20%.

Power Supply - SeaSonic FOCUS Plus Gold 550 W

Apparently this line of power supplies are popular among NAS users. But go ahead and get what you need.

Storage. Ah, now this is where it starts to get interesting.

This is the highlight of NASs. As such we need to be very careful in choosing. To start with, I went with a Samsung 970 Evo Plus 500 GB for my SSD cache. As a quick refresher, Unraid features the use of cache pools. Basically storage that’s supposed to be fast (faster than a hard drive at least) to store all your services (read: Docker containers) and handle intensive I/O operations. By default, Unraid then runs a “mover” daily that moves extra data present in the cache drive to the array of hard drives that are protected by parity.

To that purpose, I’ve chosen two Western Digital WD Red Plus 6 TBs for my hard drives. One to serve as usable space and the other to serve as parity. Choosing this wasn’t easy as Seagate does offer a very compelling product as well plus WD didn’t do itself a lot of favors when SMR gate happened. But what finally led me to WD was that I made sure to choose the Plus version to ensure that it’s CMR and that it has a quieter operating noise than Ironwolves. To be sure however, I will be using Ironwolves for when the time comes to expand my array.

Flash Drive. Samsung 32 GB BAR Plus (Metal) USB 3.1 Flash Drive

Like I said, Unraid boots off of a flash drive and this just happens to be, allegedly, the best flash drive for Unraid. So there.

Case. Fractal Design Node 304

Finally, the thing that ties these all off. A small form factor, ITX case that can handle SIX 3.5” hard drives! In all my research I haven’t found another ITX case that I can buy that has this capacity. Plus it looks all nice and sleek.

Build

Now that we got the parts together. Time to build. Here’s a montage of my NAS build.

Do note that this case does not support both a power and reset switch. It only has a power switch. A two-lead power switch to be exact. You insert this in the middle row of the block where either is supposed to go. There I saved you a few rebuilds and hours. I wish I had taken a photo of this, but I forgot and you’re a big boy now so you can figure it out.

Always, always test everything using the BIOS with just the essential parts ready, first. This has always plagued. Just plug in the, RAM, CPU, Cooler, Power Supply, and a display.

This particular motherboard supports 11th and 10th Gen Intel CPUs and features two M.2 slots for NVMe SSDs. However, you must be using an 11th Gen Intel CPU to use the front slot (the one with the built-in shield/cover). If you use a 10th Gen like I do, that slot will be disabled, and you won’t even see it pop up in the BIOS. You’d need to use the slot at the back of the motherboard. The box comes with an extra thermal pad. Use this between the case and the SSD underneath the board. This is very important.

Since the Ripjaws are a bit too tall, I needed to use the high-profile configuration of the Noctua cooler. This just means that I had to move the fan from underneath to up top. The front of the fan should face the heatsink. It will be quite obvious what the front and back of the fan are.

Also, don’t be alarmed if there’s a persisting LED light in the motherboard. As long as it’s only one, and you’re not using a graphics card like I am, it should be fine. The LED light just means that the motherboard cannot detect a graphics card.

The SATA power cable are usually chained together. It's absolutely safe to use one running cable to power multiple drives.

When flashing the boot drive with Unraid, make sure to check the “Allow boot from UEFI” checkbox. There is currently no way to allow legacy mode in this motherboard.

And that’s it for the build. I hope this has been helpful to you all. My next post will be about media automation on this very NAS. If you have any questions, just drop a comment, and I’ll do my best to respond.

Love it or hate it: here's how I setup my Mac

Thu, 07 Oct 2021 00:00:00 GMT

Updates

17 Apr 2023: Use starship prompt

15 Aug 2022: Update the fzf setup command

Rationale

Gotta be honest. For the longest time, I’ve avoided the Apple ecosystem. I’m still not that okay with it, but we have to try ‘em all, right? What with overheating MacBook Pros, to class action lawsuits over cracked M1 screens, there remains a lot to be desired in the so-called walled garden.

Setup

With that out of the way, this guide is how I set up my development environment on a MacBook Pro 2019 16in the way I want it, so I’d have something to reference should I need to do it all over again. Hope someone else finds it useful as well.

1. Xcode

Let’s start simple by installing Apple’s command line tools. Basically, things you need to get any sort of development work on your machine.

xcode-select --install

2. iTerm2

I believe it’s already tradition by now (for developers at least) to immediately replace macOS’s default terminal with iTerm2. Some would say it’s a little bloated for what you use it for, but you’re using a Mac already, so you shouldn’t complain.

Install

Installation is as simple as downloading the .dmg and installing through that.

Key repeat rate

A severely underrated QoL change, this set up requires a restart after execution, but you’d be able to glide on your keyboard once your key repeat rate is up. A boon to Vim users, terminal junkies, and fast typists alike.

defaults write NSGlobalDomain KeyRepeat -int 1
defaults write NSGlobalDomain InitialKeyRepeat -int 10
defaults write NSGlobalDomain ApplePressAndHoldEnabled -bool false

Skin

For my terminal I’m using iterm2-snazzy. Just download the page and import in iTerm2 (Preferences > Profiles > Colors > Color Presets > Import).

Hotkey summon

Coming from yakuake and guake, the ability to summon the terminal from any desktop with just a hotkey is a godsend. Here’s how to do it with iTerm2.

Go to Preferences > Keys > Hotkey > Create dedicated Hotkey window. This will create a dedicated profile that iTerm2 will use when you summon it this way. You can then edit this profile as you wish.

My hotkey is ⌘+`

I set my transparency to 10% and window size to maximized.

Shell Integration

This is a utility that “keep track of your command history, current working directory, host name, and more—even over ssh.”

This will be useful for the following feature of iTerm2.

SSH workflow

Traditionally, we SSH to other machines by writing the full command such as ssh user@host. We then learn to alias this by using the .ssh/config file and writing ssh alias instead. We can go much further in iTerm2 by not even needing to type anything. Simply make a profile and make it run the ssh command you need.

For my setup I still prefer working mostly in my keyboard and having to open the profile from the tab is just extra overhead that I want to avoid. That said, there is still an advantage to using profiles and that is to visibly notify you that you are working on a different terminal by either having a badge or a different theme or both.

For production terminals I use the red alert theme. I also use the instance name as its badge.

Finally, since I prefer having to type the ssh command myself, I have to set the profile up so it would switch profiles once it detects that I’ve changed hosts. To do that, we need to implement Automatic Profile Switching.

This iTerm2 feature requires Shell Integration which we did above. We then need to add the expected user/host entry iTerm2 needs to switch profiles.

It’ll be something like this. It should end in a glob (*) or it won’t work.

user@host*

3. Homebrew

Now that we have our initial cli tools and a spiffy terminal emulator to work in, we need one more tool to be fully integrated in the macOS (sorta open source) ecosystem—homebrew, the package manager that, like most things in a Pro Apple product, never had built in.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Afterwards you get access to package manager staples such as brew install <package-name> among others.

4. ZSH

zsh is the default (and better) shell in macOS. Aside from its benefits over bash, we’d be making it even better with some choice improvements.

starship

Before anything else, we need to de-clutter our terminal prompt. What better way to do so than by installing starship?

brew install starship

This provides a minimalist yet information-filled prompt that changes with your context.

Make sure that your iTerm2’s font has an alternative option for non-Ascii characters since Monaco (Apple’s default) does not support powerline characters that spaceship needs. I chose to go with Fira code. Just download and install.

zsh-autosuggest

Wouldn’t it be great if we had autosuggest? This may be the single most useful plugin for zsh. If you’d only install one plugin for zsh, this should be it. This allows zsh to suggest commands as you type them based on your history.

Create a ~/.zshrc.local if you haven’t already. Since by default most general-purpose settings will be in .zshrc, we’ll be using the local file for our own custom settings. Let’s also create a ~/.zsh directory. This is where zsh plugins are commonly installed by default. I prefer manually installing plugins manually (as opposed to using zsh managers such as Oh My Zsh since I think it’s easier to maintain that way) so we’re gonna clone the autosuggest repository ourselves.

git clone https://github.com/zsh-users/zsh-autosuggestions ~/.zsh/zsh-autosuggestions

Then add the following line to the top of your .zshrc.local file.

source ~/.zsh/zsh-autosuggestions/zsh-autosuggestions.zsh

Restart your terminal for it to take effect.

zsh-syntax-highlighting

Next is syntax highlighting. A minor QoL sure, but still important for those long-winded command strings.

Installation is pretty much similar.

git clone https://github.com/zsh-users/zsh-syntax-highlighting ~/.zsh/zsh-syntax-highlighting

Add the following line near the top of your .zshrc.local.

source ~/.zsh/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh

As usual, restart your terminal for it to take effect.

Vim mode+

Since I’m a Vim user (and disciple, please use Vim), I tend to want to use Vim everywhere I can get it to work. And that means here in the terminal.

Add these in .zshrc.local

##### Enable zsh vim mode
bindkey -v

##### Change cursor shape for different vi modes.
function zle-keymap-select {
  if [[ ${KEYMAP} == vicmd ]] ||
     [[ $1 = 'block' ]]; then
    echo -ne '\e[1 q'
  elif [[ ${KEYMAP} == main ]] ||
       [[ ${KEYMAP} == viins ]] ||
       [[ ${KEYMAP} = '' ]] ||
       [[ $1 = 'beam' ]]; then
    echo -ne '\e[5 q'
  fi
}
zle -N zle-keymap-select
zle-line-init() {
    zle -K viins # initiate `vi insert` as keymap (can be removed if `bindkey -V` has been set elsewhere)
    echo -ne "\e[5 q"
}
zle -N zle-line-init
echo -ne '\e[5 q' # Use beam shape cursor on startup.
preexec() { echo -ne '\e[5 q' ;} # Use beam shape cursor for each new prompt.

##### Use vim keys traversing history the smart way:
autoload -U history-search-end
zle -N history-beginning-search-backward-end history-search-end
zle -N history-beginning-search-forward-end history-search-end
bindkey "^[[A" history-beginning-search-backward-end
bindkey "^[[B" history-beginning-search-forward-end
bindkey -M vicmd 'k' history-beginning-search-backward-end
bindkey -M vicmd 'j' history-beginning-search-forward-end

A bit of a mouthful I know but bear with me.

As commented, these changes allow you to use Vim in the terminal, change the cursor shape in your terminal as you switch modes, and finally (a personal favorite) allow the down(j) and up(k) keys on Vim to search your history based on your already typed command!

This means that if you typed git and went to command mode and pressed k, this will show the previous command you started with git as well! Personally I use this feature just about every day.

Again, restart your terminal for the changes to take effect.

fzf

We’re still not done with shell goodness with fzf or fuzzy finder. This little tool allows you to zip around your terminal whether you need to quickly change directories or open a file you simply couldn’t remember the full path/name of. Just fuzzy find it!

First of, install it.

brew install fzf
$(brew --prefix)/opt/fzf/install

This allows usage of the fzf cli command and enables a bunch of commands out of the gate.

Ctrl + T allows you to search for any file and selecting one returns it in your buffer.

Opt + C allows you to search for any directory and selecting one moves you to it.

By the way, to actually use Opt as a modifier instead of inserting special characters, go to iTerm2’s Preferences -> Profiles -> Profile -> Keys and set your Option key to Esc+.

These alone however are not enough to maximize your productivity with fzf. By default, fzf uses GNU find to execute its searches. Albeit well-featured, it’s not fast. To effectively fuzzy find, fzf will have to crawl over a lot of files to get you the most accurate match. To achieve this, you need to give it a helping hand (or two).

First, use ripgrep. Arguably the fastest cli search tool out there, this makes it the best default go-to command for fzf. To do this, simply set the following environment variable in your .zshrc.local.

export FZF_DEFAULT_COMMAND='rg --files --hidden --follow -g "!{.git,node_modules}/*" 2> /dev/null'
export FZF_CTRL_T_COMMAND="$FZF_DEFAULT_COMMAND"

These let fzf know to use rg as its default search tool. Next is to exclude certain paths from search. Basically paths where you know you won’t deal with in the terminal yourself. Examples of this are the /Library and /Pictures directories.

These come with macOS with its implementation of iCloud and certain apps. If you don’t exclude these, fzf will have to go through a lot of files to even get to the relevant ones (probably the one in your ~ directory). So if you do need to access those files, skip this step.

Create a .rgignore file. Any files/directories in this file will be ignored by ripgrep.

Sample file:

/Library
/Pictures

More experienced users might note that rg won’t work for directories and you’re right. This is well documented in ripgrep#169 and ripgrep#388.

So if we can’t use ripgrep and by default fzf’s change directory feature is still using find that doesn’t ignore directories, we can’t keep waiting for it to index all those files/directories all day now can we?

We can’t. So we need another tool. Namely, fd. Touted as the simple and more user-friendly alternative to find, fd has exactly the feature we need: an ignore file.

brew install fd

So much like before, we set the environment variable:

export FZF_ALT_C_COMMAND="fd --type d"

Then we create a .fdignore file that has the same contents as .rgignore. And we’re done.

Okay we’re halfway done. The following steps for this guide are specific to the tech stack I commonly use.

5. Node

As one of the most common developer tools, there are a lot of ways to install this. My favorite is nvm. This allows you to have multiple node versions in your machine and can easily switch between them with just a simple command.

Let’s get started.

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash

Afterwards, make sure the following snippet is in your .zshrc file.

export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm

Restart your terminal and execute command -v nvm. This should output “nvm” to tell you it’s working.

Running nvm install node will install the latest version and set it as your default Node version.

6. Python

macOS should already come packaged with Python by default. But any Python dev can tell you that it’s a bad idea to have all of your projects depend on a single version of Python, let alone the system-installed one.

So how do we fix this? We use a tool much like nvm—pyenv.

brew install pyenv pyenv-virtualenv

Make sure that the following line is in your .zshrc.

eval "$(pyenv init -)"

Restart your terminal and pyenv should be working.

Usage

Start with setting the default python version you want to use (which is not the system-installed one).

##### As of the time of writing, this is the latest I want my system python to run on
pyenv install 3.9.7
pyenv global 3.9.7

Running python in your terminal should use the version you’ve set as global.

Now say you’ve got a project that runs on Python 3.8.12 that as the experienced Python developer that you are, need to use a virtualenv on?

##### Install the needed Python version first
pyenv install 3.8.12 # 1
cd my-python-project # 2
pyenv virtualenv 3.8.12 my-python-project # 3
pyenv local my-python-project # 4

So what exactly happens after you run these series of commands? 1 and 2 are pretty self-explanatory. 3 just creates the virtualenv with the name you specified (by convention it shares the name of the project) This is possible since we installed pyenv-virtualenv along with pyenv using Homebrew. And 4 creates a .python-version file that has the name of the virtualenv you want to be activated as soon as you cd into that directory. Neat, huh?

8. Docker

Last, but not least, is Docker. Thankfully this is the easiest to install as it’s literally just going to Docker for Desktop’s website and just downloading and installing. Once it’s good to go it gives you a UI that is sort of Portainer-lite that has brief information regarding the logged in Docker user, running containers, and downloaded images.

Along with this of course is the Docker runtime that lets you run the docker command on your terminal.

And with that, we finally come to the end of this so-called 12-minute read which took anything but to write. I hope this helps you and my future self in setting up your development environment, be it in macOS or Linux, the way you want it.

As long as it uses Vim, you’re cool.

How to handle the LetsEncrypt expiry (and others like it)

Sat, 02 Oct 2021 00:00:00 GMT

What happened (and what will)

As of 30 September 2021, LetsEncrypt’s DST Root CA X3 has expired. And unlike a lot of major shit that broke the internet, this was not something due to what LetsEncrypt did or failed to do. Read more about this here.

Basically, certificates will expire.

What to do

Well, for starters, if you have been keeping on top of your updates, you should have no issues whatsoever. However, for legacy devices/servers that are not, well, that’s what this post is about. There’s already a lot of related write-up regarding the whys and hows so I’d focus on the actual things that I’ve had to do to get past this issue.

Ubuntu 14.04 LTS

Let’s start with the oldest. It’s as straightforward as deleting the expired certificate from the store.

sudo rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
sudo update-ca-certificates

Ubuntu 16.04 LTS

A bit newer but still outdated, this is more simple as it has the packages it needs to update your store for you.

sudo apt install libgnutls-openssl27 libgnutls30

And there you go. For other devices such as Windows 10 and smartphone OSs, this should help.

I’ll update this post as I encounter related issues that can or can’t be fixed by the above methods.

PyRMQ, Python with RabbitMQ—simplified so you won't have to.

Sun, 28 Mar 2021 00:00:00 GMT

The need

Released almost a year ago, PyRMQ was created to make RabbitMQ more accessible than it is. RabbitMQ, in case you didn’t know, is an open source message broker. To summarize the technology in so little words: it’s a way to manage, direct, and even contain connections between systems or points in the form of messages. Okay, but what does that really mean?

Exchanges, Queues, and Messages.

RabbitMQ primarily works with the above three. Exchanges are bound to queues which in turn house the messages that get sent to exchanges. Weird loop? Bear with me. All you need to know about RabbitMQ is that you take your message (can be anything from a JSON payload, text, or even blank), pick an exchange to send it to, the exchange will direct it to whatever queue it’s bound to, and your intended recipient that is set up to listen or consume from that queue will receive it!

Now, coming from a RESTful background, this may seem unfamiliar territory, and I agree. Since RabbitMQ follows a certain protocol, AMQP, it certainly has its own learning curve. As such, numerous libraries attempt to abstract all these in their respective languages. The one I’ll highlight in this post is pika for Python. An already amazing library by itself, we felt that we could abstract it further (kinda like Gatsby for React); especially since our architecture revolves around microservices—we’d be reusing all this code to boilerplate our RabbitMQ multiple times!

Enter, PyRMQ

With PyRMQ, it’s as easy as instantiating one class!

from pyrmq import Publisher
publisher = Publisher(
    exchange_name="exchange_name",
    queue_name="queue_name",
    routing_key="routing_key",
)
publisher.publish({"pyrmq": "My first message"})

And with that, you’ve already published your first message! Feel free to look over PyRMQ’s source code to see exactly which pika classes/methods it uses to make all this work.

Great, how do I receive this message then?

from pyrmq import Consumer

def callback(data):
    print(f"Received {data}!")

consumer = Consumer(
    exchange_name="exchange_name",
    queue_name="queue_name",
    routing_key="routing_key",
    callback=callback
)
consumer.start()

Just like that. This class launches a thread safe instance to receive your message.

As of the time of writing, PyRMQ has been battle-tested over at FDFC to achieve high availability with dealing with financial processes. Furthermore, PyRMQ is proud to hold 100% coverage with its tests.

We’re always looking for contributors! If you see an issue (even a typo, really), let’s do something about it at PyRMQ’s GitHub!

Wanna learn more? Head over to PyRMQ’s documentation!

Dark mode with Gatsby.js and Tailwindcss

Tue, 11 Aug 2020 00:00:00 GMT

Modern dark mode

Since you’re most probably a developer (let’s face it you’re only here because you googled “how to implement dark mode [tech stack]”), you surely know that dark mode is fortunately prevalent in mainstream UIs. It’s not unique to rated-R sites anymore. So naturally, you’d want this feature into as many screens you’re looking at.

What we’re working with

Now, evergreen browsers have mechanisms in place to support dark mode, along with other accessibility features such as reduced motion toggle. They expose this through a flag called prefers-color-scheme. This flag can only have two values: light or dark. Now you might be thinking that hey I already have a flag which tells me which theme the user wants by default, I can just base my styling around this! And you’re right.

But what if you want to toggle it?

This is the meat of this blog post. You see, that flag cannot be toggled locally. It is based on the user’s system preferences therefore only being changeable through their system. Ergo, we need to find another way of making that logic work. So is my plot twist alert accurate here? Is it not easy? Like any self-respecting developer would tell you: it depends. In our case, it depends on Gatsby.js and Tailwincss, the two things I happened to build this website with. (If you want more info on why I chose those, feel free to read my post about it.

Tailwindcss implementation

Let’s get the easy stuff out of the way. How should we style our components with our requirement in mind? Again, this implementation is not unique to Tailwindcss, but if it gets you to use it, why the hell not? Tailwindcss is awesome.

Now most guides will tell you to set your styles up with a dark prefix or handler. This lets you organize your site and codebase based on separated style concerns.

So what prefix do I use?

It’s easy to knee-jerk it with dark. That’s what the majority uses anyway. It comes from what I believe the mobile-first design philosophy that dictates designs should revolve around mobile and just adapt for desktop. So why not do the same for dark mode?

Let me propose a counter argument: dark first.

Hear me out. Since my website’s theme is primarily “dark”, it makes sense to make the variant styles light. This is also in line with how Tailwindcss’ variants were designed to work: use plain utility classes for your default design and add variant prefixes for the alternate or secondary one. So obviously your design is different so trust your own judgment.

Creating the variants

Creating your own mode prefix in Tailwindcss is done by adding a plugin to your tailwind.config.js file.

// ...
plugins: [
    plugin(function ({ addVariant, e }) {
      const lightSelector = `.light`;
      addVariant(`light`, ({ modifySelectors, separator }) => {
        modifySelectors(({ className }) => {
          return `${lightSelector} .${e(`light${separator}${className}`)}`;
        });
      });
    }),
  ]
// ...

To summarize this block of code briefly, this just allows you to write Tailwincss utility classes with your specified selector (.light in my case). modifySelectors just formats your utility classes in a way that we can scope for when we actually implement the toggle logic.

To actually use this variant, you have to enable it for the utility classes you actually need them for.

// ...
variants: {
    backgroundColor: [`light`],
    textColor: [`light`]
}
// ...

This helps Tailwindcss be conservative in generating its utility classes, of which there are a lot by default due to its design. You can read more about Tailwincss’ variant implementation here.

Now believe it or not, that’s actually it. You can now plug classes like text-white light:text-black to your components and have the class names get manipulated properly by Tailwindcss according to our modifySelector logic. But at this point, it’s not yet actually doing anything significant. The selector that our variant is depending on is still not present nor toggle-able.

Gatsby.js implementation

Full disclosure, this solution is based on Dan Abramov’s implementation of dark mode in his blog plus a few other tutorials I’ve read about the topic. Having said that, I believe this implementation of the solution is more approachable for developers looking for flexibility with this feature.

So first off, we need to structure the logic that we need. For my blog, I wanted a simple button that will toggle the mode on and off. Like a bulb. So I made a simple component that would do this for me:

<div
  onClick={toggleTheme}
>
  {theme === `light` ? <Night /> : <Day />}
</div>

Simple enough. Then, I have to define what toggleTheme does.

function toggleTheme() {
    window.__setPreferredTheme(theme === `light` ? "dark" : "light");
}

Okay hold up. Why are we suddenly accessing window functions? What exactly is this __setPreferredTheme()?

This is actually the trick to this feature. To give you a brief idea of the issues this approach solves, implementing dark mode, or really any theme picker feature, presents the web developer with the flicker problem. We have to execute (and set) our function early enough the browser is forced to evaluate its result before first paint happens aka that brief flicker of white (or your default bg color) you see before the browser applies the user-preferred theme.

So how and where do we place this function? The how of course is by hooking it to window. It’s global and easily set via script. As to where? Remember that we have to execute this function before anything visible is rendered. Gatsby.js offers a way to do this via their setPreBodyComponents hook. As the name suggests, this allows us to set a component before our html body. And as you may have guessed, we’re gonna put a script tag.

Since this Gatsby.js hook is trigered in the gatsby-ssr.js file, we should edit it like so:

const React = require("react");

exports.onRenderBody = function ({ setPreBodyComponents }) {
  setPreBodyComponents([
    React.createElement("script", {
      key: "gatsby-dark-mode",
      dangerouslySetInnerHTML: {
        __html: `
void function() {
  window.__onThemeChange = function() {}
  var preferredTheme
  try {
    preferredTheme = localStorage.getItem('theme')
  } catch (err) { }
  function setTheme(newTheme) {
    if (preferredTheme && document.documentElement.classList.contains(preferredTheme)) {
      document.documentElement.classList.replace(preferredTheme, newTheme)
    } else {
      document.documentElement.classList.add(newTheme)
    }
    window.__theme = newTheme
    preferredTheme = newTheme
    window.__onThemeChange(newTheme)
  }
  window.__setPreferredTheme = function(newTheme) {
    setTheme(newTheme)
    try {
      localStorage.setItem('theme', newTheme)
    } catch (err) {}
  }
  var darkQuery = window.matchMedia('(prefers-color-scheme: dark)')
  darkQuery.addListener(function(e) {
    window.__setPreferredTheme(e.matches ? 'dark' : 'light')
  })
  setTheme(preferredTheme || (darkQuery.matches ? 'dark' : 'light'))
}()
    `,
      },
    }),
  ]);
};

A bit of an eyeful to be sure, but let’s just focus on the important parts there. First, we define window.__onThemeChange as a utility to be overridden by our component later. Next, we check localStorage whether we already have the user’s preference locally. We just wrap it under a try-catch to silence any issues that may arise. You can improve this if you want. Then we just define a simple function that replaces the class defined in our html tag with the theme the user chose. This function is wrapped globally with window.setPreferredTheme.

Next is just some quality-of-life line that gets and listens for the value of prefers-color-scheme. So even if the user does change the setting on a system level, we can catch it. Finally, the setTheme() at the end just initializes the theme for the first paint.

With the above hook set, the pieces are now complete. We can top off our toggler component with the hooks it’d need to function as we’d like it to.

const DarkModeToggle: React.FC = () => {
  const [theme, setTheme] = useState(undefined);
  useEffect(() => {
    // @ts-ignore
    setTheme(window.__theme);
    // @ts-ignore
    window.__onThemeChange = () => {
      // @ts-ignore
      setTheme(window.__theme);
    };
  });
  function toggleTheme() {
    // @ts-ignore
    window.__setPreferredTheme(theme === `light` ? "dark" : "light");
  }
  return (
    <div
      className={!theme ? `invisible` : `visible cursor-pointer`}
      onClick={toggleTheme}
    >
      {theme === `light` ? <Night /> : <Day />}
    </div>
  );
}

So by using React hooks, we now have a component that will load the theme on render via useEffect() and set its displayed icon depending on the selector and use of useState().

And there you have it. Once you load this component up, clicking on it should load the proper selector in our html tag which in turn triggers the loadout our variant requires which shows the correct styles. In short, we now have dark mode.

To see all these components and logic written out in all their glory, take a quick look at this website’s source code. It contains bonus content such as implementing dark mode on code that is effectively unreachable by your Tailwindcss variants such as this blog section that is styled by the Tailwindcss typography plugin. You can star or help improve it if you want. Or not. I’m not telling you what to do. Well, except for this tutorial.

Again, if you’ve read up to this point and you’re not me, wow. And thank you. I hope you’ve found this helpful.

My first website

Mon, 20 Jul 2020 00:00:00 GMT

I’ve never had a website before

I’ve been a programmer for years now, and I’ve never had my own personal website. Why? Probably due to a lot of factors. Never really been into frontend development for one. Javascript is hell for two. And I’m just not a fan of publicly sharing my thoughts.

But I need one

I basically needed a place to write blogs (or self-tutorials since no one will read this blog but me if we’re being honest) on a place I control and won’t place a paywall for. Unlike *cough* Medium *cough*.

I also dislike long-form articles

So let’s get down to the nitty gritty. This article is meant to remind me of why the hell I decided to build this website this way while it’s still fresh in my mind. I’ll try to (hopefully) make sense of it all not just for me, but for you who must have mistakenly stumbled upon this site. Succeeding articles will be the bits that go deeper with code snippets regarding the tricky parts in developing this website.

Gatsby.js, Tailwindcss, and Typescript

That’s one SEO to-do out of the way. For a quick TLDR, here’s the source code.

Let’s go one-by-one as to why I chose the stack I did.

Gatsby.js

Personally, I just wanted to try React. My frontend experience up to this point was limited to Angular and Vue.js frameworks. Seeing that I didn’t want to spend a staggering amount of time boilerplating and benchmarking my website, I went with Gatsby.js. It has built-in features that I really wanted to try like GraphQL along with its claim that it’s blazing fast™. So when I was searching for a React framework, it boiled down to Gatsby vs Next since my website fit the case for JAMstack. And between the two of them? Gatsby’s documentation and community simply ran circles around Next’s.

Tailwindcss

When I first encountered Tailwind, proponents were claiming that it’s the CSS framework to use for devs who don’t want to get down with CSS. Now frankly this actually is underrating Tailwind. You even hear this kind of sentiment all the time.

Instead of learning X, use Y.

This train of thought has always been a popular if not detrimental trope for new tech. Current thing too complicated or low-level to learn? Invent a new tool or solution to abstract it. In fact, this is the main push of CSS frameworks in general like Bootstrap or Foundation. Just use their components masquerading as classes and you’re off. But in my experience with Tailwind, it actually works both ways. Sure, you don’t need to write CSS. Sure, it’s practically inline styles. Sure, it may look “ugly” looking at it the first time.

But it actually teaches you how CSS inherently work. By combining the various utility classes you begin to understand why certain CSS styles work the way they do. Perennial questions such as how to center a div both horizontally and vertically, done. How to align contents along its axis, done. How to use flex for various spacings, done.

And when you’re done? Not only do you know how to style websites with Tailwind, you also know how to style it with just CSS.

Typescript

You may be surprised to know that this one was a bit harder to justify. I’ve always been a fan of typed languages’ type checking. It simply helps you avoid common coding mistakes and have you really think about what your code does and what it needs or returns. Having Angular 2+ with built-in Typescript experience, this should be an easy choice for me.

So what made it difficult?

Community adoption. Loads of React tutorials or even Gatsby tutorials are written in plain Javascript. Naturally this presents a barrier for adoption as the very guides you read are not like your codebase. Having the latest and bleeding edge isn’t all rainbows and butterflies if you can’t maintain it.

Nevertheless, I still want Typescript to be treated as a first-class citizen for frameworks such as Gatsby and similar frameworks. Its benefits still outweigh its cons, albeit barely. So I went ahead.

DevOps

Nothing too fancy for this. I just wanted a development and release flow that’s cheap if not free, great with opensource, and easily maintainable. Furthermore, I would want to go back to this project not just to write articles, but to take as basis for my future frontend work. To that end, the following flow may be overkill.

CI/CD is handled by Github’s Workflow.
Commit formatting is handled by Commitizen.
Versioning is handled by semantic-release.
Dependency maintenance is handled by Renovate.
Hosting is with Netlify. I’ve debated going with AWS, and I might, when I get around to developing

my own comment section.

Conclusion

I think that covers just about everything for this site. I’ll periodically update this article as I iterate over its development.

So if you’ve made it this far and you’re not me, wow.