Aumala.dev

Installing Jellyfin on a Raspberry Pi

Fri, 21 Nov 2025 00:00:00 GMT

After running Jellyfin on my desktop PC for a while, I decided to move it to a Raspberry Pi 4 for 24/7 availability. This guide walks through the complete setup process I settled on after some trial and error—from flashing the SD card with Raspberry Pi OS to configuring external storage, running Jellyfin in Docker, and even adding Transmission for torrent downloads. Along the way, I'll share the workarounds I discovered for common issues and the media management approach that works best for keeping everything organized.

Why Jellyfin

::github{repo="jellyfin/jellyfin"}

Aren't you fed up with streaming services? I know I am. Streaming fatigue is totally a thing these days. In the past months, after successfully deploying a few web apps I started wondering if I could run my own media server at home. This is where I discovered Jellyfin, and you definitely don't need to be a seasoned engineer to run it. I can guarantee it lives up to its popularity—the official clients have beautiful UIs and cover all major platforms. The community keeps releasing third-party clients that scratch whatever itches the official clients couldn't reach. My personal favorite is Symfonium, an Android music player that can integrate your Jellyfin library as well as other platforms like Plex, Kodi, and more.

I got it running on my desktop PC first, it was very easy to install and it worked great. I cannot emphasize how cool it is to have your own streaming platform curated to your personal tastes. The only downside was having to walk into my bedroom to turn on the PC when I just wanted to lay on the couch and watch something on the living room TV. To keep Jellyfin available around the clock, I repurposed an old Raspberry Pi 4 and paired it with a 4TB HDD in a USB enclosure for storage.

Using a Raspberry Pi as a server instead of a conventional PC isn't as straightforward though. I flashed the Pi's SD card three times until I finally got the definitive setup, so I decided to write this guide for future reference.

System requirements

A Raspberry Pi that can run a 64-bit OS (Pi 4 recommended; Pi 3B+ also works) with at least 2 GB of RAM.
Reliable cooling such as a heatsink and fan—the board will be on 24/7.
32 GB (or larger) microSD card for Raspberry Pi OS; the OS, caches, and packages need the extra space to avoid premature wear.
USB HDD in a powered enclosure (4 TB in this build). SSDs can run bus-powered, but make sure your enclosure or adapter is reliable.
Wired network connection (strongly recommended).

SD card setup

Start by visiting raspberrypi.com and downloading Raspberry Pi OS (64-bit). Jellyfin 10.11.0, dropped support for 32-bit ARM systems so make sure you grab the 64-bit build even if your Pi has been running a 32-bit OS. The download should give you a raspios-trixie-arm64.img.xz image. Once you have the file, verify its checksum, decompress it, and flash the microSD card:

# Flash Raspberry Pi OS 64-bit to SD card
unxz raspios-trixie-arm64.img.xz
sudo dd if=raspios-trixie-arm64.img of=/dev/sdb bs=4M status=progress

Be sure to replace /dev/sdb with the correct device path for your SD card. When the dd command finishes, safely eject the card and insert it into the Raspberry Pi.

OS setup

Power on the Raspberry Pi and complete the Raspberry Pi OS first-run wizard. Once you reach the desktop, click the Raspberry menu in the top-left corner, head to Preferences → Control Centre, and open it as shown below:

In Interfaces, toggle SSH on as shown below—this is how you'll manage the server remotely:

Next, open the System section and set the boot target to CLI. The server doesn’t need the desktop environment, so booting straight to the console keeps things lightweight:

Close Control Centre, click the network icon in the panel, and choose Advanced Options → Edit Connections… as shown here:

In the Network Connections window, remove any existing profiles and create a new Ethernet connection—wired networking offers better reliability for a home server:

When the Editing Ethernet connection window opens, switch to the IPv4 Settings tab, set the method to Manual, and enter an address that is unused on your network. My gateway is 192.168.100.1, so I reserved 192.168.100.32 for the Raspberry Pi after checking availability; use whatever tool you trust to verify the IP you pick is free. Save the changes—you should now see the new static profile in the list.

With networking configured and SSH enabled, reboot the Pi to a TTY. Before moving on to the next section, take a moment to install your preferred CLI tools and dotfiles (shell, text editors, etc.), since you’ll spend most of your time managing the server over SSH.

From this point forward I’ll refer to the account created during the Raspberry Pi OS setup wizard as pi; feel free to substitute your actual username in the commands that follow.

Because the Pi now has a fixed address, you can simplify connections from other machines by adding an entry like raspberry.local 192.168.100.32 to their /etc/hosts.

Mount external storage

Connect the 4TB HDD (in its USB enclosure) to the Raspberry Pi. The drive must be formatted as ext4 before mounting. If it is a new drive, format and mount it under somewhere like /mnt/, then adjust ownership so the pi user can manage the files:

# Format drive as ext4 (if new)
sudo mkfs.ext4 /dev/sda1

# Mount and set ownership
sudo mkdir -p /mnt/hdd0
sudo mount /dev/sda1 /mnt/hdd0
sudo chown -R pi:pi /mnt/hdd0

Add an entry to /etc/fstab so the drive mounts automatically on boot. Use the drive's PARTUUID for reliability:

sudo blkid /dev/sda1
# Copy the UUID output and add something like the following to /etc/fstab:
PARTUUID=YOUR-UUID-HERE  /mnt/hdd0  ext4  defaults,noatime  0  2

In my case I mounted it to /mnt/hdd0 so that if ever decide to add more hard drives I can just call them hdd1, hdd2, and so on. I plan on using this hard drive for more things than just Jellyfin, so the actual media libraries will be stored under /mnt/hdd0/media/.

After updating fstab, run sudo mount -a to verify the entry works. Even with fstab configured correctly, you might notice the drive failing to mount after a reboot. Cheap USB enclosures sometimes back-feed power on the 5V line, so the Pi never fully resets and the disk wakes up in a “half on” state. Always use a drive enclosure with its own power adapter (the Pi can’t supply enough current), and prioritize high-quality, USB-compliant models—many include physical power switches that let you cycle the disk if the board hangs. If you encounter this back-powering quirk, unplug and reconnect the drive or plan an upgrade to hardware that doesn’t keep the board partially powered during shutdown.

Run Jellyfin with Docker

Install Docker using the official convenience script:

curl -sSL https://get.docker.com | sh

Create directories to persist Jellyfin's configuration and cache data:

# Create directories
mkdir -p /home/pi/services/jellyfin/{config,cache}

Pull the Jellyfin container image and start it. The pi user that you created during initial setup typically has UID and GID 1000, but confirm with id pi before running the container:

docker pull jellyfin/jellyfin

docker run -d \
  --name jellyfin \
  --user 1000:1000 \
  --net=host \
  --volume /home/pi/services/jellyfin/config:/config \
  --volume /home/pi/services/jellyfin/cache:/cache \
  --mount type=bind,source=/mnt/hdd0/media,target=/media \
  --restart=unless-stopped \
  jellyfin/jellyfin

The container binds to the host network, so Jellyfin will be available at http://<raspberry-pi-ip>:8096. Complete the Jellyfin web setup wizard to add your media libraries and create an admin account. With the drive mounted and Docker configured, the Raspberry Pi is now ready to serve your media collection.

Bonus: Install Transmission

With Jellyfin running, you can add a lightweight torrent client that downloads straight to the media drive. Transmission exposes a web UI and fits well in a headless workflow.

Install the daemon:

sudo apt install -y transmission-daemon

By default the daemon runs as debian-transmission. Switch it to your pi account so file ownership aligns with the rest of the media library:

sudo systemctl edit transmission-daemon.service

Add the following override:

[Service]
User=pi

Save and exit, then reload the unit file and stop the daemon before editing its configuration:

sudo systemctl daemon-reload
sudo systemctl stop transmission-daemon

Because the service now runs as pi, Transmission keeps its configuration in ~/.config/transmission-daemon/settings.json. Open it with your editor of choice and update the download paths as follows:

{
    "download-dir": "/mnt/hdd0/seeding",
    "incomplete-dir": "/mnt/hdd0/incomplete",
    "incomplete-dir-enabled": true,
    "umask": 2,
    "rpc-whitelist": "127.0.0.1,192.168.100.*",
    "rpc-host-whitelist-enabled": false
    // ... leave the rest of the settings as they are ...
}

Here’s what each of those keys does:

download-dir: where completed torrents land (/mnt/hdd0/seeding).
incomplete-dir and incomplete-dir-enabled: keep partial downloads in /mnt/hdd0/incomplete.
umask: using 2 (octal 002) creates files/directories that are group-writable.
rpc-whitelist: allows RPC access from localhost and 192.168.100.*; change this to match your subnet. You may also need to add docker container ip addresses here if you want future containers to access transmission.
rpc-host-whitelist-enabled: set to false so you can visit the web UI using any hostname that resolves to the Pi.

If you want to tweak additional options, the Transmission project keeps an excellent reference on editing configuration files [source].

Create the directories and restart the daemon:

sudo mkdir -p /mnt/hdd0/{seeding,incomplete}
sudo systemctl start transmission-daemon

Open http://<raspberry-pi-ip>:9091 to access the Transmission web interface. Add torrents there and they'll download directly into the media hard drive.

For torrent discovery I also rely on Jackett, which aggregates multiple trackers behind a single search UI. You can run it alongside Transmission with a simple docker run invocation from the project's Docker Hub page.

Managing your media library

Whether you torrent your media or rip your own discs, you need to organize your files properly. Jellyfin expects a specific file structure, so you need to be meticulous about your directory organization—no dumping files haphazardly into the media library. What I do with Transmission is that I let download-dir keep whatever naming the tracker provides and then I curate Jellyfin's folders with hard links so seeding continues untouched. Because /mnt/hdd0 is a single ext4 filesystem, it's possible to use ln to create links without duplicating data. For example:

mkdir -p "/mnt/hdd0/media/movies/Some Film (2025)"
ln "/mnt/hdd0/seeding/Some.Film.2025.WebRip.1080p/Some.Film.2025.mkv" \
   "/mnt/hdd0/media/movies/Some Film (2025)/Some Film (2025).mkv"

Feel free to script this process or use media managers; the key is to keep the original files in place while presenting Jellyfin with cleanly named directories. Here’s a small helper I use to reorganize music while leaving the original torrent folder untouched:

#!/bin/bash

# Usage: ./organize_artist_music.sh "Artist Name"
SEED_DIR="/mnt/hdd0/seeding"
MUSIC_DIR="/mnt/hdd0/media/music"

if [ -z "$1" ]; then
  echo "Error: Please specify an artist name."
  echo "Usage: $0 \"Artist Name\""
  exit 1
fi

ARTIST="$1"
ARTIST_DIR="$MUSIC_DIR/$ARTIST"

# Create artist directory with correct permissions
mkdir -p "$ARTIST_DIR"

# Process each album by this artist
find "$SEED_DIR" -mindepth 1 -maxdepth 1 -type d -name "$ARTIST -*" | while read -r album_path; do
    # Extract album name (remove "Artist - " prefix)
  album_name=$(basename "$album_path" | sed "s/^$ARTIST - //")

  # Create album directory
  album_dir="$ARTIST_DIR/$album_name"
  mkdir -p "$album_dir"

  # Hardlink all files (ignore errors for empty dirs)
  ln -f "$album_path"/* "$album_dir/" 2>/dev/null

  echo "Linked: $ARTIST - $album_name"
done

echo "Done! All albums by '$ARTIST' organized."

For example, if I want to import all my Bad Bunny albums into Jellyfin, I drop the releases in seeding via rsync, so they get stored like this:

/mnt/hdd0/seeding/Bad Bunny - DeBÍ TiRAR MáS FOToS
/mnt/hdd0/seeding/Bad Bunny - Un Verano Sin Ti

and then run ./organize_artist_music.sh "Bad Bunny" so they’re linked into the music library like this:

/mnt/hdd0/media/music/Bad Bunny/DeBÍ TiRAR MáS FOToS
/mnt/hdd0/media/music/Bad Bunny/Un Verano Sin Ti

All of my audio lives under /mnt/hdd0/seeding, even tracks I ripped from my own CDs, so every release follows the Artist - Album pattern and keeps seeding intact. Because the files are linked rather than copied, they count once on disk but appear where Jellyfin expects them. Feel free to adapt the idea—or swap in a media manager such as beets or Lidarr—if your library follows a different layout.

Closing thoughts

After running this setup for a few weeks, I'm happy with how it's performing. The Pi handles these relatively simple services well—I've deployed a handful of containers including a Glance dashboard for monitoring, and most of the time the CPUs are idle with RAM usage sitting right under 1 GB. I have a 5V fan running 24/7, which keeps temperatures around 40°C; active cooling is essential for a device running continuously.

I do occasionally experience playback lagging due to buffering, though that's likely more about my Wi-Fi setup than the Pi itself. The real limitations I've hit are around transcoding.

Obviously there's no support for 4K content—the Pi just can't keep up with that transcoding workload. I try to stick with 1080p content encoded in H.264 or H.265, which generally plays smoothly without transcoding. Even then, on rare occasions I'll get files with audio codecs like DTS that my Google Chromecast TV can't decode natively. When that happens, Jellyfin triggers transcoding on the Pi, which maxes out the CPU and causes noticeable lag. The solution is to take the file to a beefier machine, re-encode it with ffmpeg, and copy it back to the Pi. If you have access to an NVIDIA GPU, using NVENC can save you hours compared to software encoding.

Something I didn't expect was anime subtitles—the .ass format, which is like the gold standard of subtitles, requires transcoding, and while the Pi can process it, it's not fast enough to completely avoid stuttering. For now, I stick to .srt subtitles as a workaround.

I can definitely see myself upgrading to a Mini PC with an Intel N100 processor in a few years—or maybe even sooner if I can swing it. Not only is the N100 chip perfect for Jellyfin, offering hardware-accelerated transcoding without the power draw of a full desktop, but honestly, homelabbing is such a rabbit hole: you start with one service, and before you know it you want to run a dozen more. The open source community is full of wonderful projects that enable you to live a richer life, away from the corporate overlords that want to lock you into their walled gardens. I've cancelled all my subscription services thanks to this setup, and I encourage you to try it out for yourself.

OpenJisho Version 1.10

Thu, 19 Sep 2024 00:00:00 GMT

It's been almost three years since the first release of OpenJisho. There hasn't been much activity in the repository between 2022 and 2023, but I've been working hard this week updating dependencies and configuration files in order to add new features. I'm happy to announce a new version today with some significant changes.

New feature: Input Sentence

There's a new screen that lets you input a body of text and output a list of JMdict entries for every word found. You can access this screen from the main menu.

The UI is meant to be as similar as possible to the example sentences shown in the main screen. The app doesn't have yet an algorithm for accurately splitting sentences into Japanese words, so you have to separate words with spaces (' ') for this to work.

Real world text usually has verbs conjugated in forms that are not listed in the dictionary. If you'd like to look for the correct entry, but preserve the original conjugation in your sentence you can use Tatoeba indices syntax. Here is an example:

Notice how this sentence uses "建てられた" but we want to lookup "建てる" instead. By using the curly braces ("{}") we can tell the parser to lookup the previous word, but to display the one inside the braces. This is helpful in more complex bodies of text, when you need to constantly reread the whole thing to understand it without losing any context.

New Releases on GitHub

Starting with this version, OpenJisho releases are now going to be on GitHub instead of Google Play. You can use this website to download the latest version.

The reason for this change is that my Google Play account along with the OpenJisho listing has been removed. I don't know why Google did this, but inactivity was probably one of the reasons. I don't like using Google products, so I created a Google account specifically for publishing this app and never logged in again. If I do it again, it will most certainly end the same way, so I think GitHub is the better choice for this.

Google Play was convenient for pushing new versions to users. Since I can't use it anymore, I've also added a notification in the main screen whenever a new version available. The app now checks what the latest released version is using a single HTTP request to the GitHub pages site. It looks like this:

Setting up a new Ubuntu VPS

Mon, 19 Aug 2024 00:00:00 GMT

I've been doing a lot of backend development this past few months and all my deployments so far have been with Digital Ocean droplets running Ubuntu Linux. I think it's pretty cool how much you can do with a Linux box in the cloud for so cheap, so I decided to write a little guide with all of the things that I might want/have to do every time I create a new droplet.

Configure Vim

When setting up a new VPS we'll need to edit a few configuration files. Vim is my GOTO editor on any computer, and Ubuntu does have it installed right out of the box, but Vim's default config is pretty terrible. I keep my dotfiles in a GitHub repo so I can quickly download them to any system, desktop or server.

However, when it comes to production servers I would rather use a minimal config with no plugins to avoid any security risks with third party code. I usually just copy this:

syntax on

" Use spaces instead of tabs
set tabstop=2       " The width of a TAB is set to 2.
                    " Still it is a \t. It is just that
                    " Vim will interpret it to be having
                    " a width of 2.

set shiftwidth=2    " Indents will have a width of 2

set softtabstop=2   " Sets the number of columns for a TAB

set expandtab       " Expand TABs to spaces

" escape ESC
imap kj <Esc>

"split new buffers to right
set splitright

" numbers column
set nu
set relativenumber

filetype plugin on

" reload buffers from disk when they are updated externally
set autoread

" http://vim.wikia.com/wiki/Recover_from_accidental_Ctrl-U
inoremap <c-w> <c-g>u<c-w>
inoremap <c-u> <c-g>u<c-u>

" automatically change working dir to active buffer's dir
set autochdir

" don't show matching parenthesis
let g:loaded_matchparen=1

This should go in /root/.vimrc. If you are creating more users, you should create another .vimrc on their home directory.

Add your VPS IP address to your local /etc/hosts

If you are not planning to assign a public domain to your VPS, you can just make one up in your local machine's /etc/hosts. If you're going to frequently SSH into your VPS, it's better to do ssh [email protected] than ssh [email protected]. Just add this line to /etc/hosts:

107.168.0.11 my-new-vps.com

Create a new user

If you plan to run your own software on a VPS, you should really consider doing it with a less privileged user. Any security flaw in your code could compromise your entire server if it's running as root. To create a new user "gabriel" just run:

useradd -m -s /bin/bash gabriel

I add the -s flag to specify bash because the default is sh. The -m flag just creates the user's home directory /home/gabriel.

For users in server machines, I prefer to not use passwords. You can use the su command to switch between users, or just login directly into that user via SSH. For the later to work your new user needs a copy of your local machine's public SSH key in $HOME/.ssh/authorized_keys. You can also just copy the contents of /root/.ssh/authorized_keys but make sure the new user can read it!

Setup Git repositories

Git lets you use SSH urls for remote repositories, so you can push your code into your VPS. I use this frecuently to backup my private repositories. Git is already installed in Ubuntu, so to do this just create a directory and initialize a bare repository in it:

su -l gabriel
mkdir MyProject.git
git init --bare MyProject.git

Bare repositories only have the default branch, which is master. If you attempt to push to any other branch it will fail. This is a common issue because nowadays most repositories use main as the default branch. A good solution for this is to change the default branch name to main before creating any repositories:

git config --global init.defaultBranch main

To push into the new MyProject.git directory, the url would be:

ssh://[email protected]:/home/gabriel/MyProject.git

Don't forget that MyProject.git must be writeable for the user that will SSH into your VPS every time you want to git push. This is the reason why I used su -l gabriel before creating the repository.

Please note that MyProject.git is not the root of the repository. If you want to browse the repository files you must checkout those files into an existing directory like this:

git --work-dir=/home/gabriel/MyProject --git-dir=/home/gabriel/MyProject.git checkout -f

This is useful for deploying web applications to your VPS. For most projects it's astronomically faster to just push source files with git than uploading huge binary files.

Setup OpenVPN

You can use your VPS as a VPN if you install OpenVPN. This is particularly handy for me when I'm at airports using public wi-fi. I don't recommend using this for Netflix or any other streaming service because your VPS probably has a limited bandwith.

The easiest way to install and setup OpenVPN is with openvpn-install . Just log in as root and run:

curl -O https://raw.githubusercontent.com/angristan/openvpn-install/master/openvpn-install.sh
bash ./openvpn-install.sh

After a few prompts, the script will output a .ovpn file that you can download to your local machine. Your OpenVPN client should have an option to let you import configuration from a .ovpn file.

Setup Nginx with Let's Encrypt

If you are planning to deploy a web application with HTTPS, you can get a certificate for free with Let's Encrypt. Setting this up wth Nginx is super easy, but it takes a considerable number of steps that would need their own blog post. I always use this neat guide from Digital Ocean.

Setup a Firewall

Once you have all your applications ready to run in your VPS, you should setup a firewall for security. By default Ubuntu has ufw running but it is inactive. You have to configure it using the command line.

The first thing you should do is deny all incoming traffic as a default policy:

ufw default deny incoming

Then you should allow all applications and ports that you use. You can run ufw app list to check available apps in your system. The most important one is OpenSSH. You must allow it, otherwise you'll be locked out of your VPS.

ufw allow OpenSSH

You can also allow specific ports that you use. If you have an app listening on port 8080, run:

ufw allow 8080

Sometimes you might want to allow incoming traffc only for a specific IP address. For example, if I'm using this VPS to deploy a Postgres database (port 5432), I would like the firewall to only allow my application server (address 107.168.0.11) to access it. To do that, I just run:

ufw allow from 107.168.0.11 to any port 5432

On the other hand, if the application server denied all outgoing traffic, I can allow access to the database server with:

ufw allow out from any to 107.168.0.10 port 5432

Finally enable your firewall with:

ufw enable

After enabling, you can check your current configuration with:

ufw status verbose

Biometric authentication on Android

Thu, 22 Feb 2024 00:00:00 GMT

I recently tried out Biometric authentication on Android using the new Biometric library. The new API is really good. The OS now provides a dialog for biometric authentication, making things smoother and more consistent for users. Google even has this neat guide to help get you started with it. Unfortunately it is a bit lackluster, it was not clear to me how to get the whole thing running correctly. I wanted to expand on it and write a better guide on how to store a key for encryption/decryption on the Android keystore and retrieve it using the user's fingerprint.

The use case I have in mind is the user login. My goal is to provide an alternative to the username/password form with a biometric dialog. After the user's first successful log in, I'll encrypt the user's password and keep it my app's local storage. The key for encryption/decryption will be stored in the Android keystore and will require biometric authentication for access.

Installation

To use the new library we have to include this line in the app's build.gradle dependencies block:

// 1.1.0 is the latest stable version at the time of writing
implementation("androidx.biometric:biometric:1.1.0")

Check that biometric authentication is available

Although the new library supports up to API 23, not all devices actually have biometric features. Before attempting anything, we must check if biometric authentication is available using BiometricManager.canAuthenticate():

val biometricManager = BiometricManager.from(context)
return when (val result =
    biometricManager.canAuthenticate(BIOMETRIC_STRONG)) {
    BiometricManager.BIOMETRIC_SUCCESS -> 
        TODO("start biometric authentication")

    BiometricManager.BIOMETRIC_ERROR_NONE_ENROLLED -> {
        TODO("take user to biometric enrollment")
    }

    else -> TODO("biometric authentication not available. Show error message or skip")
}

We pass the BIOMETRIC_STRONG flag because fingerprint is considered strong authentication. The keystore does not allow weak authentication for accessing keys.

This method returns different status codes that we have to handle. If the result is BIOMETRIC_SUCCESS, we are good to go. If the result is BIOMETRIC_ERROR_NONE_ENROLLED, the user needs to enroll a fingerprint on system settings. You can launch an activity for that, but it's only available on API 30:

val intent = Intent(Settings.ACTION_BIOMETRIC_ENROLL).apply {
    putExtra(
        Settings.EXTRA_BIOMETRIC_AUTHENTICATORS_ALLOWED,
        BIOMETRIC_STRONG
    )
}
startActivity(intent)

Generate a secret key in the Android keystore

Once we have determined the user's device is ready for biometric authentication, we need to generate a secret key for encryption/decryption. To create it on the Android keystore, we need an instance of KeyGenParameterSpec:

const val KEY_SIZE = 256
const val SECRET_KEY_NAME = "mySecretKeyName"

KeyGenParameterSpec.Builder(
    SECRET_KEY_NAME,
    KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT
)
    .setKeySize(KEY_SIZE)
    .setBlockModes(KeyProperties.BLOCK_MODE_CBC)
    .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_PKCS7)
    .setUserAuthenticationRequired(true)
    .build()

This builder object defines the following things about our key:

It can encrypt and decrypt.
Its size is 256 bytes.
It uses Cipher Block Chaining (CBC) block mode.
It uses PKCS7 encryption padding scheme.
Requires user authentication for access. (This is essential for biometric authentication)

Then you can use KeyGenerator to generate it:

val keyGenerator = KeyGenerator.getInstance(
    KeyProperties.KEY_ALGORITHM_AES, "AndroidKeyStore"
)
keyGenerator.init(keyGenParameterSpec)
keyGenerator.generateKey()

Encrypt the user password with the new key

Now that the key is ready, we can use it to create a Cipher object that can encrypt the user password with AES. Passwords should never be stored as plain text. We should wait until the user authenticates with their password, grab it, and then create our cipher:

const val SECRET_KEY_NAME = "mySecretKeyName"

private fun getSecretKey(): SecretKey {
    val keyStore = KeyStore.getInstance("AndroidKeyStore")

    // Before the keystore can be accessed, it must be loaded.
    keyStore.load(null)
    return keyStore.getKey(SECRET_KEY_NAME, null) as SecretKey
}

private fun getCipher(): Cipher {
    return Cipher.getInstance(
        KeyProperties.KEY_ALGORITHM_AES + "/"
                + KeyProperties.BLOCK_MODE_CBC + "/"
                + KeyProperties.ENCRYPTION_PADDING_PKCS7
    )
}

val cipher = getCipher()
val secretKey = getSecretKey(secretKeyName)
cipher.init(Cipher.ENCRYPT_MODE, secretKey)

Please note that this cipher wont work until the user authenticates. Remember that .setUserAuthenticationRequired(true) in the KeyGenParameterSpec builder? We have to pass the cipher to BiometricPrompt so that we can use it after the user authenticates with their fingerprint.

val executor = ContextCompat.getMainExecutor(activity)

val biometricPrompt = BiometricPrompt(activity, executor,
    object : BiometricPrompt.AuthenticationCallback() {

        override fun onAuthenticationFailed() {
        }

        override fun onAuthenticationError(
            errorCode: Int,
            errString: CharSequence
        ) {
            handleError(errorCode, errString)
            Toast.makeText(this@MainActivity, errString, Toast.LENGTH_LONG).show()
            TODO("exit to next screen")
        }

        override fun onAuthenticationSucceeded(
            result: BiometricPrompt.AuthenticationResult
        ) {
            // cryptoObject cannot be null because we
            // explicitly pass it on authenticate()
            val resultCipher = result.cryptoObject!!.cipher
            encryptAndPersistPassword(resultCipher, password)
            TODO("exit to next screen")
        }

    })

val promptInfo = BiometricPrompt.PromptInfo.Builder()
    .setTitle("Biometric login for my app")
    .setSubtitle("Log in using your biometric credential")
    .setNegativeButtonText("Use account password")
    .setAllowedAuthenticators(BIOMETRIC_STRONG)
    .build()

biometricPrompt.authenticate(
    promptInfo,
    BiometricPrompt.CryptoObject(cipher)
)

Here we create a BiometricPrompt object,setting the callbacks for events that occur while the user interacts with the dialog. BiometricPrompt.authenticate() takes a PromptInfo object with the text that we want to display and a CryptoObject with our cipher to show the biometric authentication to the user. Now let's talk about each callback:

In the onAuthenticationFailed() callback we don't really need to do anything. This gets called every time the fingerprint does not match. The dialog shows this error and lets the user retry multiple times before cancelling the operation.

onAuthenticationError() is for errors that cancel the operation. Maybe the user exceeded the maximum number of attempts, or it got cancelled by user action. Most of the time we can just show the error to the user and exit, but there are a few errors that should be handled differently. You can see this in handleError():

private fun handleError(errorCode: Int, errString: String) {
    if (errorCode == ERROR_NEGATIVE_BUTTON) {
        // User clicked negative button to close the dialog
        TODO("exit to next screen")
        return
    }

    if (errorCode == ERROR_USER_CANCELED) {
        // User clicked outside the dialog to close it
        TODO("exit to next screen")
        return
    }

    // Show the error to the user
    Toast.makeText(this@MainActivity, errString, Toast.LENGTH_LONG).show()
    TODO("exit to next screen")
}

onAuthenticationSucceeded() is where we get our cipher back with a valid key and encrypt the password. Here's the implementation of encryptAndPersistPassword():

private fun encryptAndPersistPassword(cipher: Cipher, password: String) {
    private val defaultCharset = Charset.forName("UTF-8")

    val encryptedBytes = resultCipher.doFinal(
        secret.toByteArray(defaultCharset)
    )
    val encryptedString = encryptedBytes.toBase64String()
    val ivString = resultCipher.iv.toBase64String()

    TODO("persist encryptedString and ivString")
}

After encrypting there are two byte arrays that we need to persist: the password and the initialization vector (IV). The cipher won't be able to decrypt the password unless you provide both. For persisting you could use SQLite or just SharedPreferences. It's far easier to persist strings than byte arrays, so I convert them to Base64 strings. These are the extension methods that I use:

fun ByteArray.toBase64String() = Base64.encodeToString(this, Base64.DEFAULT)

fun String.toBase64ByteArray() = Base64.decode(this, Base64.DEFAULT)

Decrypt the user password

The next time the user wants to sign in we can just decrypt the password and instantly authenticate with the server. Once again, we have to create a Cipher object, but this time in DECRYPT_MODE and pass the IV we persisted at the end of the previous section:

val cipher = getCipher()
val secretKey = getSecretKey()

val ivArray = ivString.toBase64ByteArray()
cipher.init(Cipher.DECRYPT_MODE, secretKey, IvParameterSpec(ivArray))

Then, similar to the previous section, we have to create a BiometricPrompt instance and call authenticate():

val promptInfo = BiometricPrompt.PromptInfo.Builder()
    .setTitle("Biometric login for my app")
    .setSubtitle("Log in using your biometric credential")
    .setNegativeButtonText("Use account password")
    .build()

biometricPrompt.authenticate(
    promptInfo,
    BiometricPrompt.CryptoObject(cipher)
)

The BiometricPrompt callback implementations are pretty similar to last time. Error handling on onAuthenticationError() is bit different because ERROR_NEGATIVE_BUTTON now has to revert the UI to the old username/password form.

if (errorCode == ERROR_NEGATIVE_BUTTON) {
    // User clicked negative button to close the dialog
    TODO("revert to username/password form")
    return
}

if (errorCode == ERROR_USER_CANCELED) {
    // User clicked outside the dialog to close it
    return
}

// Show the error to the user
Toast.makeText(this@MainActivity, errString, Toast.LENGTH_LONG).show()
TODO("exit to next screen")

Finally, the code to decrypt the password on onAuthenticationSucceeded() is very straightforward:

val decryptedPassword = resultCipher.doFinal(
    encryptedString.toBase64ByteArray()
)
TODO("log in with the decrypted password")

And that's it. We now have biometric authentication in our app. The new biometric prompt has great UX. If something goes wrong, the user can always fallback to the username/password form.

Announcing OpenJisho

Mon, 06 Dec 2021 00:00:00 GMT

I'm happy to announce that I am releasing "OpenJisho", a Japanese dictionary Android app in the Google Play store today. As implied by the name is completely open source and you can get the source code on GitHub. If this sort of thing piques your curiosity, in this post I'll go into detail about this app and explain some of the technical decisions of the project.

::github{repo="GAumala/openjisho"}

:::warning App is no longer available in Google Play, but APKs are still available on GitHub :::

This app lets you type any word, in English or Japanese, and it will show you any matching entries from well known dictionary files: JMdict and KANJIDIC.

Additionally, there is a "Sentences" tab that will show you matching example sentences from the Tatoeba project.

If you want to search entries or sentences with a particular kanji, but you don't know how to type it in, there is a feature that lets you search it by radicals using RADKFILE.

All queries work offline because dictionary files are downloaded during the first time setup and stored in a SQLite database. An internet connection is only required during the setup.

Technical details

The rest of this post is aimed at Android developers, so you can stop reading if this isn't you.

This codebase reflects pretty much all my opinions about Android development. I started this project two years ago, so it doesn't have all the newest tools like Compose or Navigation. I avoid adding libraries unless I really need them. Most of the time I try to use what the Android platform already has or I try to implement it myself, unless it's not worth the effort. For example I use HTTPUrlConnection for downloads and JSONObject for JSON serializing/deserializing. I don't use dependency injection frameworks either. I think they are too complex. Dependency injection is important though. I just do it manually, creating objects as they are needed in factory classes.

The App architecture is MVI. I use my own implementation, which I've already written about in the past, so I won't talk too much about it here. It is designed as a single activity application. Every screen is implemented as a Fragment class. I transition between fragments manually using fragment transactions, as described in this previous post. Most transitions use a simple "slide" animation implemented in XML. Some of them, like the one used for search by radicals, use the Transitions API, which looks really cool. All transition animations are implemented in the MainTransitions class, in case you want to check them out.

A funny thing about animations in this app, is that I added second activity only because I wanted to use the default animation that Android uses for activities. I really liked the default animation that android uses when it launches a new activity on my Oreo phone, so I tried implementing it for fragment transitions. Unfortunately, it was not possible. I don't remember the exact reason, but I think it was because it was implemented in a private XML file in the Oreo source code, which made it unavailable on older Android versions. So for things like entry details, I launch a secondary activity to display the next fragment just for the free animated transition. I may end up removing this later on. Having a single activity makes things a lot simpler.

For the SQLite database I use Room. There is a lot I could write about Room and how I use it for dictionary queries, but it would probably be long enough to warrant its own post. Overall I'm really happy with it, specially with the support for full-text search, which helps a lot when searching for sentences. The only problem that I have is that there is no easy way to rank full-text search results by relevance. For dictionary entries, I do try to move exact matches to the top of the list, but I use pagination to display the results list. If the best result is not on the first page, there's nothing I can do about it. If only the mega corporation behind Room had some experience with search engines...

The first time setup that populates the the SQLite database is another very large topic that could have its own post. It has to download a few files, parse them, and store the parsed data into the database in a way that is properly indexed. This is done concurrently using coroutines. It's really good. I can't imagine doing this before coroutines got released. I use channels to allow concurrent tasks to report progress at every step. Then, the UI thread can pick one task and display its current progress to the user. When that task completes, another one gets picked, and so on until all tasks complete.

It can take a couple of minutes to finish all the setup tasks. The longer time this goes on, the more likely unexpected failures can occur. One of the measures I've taken is to use a foreground service to avoid getting killed by the OS when it needs to reclaim some memory. However, even with that, there's still plenty of work needed to make it robust, like implementing checkpoints to avoid retrying tasks that already completed or sending the appropriate HTTP headers to resume downloads.

If you are interested the whole setup, you can check out the SetupWorker class. This is probably the most mission critical class of the app so I added plenty of unit tests to ensure that it works correctly and that it's able to handle unexpected errors.

I could go on all day talking about the implementation details of this app, but I think I should wrap it up here. If you are interested in working with the codebase but have questions, feel free to open an issue on GitHub and I'll try to help. I do hope to get a few collaborators for maintenance and new features.

Arch Linux won't boot, now what?

Fri, 06 Nov 2020 00:00:00 GMT

I've been using Arch Linux for the last 5 years and I'm very happy with its simplicity. It's great to be able to install only the packages that you actually need and get their the most up-to-date versions. Arch Linux is actually really stable, but once a year or so my system breaks. This usually happens after updating the system or messing with configuration files. As this sort of incident becomes more rare I tend to forget what to do to fix my broken system, so I decided to write here the common steps that I take in these situations.

Keep system backups

Before doing anything, make sure you keep system backups. I think I can't stress enough how important it is to have full backups of your hard drive. I don't usually have to restore from backup but it's good to be prepared for worst case scenario. Please bear in mind that disks can and will fail at some point. Don't lose your data.

I try to backup my hard drive before a system update at least once every 3 months. I've been keeping in my desk a USB flash drive with Clonezilla installed for years. This makes it super easy to clone the entire disk to an image in an external hard drive. I just boot Clonezilla from the USB drive and follow the "beginner" steps to clone my disk. When it comes to restoring the disk from a saved image, the process is just as simple and straight-forward.

Boot Arch from USB flash drive

When your system breaks and you can't log in, the first thing you have to do is install Arch Linux on a USB flash drive and use it as a rescue USB. This gives you a terminal that you can use to access your system. You are likely to spend a good amount with this bare bones terminal so I suggest to take a few minutes to tweak it a little to make it more comfortable for you. In my case, my keyboard has a Spanish layout so I set the keyboard layout accordingly:

loadkeys es

Mount the Linux filesystem

From the live installation, You can access your files and data by mounting the Linux filesystem partition.

mount /dev/sdaX /mnt

/dev/sdaX has to be replaced with the actual partition. You can check that by running fdisk -l. In my case, this is the output:

Device       Start        End    Sectors  Size Type
/dev/sda1     2048     196607     194560   95M EFI System
/dev/sda2   196608    8007679    7811072  3.7G Linux swap
/dev/sda3  8007680 3907028991 3899021312  1.8T Linux filesystem

As you can see /dev/sda1 is the "boot" partition, /dev/sda2 is used for swap memory and /dev/sda3 is the one that I have to mount.

Mounting the filesystem by itself is not very useful. To actually use the files and programs installed in that partition you have to log in as a known user, or "chroot" into the system.

chroot

Now that the filesystem is mounted at /mnt (or wherever you want), you can chroot like this:

arch-chroot /mnt

Now you have root access to your system, you are free to do anything you want. Before proceeding it would be wise to mount the boot partition at /boot because the root of the issue might be there.

mount /dev/sdaX /boot

Once again /dev/sdaX has to be replaced with the actual boot partition as listed by fdisk -l.

As root user, here are some things that I suggest doing:

Check recently edited config files

Did you recently edit configuration files manually for your bootloader, window manager or desktop environment? You definitely want to check those and try to revert any changes that could have gone wrong. It's a good idea to keep a backup copy of a configuration file before editing, specially if you don't really know what you are doing.

Check your kernel

If your system fails to boot with an error message Error loading \vmlinuz-linux: not found or similar, you might want to check that your kernel is installed correctly. Run pacman -Q linux and uname -r, they should have the same kernel version. If they don't, or you are just feeling paranoid, you can reinstall it with pacman -S linux.

Rebuild the initramfs image

It might be a good idea to rebuild the initial ram disk environment, specially if you reinstalled the kernel, although technically pacman hooks should take care of this. Make sure your boot partition is mounted at /boot, otherwise none of this may work, then just run this command:

mkinitcpio -P

Reinstall the bootloader

Reinstalling the bootloader or rebuilding its configuration files can also help fixing the system. This may be necessary after doing changes inside the boot partition, like when rebuilding the initramfs image.

In my case I use GRUB, so reinstalling it is fairly easy. Again, make sure the boot partition is mounted at /boot, then run these two commands:

grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB
grub-mkconfig -o /boot/grub/grub.cfg

Restore all packages to a specific date

Probably the most likely reason for Arch breaking is a system update (pacman -Syu). Sometimes new package versions introduce breaking changes that are incompatible with the rest of the system. There's not much you can do here, other than helping out in the bugtracker and waiting for the issue to get sorted out. In the mean time, you can revert all your packages to a specific date in the past using the Arch Linux Archive, a server that stores official repositories snapshots and provides URLs to retrieve them easily. If you don't know what date you should revert to, you can check pacman logs at /var/log/pacman.log. In my case, my last successful update was on September 30, so I'll use that.

First you have to edit etc/pacman.conf and set the Arch Linux Archive URL for every repository. Here's how my config file looks normally:

[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

[community]
Include = /etc/pacman.d/mirrorlist

[multilib]
Include = /etc/pacman.d/mirrorlist

To rollback, instead of including the mirrorlist, set the snapshot URL like this:

[core]
SigLevel = PackageRequired
Server=https://archive.archlinux.org/repos/2020/09/30/$repo/os/$arch

[extra]
SigLevel = PackageRequired
Server=https://archive.archlinux.org/repos/2020/09/30/$repo/os/$arch

[community]
SigLevel = PackageRequired
Server=https://archive.archlinux.org/repos/2020/09/30/$repo/os/$arch

[multilib]
SigLevel = PackageRequired
Server=https://archive.archlinux.org/repos/2020/09/30/$repo/os/$arch

Finally, run pacman -Syyuu to update the database with the archived packages. This might take a while, specially if you depend on close mirrors for fast downloads, but once it's done your system should be working as it did before. You are now effectively stuck in the past, though. Once you are sure that there are no more broken packages in the official repositories you can revert the /etc/pacman.conf changes and update your system again.

Navigating with fragments

Sun, 03 May 2020 00:00:00 GMT

Navigating between different "places" of your app, is probably one of the most complicated things in Android. There are many APIs that you can use depending on what components you are using (activities or fragments), and how do you want to pass data (if any?) between them. Unfortunately, I don't think there's a silver bullet for this, so I'll just write about the method that I like to use the most.

First of all, I think navigation is easier if you have a single activity with multiple fragments. In fact, Google recommends using single-Activity apps. I know that there have been plenty of issues with fragments over the years, but these days with AndroidX libraries, they work great. Hopefully they will get only better. This doesn't mean that you are restricted to having only one activity. There are valid cases for launching a new activity, but it's better to minimize those.

The only problem that I still have with fragments is the back stack. It's nice to have an easy way to return to previous fragments, but the fragment back stack API is very limited and clumsy. Sure, you can go back to the last fragment with popBackStack(), but if you want to go back to an arbitrary earlier fragment, or clear the whole stack, it's not very helpful. This is why I prefer to implement this stack myself and have full control over it. Going back to any previous fragment is easier if the current fragment holds the data necessary to recreate any of those fragments.

A good place to store this data is the fragment's arguments. When you create a fragment, you can use setArguments() to pass data to it inside a Bundle. This is important because the OS will persist this data so it doesn't get lost when the fragment gets destroyed for whatever reason. This isn't unique to fragments. It also happens to activities and extra data passed to them via Intent. If you pass your back stack data via arguments, it will never get lost.

Consider a scenario in which there is a MyListFragment and when users click one of the items in the list they navigate to MyDetailFragment so that they can see more data about the selected item and maybe do some editing. If they do edit it, then when they go back to the first fragment, the update should be instantly reflected on the list.

Here's how I would implement this. The code for going into MyDetailFragment would be something like this:

val listState = captureListState()
val bundle = Bundle()
bundle.putParcelable("myListState", captureListState())

val newFragment = MyDetailFragment()
newFragment.arguments = bundle

fragmentManager.beginTransaction()
    .setCustomAnimations(
        R.anim.enter_from_right,
        R.anim.exit_to_left,
        R.anim.enter_from_left,
        R.anim.exit_to_right)
    .replace(R.id.container, newFragment)
    .commit()

The function captureListState() returns a Parcelable object that MyDetailFragment can retrieve from its arguments and use to reconstruct the list. Implementing Parcelable manually is error prone so I prefer to have some tool do it for me. I usually go with data classes annotated with @Parcelize, or AutoValue if I have to do this in Java.

When MyDetailFragment is created, it should retrieve listState from arguments and keep it somewhere convenient like a view model so that it can be updated if the user edits the data. Then, an OnBackPresedCallback should be added to reconstruct MyListFragment when the user presses back.

private val onBackPressedCallback = object: OnBackPressedCallback(true) {
    override fun handleOnBackPressed() {
        val bundle = Bundle()
        bundle.putParcelable("savedListState", getUpdatedListState())

        val newFragment = MyListFragment()
        newFragment.arguments = bundle

        fragmentManager.beginTransaction()
            .setCustomAnimations(
                R.anim.enter_from_left,
                R.anim.exit_to_right,
                R.anim.enter_from_right,
                R.anim.exit_to_left)
            .replace(R.id.container, newFragment)
            .commit()
    }
}

Once again I replace the current fragment. There are no fragments to "pop" here. When the user presses back, a new instance of MyListFragment is created with the updated list state. Since there's no back stack collecting previous fragments, I am free to choose where to navigate, not just back to MyListFragment. Under certain conditions I may want to go back to somewhere else like HomeFragment and completely forget about MyListFragment. With this approach, is super easy to choose a new destination in handleOnBackPressed().

Notice that the animations used here are the "opposite" of the previous transaction. This gives the illusion of going back to the original fragment. End users will never realize that it's actually a new fragment with the same state. Rather than modifying existing fragments in a back stack, I create a new "copy" of that fragment with updated data. I personally like this functional approach to managing fragments, but it is not perfect. Here are some potential issues:

Scaling to larger stacks

This example works really nice because there's only one fragment the user can go back to. In fact you can get the same result for free by calling addToBackStack() in the transaction. But what if there were four, five, or an unbounded number? You'd have to add a value to the bundle for each of them. This isn't a common thing, most apps have only so many fragments in the back stack. But, if you are struggling with this, then it's better to use something like a queue in the arguments bundle that you pop each time user presses back. It might be a little more work, but it pays off. If you implement the most appropriate data structure for your use case, managing your fragment history is a breeze.

Coupling fragments

For a fragment to recreate its previous fragment, it has to know about it. This means that these two are coupled. Coupling is generally a bad thing in programming, and this is no exception. Fragments are meant to be reusable. If it only works if it follows a particular fragment, then it's not very reusable.

For example, if I have a ChangePasswordFragment I will certainly navigate to it from SettingsFragment so that users can change passwords at any time. Additionally, I may want to navigate to it from LoginFragment after an user clicks "Forgot my password" and proves to be the legitimate account owner. This fragment must be reused in both cases, but they must return to completely different places once the task is complete.

I shouldn't have to modify ChangePasswordFragment every time I want to reuse it. It's better if the fragment itself knows nothing about the fragment(s) that came before it. A solution for this would be to declare an abstract method in ChangePasswordFragment that gets called when the user should navigate away from it. It would be something like this:

abstract class PasswordChangeFragment: Fragment() {

    abstract fun onQuit()

    // actual change password UI & logic here. 
    // When the user presses back or sets a new password, 
    // onQuit() gets called.
}

Then, PasswordChangeSettingsFragment and PasswordChangeLoginFragment extend it, recreating a different fragment in the onQuit() method. I'm personally not a big fan of inheritance, but this seems like an appropriate case to use it.

Complex views

For some views, it may be too difficult to capture its current state and recreate it in a new fragment. This may be even impossible for some custom views that only expose a limited API. Ideally all views should be simple and easy to reproduce, but sometimes it can't be helped.

I once tried to restore a SearchView inside a Toolbar after displaying search results in a new fragment. In the end, I decided that it was far easier to just leave it alone and display search results in a secondary activity.

Large amounts of data

Passing data in arguments is really convenient for persistence, but it is not meant to handle large amounts of data. Bear in mind that space for persisted data is limited and serialization should run as fast as possible. Arguments should only hold the minimum amount of data for the fragment to be able to work.

If a list fragment displays hundreds of thousands of items, it's probably not a good idea to put them all in a Bundle. If the data is loaded from a server, the it might be better to just reload it when the user returns to ensure they always see the most up-to-date data. If you are confident about the data not changing too often then you should cache it so that reloading is instant.

Animations

There are a few very solid APIs for animating fragment transitions, but you may face issues. A quick Stack Overflow search shows hundreds of questions for fragment animations. The new FragmentContainerView addresses one of these bugs, so make sure you are using it.

Even after fixing all bugs, not every animation may be possible with fragments. For instance, the only Android device I currently own runs Android Pie. I really like the default animation used for activities in this version. Unfortunately, that animation can't be recreated with fragments by simply copying the source, as it uses a cliprect tag, which is a private component. I have no idea why they do this.

This means that if I want to use that particular transition, I have to launch a new activity. There's no other way around it. And it goes without saying that newer versions of Android may replace it with something completely different, so its not reliable.

There may be other shortcomings that I've missed here, but I've had great results with this method. I use it 90% of the time. In very few cases, I find it easier to just launch a new activity with startActivityForResult(). I think it's super important to use the best tool for the job instead of rooting for a one size fits all solution.

I'm also keeping an eye on new APIs like the Navigation component. I'll be happy to adopt them once they become more stable, and prove to be objectively better, but for now I'm good with this.

Creating stickers with GIMP

Sat, 01 Feb 2020 00:00:00 GMT

I'm no graphic designer, but every time I need to do some basic image editing I go with GIMP. I always try to go with free software, even for the most meaningless tasks. GIMP may not be the most popular program, or even the best one, but it is free and you can do a lot with it. To illustrate this, I'll show you how to use GIMP to create stickers for Telegram and WhatsApp. The goal of this tutorial is to use create this sticker from a photo:

Even if you are not interested in stickers, you can still learn how to work with layers, remove the background from photos, and add smooth borders to any kind of shape.

Setup the canvas

To begin, create a new 512 x 512 pixels image with transparent background. Click File > New in the menu bar and fill out the dialog form. Make sure you open advanced options and fill with transparency, otherwise you'll get a white background.

Now that the canvas is ready it's time to open the image you want to use for your sticker. Click File > Open in the menu bar and select your file. You should have two windows like this:

Use the Rectangle Select Tool (R) to select the part of the image you are interested in (it doesn't have to be perfect tight), copy it, and paste it into the empty canvas. If it's bigger that the canvas you can use the Scale Tool (Shift + S) to shrink it. Since you pasted the image into the canvas, you probably have a floating selection layer with the pasted content in the layers window. This needs to be in a proper layer so right click it and click To New Layer in the context menu. If you don't have any floating layers, then it probably already got created or merged with the background, so you can just skip to the next section.

Remove the background

After the new layer is ready it's time to cut out the image with the desired shape and remove the background. This is the longest step, but it isn't really that hard. You just need a little patience. First, zoom in to at least 200% and use the Free Select Tool (F) to click on the points where you want to cut. You complete the selection by ending at the same point where you started it.

Here you should strive for smooth shapes. It doesn't have to be perfect but avoid sharp corners or spikes. Also try to leave some space between the selection and the content that you want to be visible. The resulting selection looks like this:

After the selection is ready, it's easier to cut it out by just sending it to a new layer. You can create the new layer in the same way as you did after pasting and scaling. The only difference is that this time the selection isn't floating. To float it, right click inside the selection and then click Select > Float in the context menu.

Now that you have a floating layer with the actual sticker content, you can remove the background by deleting the underneath layer. In the layers window, right click it and then click Delete Layer in the context menu.

Now that you finished cutting, you should only have one layer that covers the whole canvas and contains the sticker at the center. If you have more layers, merge them down. In the layers window, right click the top most layer and click Merge Down.

Add a border

The sticker is almost ready, but it's good practice to include a white border so that it is perfectly visible no matter what background is used in the messaging app. For this matter you must select the sticker's precise shape. The fastest way to do this is by using the Fuzzy Select Tool (U) to click anywhere in the transparent background and then inverting that selection by clicking Select > Invert in the menu bar.

Once you have the selection ready, create the border selection by clicking Select > Border in the menu bar and fill out the dialog form. I usually go with a border size of 4px.

The border is not created yet, there is only a selection for it. Before proceeding, you might want to make it smoother by applying a Gaussian Blur. To do this, click Filters > Blur > Gaussian Blur in the menu bar and fill out the dialog form. The only relevant values are Size X and Size Y. I usually go with a value of 4 for both, just like with the border size.

Now that the border selection is ready, just use the Bucket Fill Tool (Shift + B) to paint it white, and that's it! You are ready to export.

Export and distribute

It is recommended to export in the WebP format, as it is the most efficient format for stickers, so choose that in the export dialog. Unfortunately, this isn't supported everywhere yet, so you might want to export as PNG as well. If you keep the GIMP .xcf file, you can later export again in whatever formats you may need. Now you are ready to upload your sticker to your platform of choice.

Uploading to Telegram is as easy as texting the @stickers bot and following its instructions. Once your pack is ready, users can download it from within the app.

WhatsApp is more complicated as it requires you to build a whole app to distribute your stickers. The good news is that WhatsApp already provides sample apps for third party stickers so you can just fork the repo and replace the sample sticker pack with your own.

Working with byte streams in Kotlin

Mon, 27 Jan 2020 00:00:00 GMT

Sometimes it is necessary to process data that is so large that it is no longer practical to load it all into memory. This is often the case in mobile apps, where computational resources are very limited, so it's better to stream the data and avoid an OutOfMemoryException. Recently I was working in an app written in Kotlin that processes big files, up to 300 MB, so I decided to write down some useful tips for working with byte streams in Kotlin.

Extension functions

The Kotlin standard library has lots of extension functions that improve existing Java APIs. Most of them are really easy to implement, but nevertheless it's great to have them at your disposal, plus, they often make code more readable. Here are some of my favorite extensions for byte streams:

File.inputStream() Convenient function for opening an InputStream to a file.
File.outputStream() Convenient function for opening an OutputStream to a file.
InputStream.reader() Creates a Reader from an InputStream. Useful for parsing text files. File also has a similar function, in case you don't need the InputStream.
Closeable.use(). This makes sure that the stream is closed after you are done using it. This is great for avoiding leaking resources.
InputStream.copyTo(). This copies the contents of an InputStream to a OutputStream. This is super useful for copying or downloading files.

As an example of the usage of some of these functions here's how you could implement a function for copying files:

fun copyFile(sourceFile: File, destinationFile: File) {
    sourceFile.inputStream().use { input ->
        destinationFile.outputStream().use { output ->
          input.copyTo(output) 
        }
    }
}

This looks nice, but there's no need to add this to your codebase because there is already and extension function for copying files, File.copyTo(), and it does exactly this but contains additional validations and configurable options.

The standard library is great, but it can only contain so many functions. As it is included as a regular dependency in Android projects, new problems could arise if it becomes too bloated. There are some things that you will have to implement yourself or look elsewhere. The next section is an example of this.

Tracking the number of processed bytes

Neither Java nor Kotlin have a built-in solution for tracking the number of processed bytes in a stream so that the UI can display the progress of the task. This is important because if you are processing large amounts of data, it will take a proportionally long amount of time. The user should be able to see the progress of the task and estimate how much time is left to complete it.

The solution is to create wrapper classes that monitor the invocations of the read() and write() methods, tracking the number of processed bytes and calling a lambda function whenever this number increases. Here's how I implemented it:

class ObservableInputStream(private val wrapped: InputStream,
                            private val onBytesRead: (Long) -> Unit): InputStream() {
    private var bytesRead: Long = 0

    @Throws(IOException::class)
    override fun read(): Int {
        val res = wrapped.read()
        if (res > -1) {
            bytesRead++
        }
        onBytesRead(bytesRead)
        return res
    }

    @Throws(IOException::class)
    override fun read(b: ByteArray): Int {
        val res = wrapped.read(b)
        if (res > -1) {
            bytesRead += res
            onBytesRead(bytesRead)
        }
        return res
    }

    @Throws(IOException::class)
    override fun read(b: ByteArray, off: Int, len: Int): Int {
        val res = wrapped.read(b, off, len)
        if (res > -1) {
            bytesRead += res
            onBytesRead(bytesRead)
        }
        return res
    }

    @Throws(IOException::class)
    override fun skip(n: Long): Long {
        val res = wrapped.skip(n)
        if (res > -1) {
            bytesRead += res
            onBytesRead(bytesRead)
        }
        return res
    }

    @Throws(IOException::class)
    override fun available(): Int {
        return wrapped.available()
    }

    override fun markSupported(): Boolean {
        return wrapped.markSupported()
    }

    override fun mark(readlimit: Int) {
        wrapped.mark(readlimit)
    }

    @Throws(IOException::class)
    override fun reset() {
        wrapped.reset()
    }

    @Throws(IOException::class)
    override fun close() {
        wrapped.close()
    }
}

class ObservableOutputStream(private val wrapped: OutputStream,
                           private val onBytesWritten: (Long) -> Unit): OutputStream() {
    private var bytesWritten: Long = 0

    @Throws(IOException::class)
    override fun write(b: Int) {
        wrapped.write(b)
        bytesWritten++
        onBytesWritten(bytesWritten)
    }

    @Throws(IOException::class)
    override fun write(b: ByteArray) {
        wrapped.write(b)
        bytesWritten += b.size.toLong()
        onBytesWritten(bytesWritten)
    }

    @Throws(IOException::class)
    override fun write(b: ByteArray, off: Int, len: Int) {
        wrapped.write(b, off, len)
        bytesWritten += len.toLong()
        onBytesWritten(bytesWritten)
    }

    @Throws(IOException::class)
    override fun flush() {
        wrapped.flush()
    }

    @Throws(IOException::class)
    override fun close() {
        wrapped.close()
    }
}

EDIT 2021-01-14: A minor bug in ObservableInputStream has been fixed.

EDIT 2021-01-25: It has been brought to my attention that not all InputStream methods were originally implemented in this snippet and this could cause problems. Methods available(), mark(), markSupported(), skip(), and reset() have now been added. Also, Java 9 introduces new InputStream methods readNBytes() and readAllBytes(). If you target Java 9 or later then you should override these methods as well to update the read bytes counter appropriately, otherwise you may end up with mysterious bugs.

To use this, wrap the original stream with one of these classes, attaching a lambda function to execute every time the number of processed bytes increases. For example here's how you could report progress while downloading a file via HTTP:

fun downloadFileViaHTTP(url: String, destinationFile: file) {
    val urlObj = URL(url)
    val connection = urlObj.openConnection()

    // To calculate progress, we need to now the total size 
    // beforehand. We can get that info from HTTP headers
    val contentLength = connection.getHeaderField("Content-Length")
    val totalDownloadSize = contentLength.toLong()

    // Wrap the connection input stream with ObservableInputStream
    // in order to monitor it.
    val urlInputStream = ObservableInputStream(connection.getInputStream()) {
        val progress = it * 100 / totalDownloadSize
        // updateProgress() should post a message to the UI thread 
        // to update a progress bar or a similar widget
        updateProgress(progress.toInt())
    }

    urlInputStream.use { input ->
        destinationFile.outputStream().use { output ->
            input.copyTo(output)
        }
    }
}

Once again I use the convenient InputStream.copyTo() function to move bytes from the ObservableInputStream into the OutputStream that writes the file to internal storage. But, what if you wanted to do the opposite? There is no OutputStream.copyTo(), so how could you transfer data from OutputStream to InputStream?

Transferring data from `OutputStream` to `InputStream`

This might sound a little odd, but there are legitimate cases for doing this. Just google "Java OutputStream to InputStream" and you'll get thousands of results. Besides, it's a good use case for one of Kotlin's most recent features. Ideally you should always read from InputStream and then write into OutputStream, but imagine a scenario in which you know that some external library is about to write data to an OutputStream but you'd like to read that output, apply some transformation and then write it to a new stream.

At first glance this seems impossible because OutputStream doesn't have a read() method, only write(). Java's byte streams aren't as easy to compose as UNIX streams in which you can just use the | operator to redirect the output of one program to the input of another. What you would have to do here is write to a "mocked" OutputStream that doesn't really write data anywhere, just holds on to it until you request it. There is a Java class for this, PipedOutputStream, but there's still one little problem.

PipedOutputStream lets you create a PipedInputStream so that you can read() the data that is being written, but none of this changes the fact that write() blocks the calling thread. This means that if you want to read the data as it is being written, you need to do it concurrently. The good news is that Kotlin has coroutines now, so you can easily offload the writing to new coroutine, and read on the current one.

Finally, here is a little example to illustrate how to use PipedOutputStream. Suppose you want to download a file via FTP, but you want to decompress it before it gets written to the internal storage. The solution is to download in one coroutine and decompress in the other one.

private fun downloadStream(ftpClient: FTPClient,
                           fileName: String,
                           output: PipedOutputStream) {
    // ftpClient immediately writes the downloaded bytes
    // to the output stream
    ftpClient.retrieveFile(fileName, output)
}

private fun decompressAndWriteStream(
    sourceStream: PipedOutputStream
    destinationStream: OutputStream) {

    val srcInputStream = PipedInputStream(sourceStream)
    val decompressedInputStream =
        GZIPInputStream(srcInputStream)
    decompressedInputStream.copyTo(destinationStream)
}

suspend fun downloadFile(coroutineScope: CoroutineScope,
                         destFile: File,
                         fileName: String) {
    val ftpClient = FTPClient()
    prepareForDownload(ftpClient)
    val fileOutputStream = destFile.outputStream()

    try {
        val pipedOutputStream = PipedOutputStream()

        // launch a new coroutine to download the data
        val job = coroutineScope.launch {
            downloadStream(ftpClient, fileName, pipedOutputStream)
        }

        // decompress in the current coroutine while 
        // the other one downloads the data
        decompressAndWriteStream(
            sourceStream = pipedOutputStream,
            destinationStream = fileOutputStream)

        // Once the file is completely written, the coroutine should 
        // have finished, but just to be safe make sure it exits.
        job.join()

    } finally {
        fileOutputStream.close()
        ftpClient.disconnect()
    }
}

Coroutines make dealing with concurrency much easier. In fact, I was very surprised to see this code work correctly on my first try. Since I process multiple large files in parallel I make extensive use of Coroutines. I think this is my favorite feature and the biggest reason to switch to Kotlin for Android development.

Implementing MVI in Android

Tue, 30 Apr 2019 00:00:00 GMT

Android development should be easy. Most apps that I've worked in aren't that complex, they usually boil down to just displaying some data from a remote server and then letting the user post new data to that server. Nevertheless, their codebases often ended up being ugly and convoluted. After all these years I think I have finally figured out how to avoid common pitfalls.

I think that the main reason why messy codebases are so common is that Android developers were left on their own when it came to app architecture. Google didn't care about it until Google I/O 2017, when they announced architecture components. Even today, almost all Android guides from Google show examples that directly dump all code into activities of fragments, leading to chaos as the app grows. I understand that they do this for the sake of brevity, but since developers (myself included) tend to copy & paste snippets from these guides, it ends up being problematic.

I was very excited after the architecture components announcement. I think it's really cool that finally Google decided to give developers tools to address the elephant in the room. In particular, I believe ViewModel and LiveData could be great building blocks for a solid architecture. I'm a big fan of The Elm Architecture, among other functional languages, so I tried figure to how to use architecture components to bring some of these great ideas found in Elm and the likes into my Android apps.

Among all the popular design patterns found in the Android community, Model-View-Intent (MVI) seemed like the closest one to what I wanted. I decided to implement it with architecture components in some of my apps and after some trial and error, extracted the useful classes into this library to make it easier to implement the pattern effectively in new apps.

::github{repo="GAumala/mvi-android"}

I am very pleased with the results so in this post I want to explain why MVI is so effective, and how I use this library in my apps. For this matter I'll provide an example app, a tiny Reddit client in which you enter the name of a subreddit and load the current top posts.

You can find the entire source code in the library's repository. Feel free to checkout the code and run it in your own devices. In the following sections I'm going to break down the important parts and show how I use the library to implement the pattern.

:::warning Due to changes in the Reddit API, the example app linked here no longer works :::

The State

The most important thing in this application is to manage state and keep the UI synchronized with it. What would the state for this app look like? If I were to draw it as finite state machine diagram, it would have the following nodes:

Input state. Initially, there should be a form for the user to input the name of a subreddit to load the posts.
Loading state. After submitting a valid subreddit name, the app should show a spinner while loading the posts from the network.
Posts ready state. If posts are loaded successfully, they should be displayed with the option to click and open the links in a browser.
Error state If a network error occurs while loading posts, an error message must be displayed.

Now let's define a class that models this state. State classes should just be immutable data. You should be able to derive equals(), toString(), and even implement Parcelable with little or no effort. If you are using Java I recommend using AutoValue for this matter. If you are using Kotlin, data classes are all you need.

Here's the state for our Reddit client:

public abstract class RedditState {

    // Display a form so that the user can submit a 
    // subreddit name to load posts. 
    @AutoValue
    public static abstract class Input extends RedditState {
        public abstract @StringRes
        int errorResId();

        public static Input create(int errorResId) {
            return new AutoValue_RedditState_Input(errorResId);
        }
    }

    // Posts are loading, better show a spinner in the meantime.
    @AutoValue
    public static abstract class Loading extends RedditState {
        public abstract String subredditName();

        public static Loading create(String subredditName) {
            return new AutoValue_RedditState_Loading(subredditName);
        }
    }

    // Posts are ready to be displayed.
    @AutoValue
    public static abstract class Ready extends RedditState {
        public abstract String subredditName();

        public abstract List<Post> posts();

        public static Ready create(String subredditName, List<Post> posts) {
            return new AutoValue_RedditState_Ready(subredditName, posts);
        }
    }


    // Something went wrong loading the posts. 
    //Show an error message
    @AutoValue
    public static abstract class Error extends RedditState {
        public abstract String message();

        public static Error create(String message) {
            return new AutoValue_RedditState_Error(message);
        }
    }

    public static RedditState createInitialState() {
        return Input.create(-1);
    }
}

As you can see, every possible state is modeled as a RedditState subclass using AutoValue. It is similar to Kotlin's sealed classes. The disadvantage here is that Java lacks pattern matching for types, so it requires some unsafe casts. It's still good enough for me because it helps me avoid null references.

Side Effects

Now that the state is ready we have to define the side effects. Ideally all of our code should have pure functions. These special functions are preferred because they always return the same values if provided with the same parameters. If all of our functions were so predictable, then fixing bugs is very easy and straight forward. Just run the function again with those same parameters on any environment and locate the line where it went wrong. Unfortunately, an app like that isn't very useful. Non determinism is unavoidable and even desired in a few parts of almost every app. A few examples of this are:

Random number generation
Using a file system
Sending & Receiving data over a network

In order to be able to trigger non-deterministic computations in pure functions, these functions will return side-effect values. These values will be passed to another object, a SideEffectRunner, which will read the data and figure out how to execute the desired side-effect and post the result.

The only side effect that we want in this Reddit client, is the ability to fetch a subreddit's top posts from Reddit's servers. This is non-deterministic because the top posts change over time, and the request can fail due to a network error.

Here's the definition of the RedditSideEffect class:

public abstract class RedditSideEffect {
    @AutoValue
    public static abstract class FetchPosts extends RedditSideEffect {
        public abstract String subredditName();

        public static FetchPosts create(String subredditName) {
            return new AutoValue_RedditSideEffect_FetchPosts(subredditName);
        }
    }
}

The FetchPosts class has only one attribute: the subreddit name. This is because these classes are plain values with the minimum necessary data for the SideEffectRunner to execute it. FetchPosts is a subclass of RedditSideEffect because different side-effects need different kinds of data. In a full-featured Reddit client there should be more side-effects, like the ability to submit new posts, or upvote/downvote existing ones. Each of them would need different data, so they would be modeled with different subclasses of RedditSideEffect.

While FetchPosts contains the parameters to run the side effect. The class that actually executes it is RedditSideEffectRunner:

import com.gaumala.mvi.ActionSink;
import com.gaumala.mvi.SideEffectRunner;

public class RedditSideEffectRunner
        implements SideEffectRunner<RedditState, RedditSideEffect> {
    private final Resources resources;

    public RedditSideEffectRunner(Resources resources) {
        this.resources = resources;
    }

    @Override
    public void runSideEffect(ActionSink<RedditState, RedditSideEffect> sink,
                              RedditSideEffect sideEffect) {

        if (sideEffect instanceof RedditSideEffect.FetchPosts)
            fetchPosts(sink, (RedditSideEffect.FetchPosts) sideEffect);

    }

    private void fetchPosts(ActionSink<RedditState, RedditSideEffect> sink,
                            RedditSideEffect.FetchPosts sideEffect) {

        FetchPostsTask.run(
              resources,
              sideEffect.subredditName(),
              res -> sink.submitAction(FetchPosts.create(res)));

    }
}

This class has a public method runSideEffect(), which takes an ActionSink object and an RedditSideEffect value. It identifies the RedditSideEffect.FetchPosts value and runs the desired effect using FetchPostsTask, a class executes the HTTP request in a background thread and invokes a callback lambda once the work is complete.

The ActionSink object is used to update the state with the result of the side effect. This interface exposes a single method: submitAction(). As implied by the name it receives "actions", or intents to do something stateful, which I'll describe in the next section.

Actions

Actions are values that can update the state. They have a update() method that takes the current state, and returns a new one, because the state is an immutable value. Optionally, this method can also return a side effect to be executed immediately. This method assumed to be a pure function, it can't perform I/O or mess with global variables to calculate the new state . It can only take into account the action's data and the current state. Here's the definition of the FetchPosts action mentioned on the previous section:

import com.gaumala.mvi.Action;
import com.gaumala.mvi.Update;

@AutoValue
public abstract class FetchPosts 
  extends Action<RedditState, RedditSideEffect> {

    public abstract FetchPostsRes res();

    @NonNull
    @Override
    public Update<RedditState, RedditSideEffect> update(
        RedditState currentState) {

        if (!(currentState instanceof RedditState.Loading))
            return new Update<>(currentState);

        RedditState.Loading state = (RedditState.Loading) currentState;
        FetchPostsRes res = res();
        if (res instanceof FetchPostsRes.Success)
            return updateWithSuccess(state, (FetchPostsRes.Success) res);

        return updateWithError((FetchPostsRes.Error) res);
    }

    private Update<RedditState, RedditSideEffect> updateWithError(
            FetchPostsRes.Error res) {

        RedditState newState = RedditState.Error.create(res.message());
        return new Update<>(newState);
    }

    private Update<RedditState, RedditSideEffect> updateWithSuccess(
            RedditState.Loading state,
            FetchPostsRes.Success res) {

        RedditState newState = RedditState.Ready.create(
                state.subredditName(),
                res.posts());
        return new Update<>(newState);
    }

    public static FetchPosts create(FetchPostsRes res) {
        return new AutoValue_FetchPosts(res);
    }
}

This action takes as constructor parameter the result of FetchPostsTask: FetchPostsRes, which can either be Success or Error, and returns a new state of type Ready or Error respectively. This action handles the server's response, but what about the action that triggers the side effect that sends the request? It is this one, CallFetchPosts:

import com.gaumala.mvi.Action;
import com.gaumala.mvi.Update;

@AutoValue
public abstract class CallFetchPosts
        extends Action<RedditState, RedditSideEffect> {
    public abstract String subredditName();

    @NonNull
    @Override
    public Update<RedditState, RedditSideEffect> update(RedditState state) {

        if (!(state instanceof RedditState.Input))
            return new Update<>(state);

        if (subredditName().isEmpty()) {
            RedditState newState = RedditState.Input.create(
                    R.string.empty_string_error);
            return new Update<>(newState);
        }

        RedditState newState = RedditState.Loading.create(subredditName());
        RedditSideEffect sideEffect =
                RedditSideEffect.FetchPosts.create(subredditName());
        return new Update<>(newState, sideEffect);
    }

    public static CallFetchPosts create(String subredditName) {
        return new AutoValue_CallFetchPosts(subredditName);
    }
}

This action takes the subreddit name that the user entered, validates that it is not empty and then returns a new Loading state along with a side effect. This returned side effect is then passed to RedditSideEffectRunner so that it can execute FetchPostsTask and finally submit a FetchPosts action to ActionSink with the result.

Views

Every time an action updates the state the views should readjust themselves to let the user visualize the new state. To do this, the application's views should be managed by a "UI" object that extends BaseUI<T>. This abstract class has a rebind() method that has a single parameter: the current state. This method is called after every update() so that the views always stay in sync with the state.

Before showing the implementation of rebind() for the Reddit client, let's talk about the layout and what views are used. Here's the XML:

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    xmlns:app="http://schemas.android.com/apk/res-auto">

    <androidx.appcompat.widget.Toolbar
        android:id="@+id/toolbar"
        android:layout_width="match_parent"
        android:layout_height="?attr/actionBarSize"
        android:elevation="4dp"
        app:title="@string/reddit"
        app:theme="@style/ThemeOverlay.AppCompat.Dark.ActionBar" />

    <include layout="@layout/subreddit_form_view"
        android:layout_marginTop="?attr/actionBarSize"
        android:layout_width="match_parent"
        android:layout_height="match_parent" />

    <androidx.recyclerview.widget.RecyclerView
        app:layoutManager="androidx.recyclerview.widget.LinearLayoutManager"
        android:id="@+id/posts_recycler"
        android:layout_marginTop="?attr/actionBarSize"
        android:visibility="gone"
        android:layout_width="match_parent"
        android:layout_height="match_parent"/>

</FrameLayout>

This layout has 3 main elements:

Toolbar
RecyclerView
Input form (Nested layout with the form for the subreddit name).

The idea is to switch between the form and the RecyclerView depending on the state. If the state is instance of Input, the form should be visible while the RecyclerView is hidden. For any other states, the RecyclerView is shown instead because progress bars, error messages and posts can be shown inside it. With that being said, here's a snippet of RedditGUI containing the rebind() implementation:

import com.gaumala.mvi.ActionSink;
import com.gaumala.mvi.BaseUI;

class RedditGUI extends BaseUI<RedditState> {

    private final Context ctx;
    private final ActionSink<RedditState, RedditSideEffect> sink;
    private final ActionBar actionBar;

    private final View inputForm;
    private final TextInputLayout subredditInputLayout;
    private final View submitButton;
    private final GroupAdapter postsAdapter;
    private final RecyclerView postsRecycler;
    private final Toolbar toolbar;

    RedditGUI(@NonNull LifecycleOwner owner,
              @NonNull LiveData<RedditState> liveState,
              ActionSink<RedditState, RedditSideEffect> sink,
              View view,
              ActionBar actionBar) {
        super(owner, liveState);

        this.ctx = view.getContext();
        this.sink = sink;
        this.actionBar = actionBar;
        this.postsAdapter = new GroupAdapter();

        inputForm = view.findViewById(R.id.subreddit_input_form);
        subredditInputLayout = view.findViewById(R.id.subreddit_input_layout);
        submitButton = view.findViewById(R.id.submit_button);
        postsRecycler = view.findViewById(R.id.posts_recycler);
        toolbar = view.findViewById(R.id.toolbar);

        // set the adapter and dividers to RecyclerView
        setupRecyclers();
    }

    @Override
    public void rebind(RedditState state) {
        if (state instanceof RedditState.Input)
            showInputForm((RedditState.Input) state);
        else
            showPostsRecyler(state);

        // show the subreddit name in the title, or just "Reddit" 
        // if the user hasn't picked a subreddit yet
        toolbar.setTitle(getTitle(state));
    }

This class holds references to all the views and shows only the appropriate views for the current state every time rebind() is called. The rebind() method makes no assumptions about the state transitions, it simply reacts to the current state calling the necessary methods on every view. You may be inclined to believe that resetting properties on every view after every update might be inefficient, but it really isn't because android views are smart enough to avoid redrawing things that haven't actually changed.

You may notice that it calls a super constructor with two parameters of type LifecycleOwner and LiveData<RedditState> respectively. These two objects are used under the hood to subscribe to changes in the state and call rebind() after every update.

Adjusting views isn't RedditGUI's only responsibility. It also handles UI events like button clicking and window scrolling. Just like with RedditSideEffectRunner, RedditGUI also receives an ActionSink to submit actions and trigger state changes in response to UI events. For example, here's how the showInputForm() method used in rebind() sets a click listener to the submit button:

private void showInputForm(RedditState.Input state) {

    // ...adjust some views

    // handle click event
    submitButton.setOnClickListener(v -> {
        String inputText = subredditInputLayout
                .getEditText().getText().toString();
        sink.submitAction(CallFetchPosts.create(inputText));
    });
}

When submitButton is clicked, a CallFetchPosts action is submitted with the text input by the user.

The Dispatcher

MVI establishes an unidirectional cycle between three parts: intent -> model -> view. In this app, these parts are represented as follows:

Intent: Listening to UI events or side effect results is handled by the ActionSink. Both RedditGUI and RedditSideEffectRunner call this object when they have to deliver actions.
Model: Processing data from intents to generate new states is handled by Action classes and their update() method.
View: Readjusting the Views in order to reflect the latest state returned by update() is handled by RedditGUI and its rebind() method. As this class also generates intents, it is evident how things come full circle.

Some people view this pattern as the following function composition: view(model(intent())). That composition is still present in this app, but it uses class methods instead of plain functions because we are still stuck in Java's OOP world. It's roughly something like this: rebind(update(submitAction())).

To glue everything together some sort of "observer" is needed. Here the Dispatcher class implements this functionality. As you may have already guessed it dispatches actions, triggering state changes and side effects. It implements ActionSink, the interface that RedditGUI and RedditSideEffectRunner use submit actions. It takes a SideEffectRunner object in its constructor so that it can execute the side effects returned by the received actions. Additionally, it holds the LiveData object with the current state so it manages the state and lets observers like RedditGUI react to state changes.

Unlike BaseUI or SideEffectRunner, you don't have to extend this class, you simply create an instance with the appropriate type parameters.

The dispatcher is kept inside a ViewModel so that the application state can persist configuration changes like screen rotation. There is a DispatcherViewModel class that does exactly that. It is parametrized just like Dispatcher, but since you don't instantiate view models directly in android, it's better to extend DispatcherViewModel with the desired type parameters. For example, here's the view model used for the Reddit client:

import com.gaumala.mvi.DispatcherViewModel;
import com.gaumala.mvi.Dispatcher;

public class RedditViewModel 
    extends DispatcherViewModel<RedditState, RedditSideEffect> {

    RedditViewModel(Dispatcher<RedditState, RedditSideEffect> dispatcher) {
        super(dispatcher);
    }
}

This class doesn't do anything special. It is merely a convenience needed due to the parametrization of the DispatcherViewModel type. Since it extends DispatcherViewModel, it gets the public methods: getLiveState(), and getActionSink() which return objects of type LiveData<RedditState> and ActionSink<RedditState, RedditSideEffect> respectively. These two objects are needed to connect all the classes implemented so far because they are able to read the current state and update it via actions.

Running the app

Now that there's a view model that persists the dispatcher and application state it's time to wire up everything and run the app. The first thing to do is to instantiate the view model along with all the dependencies (the initial state, dispatcher and side effects runner). I like do this in implementations of ViewModelProvider.Factory.

import com.gaumala.mvi.Dispatcher;

public class ViewModelFactory implements ViewModelProvider.Factory {
    private final Fragment fragment;

    public ViewModelFactory(Fragment fragment) {
        this.fragment = fragment;
    }

    @NonNull
    @Override
    public <T extends ViewModel> T create(@NonNull Class<T> modelClass) {
        // The fragment's arguments could be used here to create 
        // the initial state
        RedditState initialState = RedditState.createInitialState();

        RedditSideEffectRunner runner =
                new RedditSideEffectRunner(fragment.getResources());

        // optionally, startup side effects could be run here

        Dispatcher<RedditState, RedditSideEffect> dispatcher =
                new Dispatcher<>(runner, initialState);

        return (T) new RedditViewModel(dispatcher);
    }
}

The create() method is guaranteed by the architecture components library to run only once, when the view model has not been yet created. It is not necessary for ViewModelFactory to keep a reference to the fragment, but I do it because it is very often useful. In this case it gives me access to a Resources reference that I use in in RedditSideEffectRunner to create appropriate error messages. Also, The initial state could created with a function that returns a different value depending on the fragment's argument bundle.

This view model is instantiated by RedditFragment, the fragment that renders the views for our Reddit client. Here's how it starts the app:

public class RedditFragment extends Fragment {
    @Nullable
    @Override
    public View onCreateView(@NonNull LayoutInflater inflater,
                             @Nullable ViewGroup container,
                             @Nullable Bundle savedInstanceState) {
        setHasOptionsMenu(true);
        RedditViewModel viewModel = ViewModelProviders
                .of(this, new ViewModelFactory(this))
                .get(RedditViewModel.class);

        View view = inflater.inflate(
                R.layout.reddit_fragment, container, false);
        ActionBar actionBar = setupActionBar(view.findViewById(R.id.toolbar));

        gui = new RedditGUI(
                this.getViewLifecycleOwner(),
                viewModel.getLiveState(),
                viewModel.getUserActionSink(),
                view, actionBar);
        gui.subscribe();

        return view;
    }
}

RedditFragment only implements one lifecycle method: onCreateView() which does the following two things before returning a view:

Creates the view model if it doesn't already exist.
Creates a RedditGUI fetching a few dependencies from the view model so that it can observe state changes and submit actions.
Calls the RedditGUI.subscribe() method to start observing state changes.

That's it! This is all that's necessary to get the app running. Notice that there is no need to implement onStop() or any other lifecycle method because LiveData object used by RedditGUI automatically unsubscribes from state changes when the fragment stops. Network requests could still be loading while the fragment is stopped or recreated and there won't be any memory leaks because the side effects runner never keeps a reference to the activity or fragment. More complex apps may need to implement other lifecycle methods, and that's ok, but most of the time it is not necessary thanks to LiveData.

Testing

I think testing is very important, and a good architecture should make it easy to write tests. Since state and actions are defined as immutable values it is very easy to come up with reproducible test cases. You can quickly verify if a chain of actions ends up with a particular state or if it returns any desired side effects.

One of the reasons why ActionSink is an interface instead of a concrete class is that this makes it possible to use a different implementation in in testing environments that doesn't rely on the android framework and exposes additional methods for making assertions about the generated states and side effects.

Here's a test that asserts that the app reaches the RedditState.Ready state when the user inputs a valid subreddit and the server request succeeds:

@Test
public void should_display_posts_in_absence_of_errors() {
    RedditState initialState = RedditState.createInitialState();
    TestSink<RedditState, RedditSideEffect> sink =
            new TestSink<>(initialState);

    sink.submitAction(CallFetchPosts.create("news"));
    sink.submitAction(FetchPosts.create(
            ResponseMocks.fetchPosts_Success));

    RedditState currentState = sink.getCurrentState();
    RedditState expectedState = RedditState.Ready.create(
        "news",
        ResponseMocks.fetchPosts_Success.posts());
    assertThat(currentState, is(equalTo(expectedState)));
}

This test uses a custom ActionSink implementation, TestSink that exposes a getCurrentState() method that lets you peek into the current state so that you can make assertion about its value. It also exposes a getGeneratedSideEffects() to peek into all the generated side effects so far. Here's another test that asserts that a FetchPosts side effect is generated by these same two actions:

@Test
public void should_generate_FetchPosts_side_effect_in_absence_of_errors() {
    RedditState initialState = RedditState.createInitialState();
    TestSink<RedditState, RedditSideEffect> sink =
            new TestSink<>(initialState);

    sink.submitAction(CallFetchPosts.create("news"));
    sink.submitAction(FetchPosts.create(
            ResponseMocks.fetchPosts_Success));


    List<RedditSideEffect> actualSideEffects = sink.getGeneratedSideEffects();
    List<RedditSideEffect> expectedSideEffects = Collections.singletonList(
            RedditSideEffect.FetchPosts.create("news"));
    assertThat(actualSideEffects, is(equalTo(expectedSideEffects)));
}

"Real" apps

This tiny Reddit client only has one fragment because it only does one thing: fetch posts from a particular subreddit. This isn't the kind of app I'm targeting but it's good enough as an example. A more realistic Reddit client would let you do a lot more things like:

Sign in to your Reddit account
Fetch posts from your frontpage
Manage your subscriptions
Check your DMs

The list goes on, but I'm sure you get the idea. "Real" apps have lots of features. With this design pattern, each of these features would be implemented in its own fragment, defining its own state and side effects, isolated from the other features. It's like having your app made of dozens of little apps that are easy to manage.

The app I'm currently working on in my day job is made of 24 "little apps" occupying 3 MB just in Java source files. It also has an additional 3 MB of legacy java code that I can't really touch, let alone convert to this pattern. It's been less than 6 months of development and it is very likely that we'll add many more features before the year ends. No matter how much it grows I'm confident that it will remain as maintainable as always.