Uri Sh.

Enhancing the VPN Profile Switcher: Introducing Local DB Creation and a Self-Test Tool

2024-08-18T00:00:00+00:00

In my last post, I mentioned upcoming enhancements to the VPN Profile Switcher, and today, I’m excited to share two new scripts that take this project to the next level. These scripts not only add functionality but also pave the way for a more robust and self-sufficient toolset.

1. Creating a Local Database with `get_groups_and_countries_to_tsv.sh`

One of the critical components of the VPN Profile Switcher is the ability to maintain an updated list of available VPN server types and countries. Previously, this information was stored in the db branch of the repository and updated through a GitHub action. However, this approach comes with a caveat: the GitHub action will be canceled after two months of inactivity in the repository, potentially leaving the database outdated. To address this, I’ve developed the get_groups_and_countries_to_tsv.sh script. This script enables users to generate a local database of VPN server types (like TOR and P2P) and the countries where these services are available. By running this script, you can ensure that your local database is always up-to-date, independent of the GitHub action.

The script relies on wget and jsonfilter to mimic the functionality of the GitHub action. The action itself, housed at curl-then-jq-shell-action, utilizes curl and jq for its operations. With this script, you no longer need to rely on the remote repository’s database, making the db branch redundant and streamlining your setup.

2. Building a Self-Test Tool with `vpn-status-checker-openwrt.sh`

Another essential aspect of managing VPN connections is ensuring that your VPN is functioning correctly and securely. To this end, I’ve started building a self-test tool with the vpn-status-checker-openwrt.sh script. This script is designed to perform basic checks such as DNS leak tests and connection status verification, similar to what you might find on the NordVPN What Is My IP page, but in a command-line interface (CLI) format.

While this script is in its early stages, it lays the groundwork for a more comprehensive self-test tool that can be integrated into the VPN Profile Switcher. It’s a straightforward yet powerful way to ensure your VPN is operating as expected, providing peace of mind in an increasingly complex digital landscape.

Looking Ahead: WireGuard Testing and Beyond

As I continue to enhance the VPN Profile Switcher, one area I plan to explore further is testing the connectivity and robustness of WireGuard. While I haven’t had the chance to dive into this yet, it’s on my radar, and I’m eager to see how WireGuard performs in various scenarios.

In the meantime, these two scripts represent significant steps forward in making the VPN Profile Switcher more self-reliant and user-friendly. Whether you’re looking to maintain an up-to-date local database or ensure your VPN connection is secure, these tools are here to help.

Stay tuned for more updates, including additional features and test results, as I continue to refine and expand the capabilities of the VPN Profile Switcher!

Updating the `vpn-profile-switcher.sh` to Include WireGuard Support

2024-07-30T00:00:00+00:00

In a recent update to the vpn-profile-switcher.sh script, I have added support for WireGuard, expanding its functionality beyond the initial OpenVPN compatibility. This update allows users to choose between OpenVPN and WireGuard for their VPN configurations on OpenWRT. Below, I will walk you through the changes and explain the new options and functionalities added to the script.

Overview of the Script

The vpn-profile-switcher.sh script automates the process of retrieving the recommended NordVPN server, downloading its configuration file, setting the credentials, and configuring OpenWRT to use this server. Initially designed for OpenVPN, the script now includes support for WireGuard, a faster and more efficient VPN protocol.

Key Updates for WireGuard Support

New Options in the Usage Function

To accommodate WireGuard, new command-line options have been added:

printf "  -t | --type < openvpn | wireguard >,\t\tSelect either OpenVPN or WireGuard. Default is OpenVPN.\n"
printf "  -i | --interface < interface >,\t\tSpecify interface to use for WireGuard. Default is 'nordlynx'. Applicable only for WireGuard.\n"

The --type option allows the user to select the VPN type, either openvpn or wireguard. The --interface option specifies the interface to use for WireGuard, with nordlynx as the default.

Verification Functions

Two new functions ensure that valid values are provided for the --type and --interface options:

function verify_vpn_type() {
    if [ ! "$1" == "openvpn" ] && [ ! "$1" == "wireguard" ]; then
        logger -s "($0) VPN type must be either openvpn or wireguard, your input was: $1."
        exit 1
    else
        echo $1
    fi
}

function verify_protocol() {
    if [ ! "$1" == "tcp" ] && [ ! "$1" == "udp" ]; then
        logger -s "($0) Protocol must be either udp or tcp, your input was: $1."
        exit 1
    else
        echo $1
    fi
}

Fetching Recommendations

The get_recommended function fetches the recommended server configuration. If WireGuard is selected, it also retrieves the server’s public key:

function get_recommended() {
    _url="https://api.nordvpn.com/v1/servers/recommendations?"
    if [ ! -z "$group_identifier" ]; then
        _url=${_url}"filters[servers_groups][identifier]="${group_identifier}"&"
    fi
    if [ ! -z "$country_id" ]; then
        _url=${_url}"filters[country_id]="${country_id}"&"
    fi
    _url=${_url}"filters[servers_technologies][identifier]="${vpn_type}"_"${protocol}"&limit="${recommendations_n}
    logger -s "($0) Fetching VPN recommendations from: $_url"
    _json=$(wget -q -O - "$_url") || true
    recommended=$(jsonfilter -s "$_json" -e '$[0].hostname') || true
    recommendations=$(jsonfilter -s "$_json" -e '$[*].hostname') || true
    loads=$(jsonfilter -s "$_json" -e '$[*].load') || true
    if [ "$vpn_type" == "wireguard" ]; then
        public_key=$(jsonfilter -s "$_json" -e '$[0].technologies[*].metadata[*].value') || true
    fi
}

Configuration Handling

The script includes functions for checking existing configurations and enabling or disabling them as needed:

function check_in_configs() {
    if [ "$vpn_type" == "openvpn" ]; then
        server_name=$(uci show openvpn | grep $recommended.$protocol | awk -F '\.' '/config/{print $2}')
    elif [ "$vpn_type" == "wireguard" ]; then
        server_name=$(uci show network | grep "${wg_iface}.*endpoint_host.*${recommended}" | sed -E 's/.*_([a-z]{0,2}[0-9]{1,3}).*='\''(.*)'\''$/\1/')
    fi
}

function check_enabled() {
    if [ "$vpn_type" == "openvpn" ]; then
        enabled_server=$(uci show openvpn | grep "enabled='1'" | awk -F '\.' '/.*/{print $2}')
    elif [ "$vpn_type" == "wireguard" ]; then
        enabled_server=$(uci show network | grep "${wg_iface}.*disabled='0'" | sed -E 's/.*_([a-z]{0,2}[0-9]{1,3}).*='\''.*'\''$/\1/')
    fi
}

Creating and Enabling Entries

For WireGuard, the script creates a new entry and configures it with the appropriate parameters:

function create_new_entry() {
    if [ "$vpn_type" == "openvpn" ] then
        new_server=$(echo "$recommended" | sed 's/[\.\ -]/_/g' | sed "s/com/$protocol/g")
        uci set openvpn.$new_server=openvpn
        uci set openvpn.$new_server.config="/etc/openvpn/$recommended.$protocol.ovpn"
    elif [ "$vpn_type" == "wireguard" ] then
        new_server=$(echo "$recommended" | sed -E 's/([a-z]{2}[0-9]{1,3}).*''$/\1/')
        uci set network.${wg_iface}_peer_${new_server}="wireguard_$wg_iface"
        uci set network.${wg_iface}_peer_${new_server}.public_key="$public_key"
        uci set network.${wg_iface}_peer_${new_server}.endpoint_host="$recommended"
        uci set network.${wg_iface}_peer_${new_server}.endpoint_port="51820"
        uci set network.${wg_iface}_peer_${new_server}.persistent_keepalive="25"
        uci set network.${wg_iface}_peer_${new_server}.route_allowed_ips="1"
        uci add_list network.${wg_iface}_peer_${new_server}.allowed_ips="0.0.0.0/0"
        uci add_list network.${wg_iface}_peer_${new_server}.allowed_ips="::/0"
        uci set network.${wg_iface}_peer_${new_server}.description="$recommended"
    fi
}

Restarting Services

Depending on the VPN type selected, the script restarts the appropriate service:

function restart_openvpn() {
    uci commit openvpn
    /etc/init.d/openvpn restart
}

function restart_wireguard() {
    uci commit network
    /etc/init.d/network restart
}

Execution

Finally, the script’s main execution block processes the provided command-line arguments and invokes the necessary functions to configure the VPN:

check_required

while [ ! -z "$1" ]; do
    case "$1" in
    -h | --help)
        show_usage
        ;;
    -t | --type)
        shift
        vpn_type=$(verify_vpn_type $1)
        ;;
    -p | --protocol)
        shift
        protocol=$(verify_protocol $1)
        ;;
    -c | --country)
        shift
        country_id=$(country_code $1)
        ;;
    -g | --group)
        shift
        group_identifier=$(server_groups $1)
        ;;
    -r | --recommendations)
        shift
        recommendations_n=$(check_is_num $1)
        ;;
    -d | --distance)
        shift
        load_distance=$(check_is_num $1)
        ;;
    -l | --login-info)
        shift
        secret="$1"
        ;;
    -i | --interface)
        shift
        wg_iface="$1"
        ;;
    -b | --db-location)
        shift
        db_location=$(verify_db_location $1)
        ;;
    *)
        show_usage
        ;;
    esac
    shift
done

# Fetch the recommended server
get_recommended

# Check if the recommended server is already configured
check_in_configs

# Disable the current server configuration if it exists
if [ ! -z "$server_name" ]; then
    disable_current_entry $server_name
fi

# Create a new entry for the recommended server if it doesn't already exist
if [ -z "$server_name" ]; then
    create_new_entry
else
    enable_existing_entry $server_name
fi

# Restart the appropriate VPN service
if [ "$vpn_type" == "openvpn" ]; then
    restart_openvpn
elif [ "$vpn_type" == "wireguard" ]; then
    restart_wireguard
fi

# Clean up and exit
unset_variables

Conclusion

The vpn-profile-switcher.sh script now offers robust support for both OpenVPN and WireGuard, providing flexibility and efficiency for users. This update simplifies the process of configuring VPN profiles on OpenWRT, ensuring a seamless experience regardless of the chosen VPN protocol. For more details, refer to the dev branch on the repository on GitHub.

In the next post, I will cover adding some nice to have features as separate scripts, and perhaps add some test results. Stay tuned for more updates and enhancements to the VPN Profile Switcher script.

Upgrading the VPN Profile Switcher Script for OpenWRT

2024-07-28T00:00:00+00:00

In this post, I will walk you through the process of updating and upgrading a shell script I originally wrote a few years ago for changing the VPN profile on OpenWRT routers. This script connects to NordVPN’s recommended servers, and I’ve recently added support for WireGuard (NordLynx) alongside the existing OpenVPN functionality.

The original script was intended for OpenWRT installations on TP-Link Archer C20i and Raspberry Pi 3B+ with 2019 firmware. You can find the VPN Profile Switcher script here.

The script queries NordVPN’s API for the recommended server and then updates the OpenVPN or WireGuard configuration on the router accordingly.

Here, I’ll discuss the upgrade process, including adding support for WireGuard, and share the essential parts of the script without exposing application-specific details.

Understanding the Script’s Purpose

The primary goal of the script is to:

Fetch the recommended NordVPN server using their API.
Check if the recommended profile exists on the router.
If the profile does not exist, download it and configure it with the saved username and password.
Update the router’s VPN configuration to use the new profile.
Clean up old profiles, keeping only the currently connected and previously connected profiles.

Research and Preparation

Since NordVPN started offering WireGuard servers (NordLynx) but didn’t provide a tutorial for manual connection on OpenWRT, I had to spend some time researching how to set it up. I found an unofficial documentation for NordVPN’s API long time ago, and used a comment on a GitHub gist as a starting point for extracting the necessary details.

Setting Up WireGuard

First, I set up WireGuard on my development machine (a Mac) and then ported it to OpenWRT. Since I was unfamiliar with WireGuard, this involved a lot of trial and error. I used Claude (Anthropic’s AI) to help convert cURL commands to the lighter wget command suitable for OpenWRT. Despite Claude’s help, I had to install the wget-ssl package to avoid errors from NordVPN’s servers.

With the necessary data extracted using jsonfilter, I created a WireGuard interface using LuCI, OpenWRT’s web interface. This initial success allowed me to formalize my approach in a script.

Script Details

Below is a simplified description of the script’s structure and logic. The actual script is more detailed and includes error handling and logging.

Check for Required Packages:

 function check_packages() {
     opkg list-installed wget-ssl | grep -q . || opkg list-installed curl | grep -q .
     if [ $? -ne 0 ]; then
         logger -s "($0) Error: either wget-ssl or curl are required to run this script. Please install either one of the packages."
         exit 1
     fi
 }

Fetch Credentials:

 function get_credentials() {
     if which curl >/dev/null 2>&1; then
         response_json=$(curl -s -u token:"$access_token" https://api.nordvpn.com/v1/users/services/credentials)
     else
         response_json=$(wget -q -O - --auth-no-challenge --user=token --password="$access_token" https://api.nordvpn.com/v1/users/services/credentials)
     fi
     if [ $? -ne 0 ]; then
         logger -s "$(0) Error: Failed to fetch credentials. Please check your access token and internet connection."
         exit 1
     fi
 }

Save Credentials:

 function save_credentials() {
     if [ "$vpn_type" == "openvpn" ]; then
         _username=$(jsonfilter -s "$response_json" -e '@.username')
         _password=$(jsonfilter -s "$response_json" -e '@.password')
         echo "$_username" > /etc/openvpn/secret
         echo "$_password" >> /etc/openvpn/secret
     else
         _key=$(jsonfilter -s "$response_json" -e "@.nordlynx_private_key")
         uci set network.$wg_iface=interface
         uci set network.$wg_iface.proto='wireguard'
         uci set network.$wg_iface.private_key=$_key
         uci set network.$wg_iface.listen_port='51820'
         uci set network.$wg_iface.addresses='10.5.0.2/32'
         uci commit network
     fi
 }

Script Execution:

 check_packages
 get_credentials
 save_credentials

For more details, you can review the full script in the GitHub repository.

In the next post, I’ll cover the final integration of the updated script into the VPN Profile Switcher, ensuring seamless switching between OpenVPN and WireGuard profiles.

Integrating Machines Using Android and Python - Part 3

2024-07-24T00:00:00+00:00

Overcoming Dynamic Screen Challenges with ADB and Appium

When working with consumer-level machines integrated into larger systems, I faced a significant challenge: dynamic screens. These screens, which change frequently based on user interactions, made it impossible to control the machine using just ADB. This is where Appium came in handy.

Setting Up Appium for Development

Initially, I set up Appium Desktop to explore the possibilities. This graphical interface allowed me to interact with the device and see the UI hierarchy in real-time, which was crucial for understanding how the app’s screens were structured. After gaining enough insight, I transitioned to using the Appium CLI version through NPM for more streamlined and automated control.

Crafting the ADB Hierarchy Handler

With the knowledge gained, I wrote a custom class to handle interactions with the device. This class, ADB_Hierarchy_Handler, encapsulates the logic for connecting to the device, parsing the UI hierarchy, and performing actions based on dynamic screen content. While I can’t share the full implementation, here’s an overview of its functionality:

Initialization and Connection: The class starts the ADB server, connects to the device, and initializes Appium. If no device is found, it raises a custom error and exits.
UI Hierarchy Parsing: Using xmltodict and regular expressions, the class parses the XML representation of the UI hierarchy to locate elements dynamically.
Action Execution: Based on the parsed data, it performs actions such as clicking buttons or entering text.

Here’s a description of the critical sections of the class:

import time
import os
import subprocess
import re
from ppadb.client import Client
from appium import webdriver
from selenium.common import exceptions as SE
import xmltodict

class CustomError(Exception):
    pass

class ADB_Hierarchy_Handler:
    print(os.environ['ANDROID_HOME'])
    os.system('adb start-server')
    adb = Client()
    devices = adb.devices()

    if len(devices) == 0:
        print('no device attached')
        raise CustomError("no device attached to ADB server")

    appium_server = subprocess.Popen('appium', shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

    caps = {
        "platformName": "Android",
        "automationName": "UiAutomator2",
        "skipServerInstallation": True,
        "disableWindowAnimation": True
    }

    driver = webdriver.Remote("http://localhost:4723/wd/hub", caps)

    def parse_ui_hierarchy(self):
        xml_data = self.driver.page_source
        data_dict = xmltodict.parse(xml_data)
        return data_dict

    def find_element(self, search_dict, field):
        elements = get_recursively(search_dict, field)
        return elements

    def click_element(self, element):
        bounds = get_bounds_as_dict(element['@bounds'])
        center = get_center(bounds)
        self.driver.tap([(center['x'], center['y'])])

Utilizing Helper Functions for Parsing

I leveraged several helper functions to streamline the process of parsing the UI hierarchy and extracting necessary information:

import re

bounds_regex = re.compile("\\[(\\d+),(\\d+)\\]\\[(\\d+),(\\d+)\\]")

def get_recursively(search_dict, field):
    fields_found = []
    for key, value in search_dict.items():
        if key == field:
            fields_found.append(value)
        elif isinstance(value, dict):
            results = get_recursively(value, field)
            fields_found.extend(results)
        elif isinstance(value, (list, tuple)):
            for item in value:
                if isinstance(item, dict):
                    more_results = get_recursively(item, field)
                    fields_found.extend(more_results)
    return fields_found

def get_bounds_as_dict(bounds_str):
    bounds = {}
    matches = bounds_regex.search(bounds_str)
    bounds['left'] = int(matches.group(1))
    bounds['top'] = int(matches.group(2))
    bounds['right'] = int(matches.group(3))
    bounds['bottom'] = int(matches.group(4))
    return bounds

def get_center(bounds_dict):
    coordinates = {}
    coordinates['x'] = (bounds_dict['right'] - bounds_dict['left']) // 2 + bounds_dict['left']
    coordinates['y'] = (bounds_dict['bottom'] - bounds_dict['top']) // 2 + bounds_dict['top']
    return coordinates

Wrapping Up and Testing

The class includes a main test case to ensure everything works as expected:

if __name__ == "__main__":
    try:
        handler = ADB_Hierarchy_Handler()
        ui_data = handler.parse_ui_hierarchy()
        elements = handler.find_element(ui_data, 'desired_element_id')
        if elements:
            handler.click_element(elements[0])
    except CustomError as e:
        print(f"An error occurred: {e}")

This code initializes the handler, parses the UI hierarchy, finds the desired element by its ID, and clicks it. This approach enabled me to automate interactions with the device efficiently, overcoming the challenges posed by dynamic screens without relying on OpenCV.

Conclusion

Through the use of Appium and careful parsing of the UI hierarchy, I successfully automated the control of machines with dynamic screens. This solution proved robust and adaptable, paving the way for more efficient integration of consumer-level machines into larger systems.

I also believe that using Appium and a virtual device runnning on the PC controlling the robot could have been a better solution for the first machine, instead of the unused cell phone we used to get that working. I should have tried to get it going, but I didn’t have the time to revive the old machine and tinker with it. I’ll definitely consider it for future projects.

Openwrt On Raspberry Pi For Shell Script Development

2024-07-11T00:00:00+00:00

Setting Up an OpenWRT Dev Environment on a Raspberry Pi 3 B+

It’s been a while since I last tinkered with something fun, but I finally got around to setting up an OpenWRT development environment on my Raspberry Pi 3 B+. This time, it’s for version 2 of my VPN profile switcher script, which you can find on my GitHub repo.

Note: this post is about setting up a shell script dev environment on OpenWRT, not an actual OpenWRT development environment. For that, please follow the official developer’s guide

Flashing OpenWRT

To start, I obtained the most recent OpenWRT image for the Raspberry Pi and flashed it onto an SD card. The process was simple - download the image, use Balena Etcher, and you’re ready to proceed. Afterward, I powered on the Pi, set a password, and added my SSH keys from my main machine. It’s important to prioritize security and accessibility, isn’t it?

To make sure I had enough space for everything, I followed OpenWRT’s guide to expand the filesystem for ext4. This step is crucial, especially if you plan on installing packages or running scripts.

For some reason, I had to also install e2fsck as well, so my process is as follows:

opkg update
opkg install parted tune2fs resize2fs e2fsck

parted
p
resizepart 2 8GB # or whatever size you want
q

mount -o remount,ro /                  #Remount root as Read Only
tune2fs -O^resize_inode /dev/mmcblk0p2    #Remove reserved GDT blocks
# This failed, so I had to run e2fsck
e2fsck -f /dev/mmcblk0p2                #Answer yes to all questions
tune2fs -O^resize_inode /dev/mmcblk0p2    #Remove reserved GDT blocks
fsck.ext4 /dev/mmcblk0p2                  #Fix part, answer yes to remove GDT blocks remnants
reboot

# After reboot
resize2fs /dev/mmcblk0p2

WiFi Hotspot and WAN Setup

Next, I turned the Pi into a WiFi hotspot and set the ethernet port as WAN. This means the Pi can share its internet connection wirelessly while staying wired to my main router.

I actually tried to set up the wireless interface on the pi as a STA (client) + AP (master), but it didn’t work out. I ended up using the USB WiFi adapter as a WWAN instead. You can find some threads on the topic here, and some more info on the travelmate support thread. Apperantly it is possible w/ Raspoberry pi OS, here’s a how-to.

Adding a WiFi WWAN

Now, for a bit of fun – I had a spare TL-822N USB WiFi adapter and decided to add a WiFi WWAN. I followed what worked for someone on the OpenWRT forum, but ran into some issues with the adapter repeatedly failing to connect. After some head-scratching, I figured it might be a power issue. Swapped out the power supply for a beefier 3W one - and the problem was solved.

So, assuming you have a beefy enough power supply, install the following packages:

opkg update
opkg install kmod-usb2 rtl8192eu-firmware kmod-rtl8xxxu

Why the WWAN?

You might wonder why go through the hassle of adding a WWAN. Well, I wanted to keep the ethernet port on my main router free and have a ready-to-use travel router for future adventures.

Getting OpenVPN Running

With the networking sorted, it was time to get OpenVPN up and running with NordVPN. I followed my own instructions from the repo’s README – always nice when past me makes things easier for future me.

SSH Convenience

To make my life easier, I ensured my SSH connection was rock-solid and installed nano on the Pi. On my main machine, I set up the SSH FS extension in VS Code and installed the openssh-sftp-server package on the Pi, following OpenWRT’s documentation. This setup lets me edit files on the Pi directly from VS Code, which provides some useful quality of life features, such as switching between files easily, syntax highlighting, and Copilot.

Final Touches

Lastly, I installed git and git-http packages on the Pi so I could clone and manage repos directly. With everything in place, my OpenWRT dev environment was ready to roll.

Edit: 2024-07-24

When trying to work on the script on the Pi, I had a bit of trouble pushing the changes to the repo. I decided to set up a new SSH key on the Pi and add it to my GitHub account. This way, I could push changes without any issues. Sicne the default SSH agentt on tthe standard OpenWRT is dropbear, I followed this how-to on the OpenWRT wiki. Here’s how I did it:

mkdir ~/.ssh
dropbearkey -t ed25519 -f ~/.ssh/id_dropbear
dropbearkey -y -f ~/.ssh/id_dropbear

I then copied the public key to my GitHub account and added the private key to my SSH agent. This way, I could push changes to the repo without any issues.

I also set the username and email for the git config:

git config --global user.name "Your Name"
git config --global user.email "[email protected]"

I had trouble working with source control from my remote sshfs connection, so I decided to clone the repo directly to the Pi. This way, I could work on the script without any issues. That’s a bit of a bummer, but it’s a small price to pay for a stable development environment.

Wrapping Up

So there you have it – a solid OpenWRT development environment on a Raspberry Pi 3 B+. Perfect for freeing up your main router’s ethernet connection or having a travel-ready router. Happy tinkering!

Cheers, Uri

Integrating Machines Using Android and Python - Part 2

2022-01-21T00:00:00+00:00

For the next iteration of our machine, a different machine was selected because the first one wasn’t quite up to the task. Besides functioning poorly (for our purposes, at least), the Bluetooth connection wasn’t reliable enough.

Asking around, one of the entrepreneurs’ CTO heard of a (supposedly) better machine that had been “hacked” by one of the distributor’s clients. A few days passed, and we got to play with a new (secondhand) machine, already half disassembled by one of the mechanical engineers already modeling its parts.

First thoughts, again

Since the machine’s electronics were mostly out, I first tried figuring out what’s what. I found a microcontroller board at the back of the machine, and found that it was driven by a microcontroller I sort of know - STM32F103. I tried figuring out what each IO was doing, and doing so I found the TX/RX lines going to the touch screen at the front of the machine. I then followed SparkFun’s tutorial on using a logic analyzer with Sigrok, to capture the UART comm.

At this point my goal was to record the communication sent back and forth between the uC and the touch screen, and to figure out how to trigger certain functions.

It turned out to be a good learning experience, but not a worthwhile one. The communication protocol used in this particular machine was too complicated for me to decipher, as it was using what seemed to be changing in length with every operation. Without some prior knowledge about the communication scheme, it seemed like a lot of work was still ahead of me if I were to continue down this road.

As the communication seemed too complex to replicate, I took another look at the machine. I found that what I assumed to be only a touch screen had actually a full microprocessor board driving it. When I first looked at the disassembled machine, I thought the whole machine was being driven by some firmware running on the STM32 board, and that the UART communication to the touch display was to a daughter board. But finding that the other board was sporting a microprocessor, in what seemed to be a smartphone development board form, changed my perspective. I was now thinking that the smartphone dev board was the actual computer doing the heavy lifting, while the STM32 board was functioning as a peripheral to it.

I shared my findings with the consultant who suggested we use Macrodroid for the first machine. He suggested we start the machine and prod around the GUI on the touch screen to get a sense of what was the OS running behind the scenes. He also suggested it would probably be Android of some sort, as it was relatively easy to get that working as a touch screen UI for the machine.

It took a while, but eventually, I got to the machine’s settings by trying to set the system language for the UI. It seems the developers of the GUI decided to use the OS language settings, and so I could confirm it was running a recent enough Android system.

I also talked with the distributor, and through him got to the guy who “hacked” the machine prior to me. The last guy got to about where I got, by prodding the UI of the machine set in his office. But this guy got blocked by the distributor’s password - whereas in the case of my machine the distributor had unlocked it specifically for our use.

Stay Tuned

In the next post, I will cover how I used ADB and Appium to overcome the challenges posed by the dynamic screens of the second machine. Stay tuned for a detailed walkthrough of the process and the final solution!

Integrating Machines Using Android and Python - Part 1

2022-01-11T00:00:00+00:00

As part of working for an engineering firm, I sometimes need to integrate machines into a system designed by one of the firm’s mechanical engineers. This could be part of a POC, so we can move things a bit quicker. Or it could be part of our final product - integrating proven products into our designs means we can focus on getting the final machines built quicker, and less prone to errors. This is done on a regular basis in different disciplines of engineering. In mechanical engineering, a lot of designs integrate standard parts such as bearings or motors, for example. In electronics engineering, relying on pre-built modules in designs could save a lot of hassle.

But in this one machine we’re building, I needed to integrate a couple of different machines that required a different set of skills, since they were (definitely) not meant to be included in any larger machine - and that’s what I want to write about. In this particular machine, we needed to incorporate a consumer-level machine into our bigger system. We ended up incorporating two different machines, one for the POC, and a different one for the first real iteration of the machine.

What are my options, anyway?

There are a few different options for controlling machines that have Android interfaces.

I’ve had a bit of a hard time figuring out how to get it working, so I wrote it down. Mostly, to not have to search for answers again in the future.

These options include:

Automate the machine through macros I wrote with webhooks, controlling an app designed by the machine manufacturer, running on a phone hidden inside our machine.
Automate the machine by recording UART between the controller and the UI device (started, deemed too hard).
Automate the machine by reverse engineering the controller firmware (never tried, WAYYYY too hard for the specific machine).
Automate the machine by leveraging ADB:
- Blindly.
- UIAutomator (deprecated by Google, works only for static content).
- OpenCV pattern matching (last resort).
- Appium (UIAutomator2 driver, FOSS, QA tool).

The first machine

The first machine was selected thanks to its ability to be controlled remotely via an Android app that sent commands through Bluetooth.

Initial probing

Since I knew it was Bluetooth-based, I first tried sniffing the communication, using Wireshark and Nordic nRF. I ordered an nRF52840 Feather from Adafruit, and followed Adafruit’s great guide for installing and using that to sniff BLE communication between the machine and the application installed on my phone. I also used Nordic’s own nRF Connect on my phone (there’s a desktop app as well) to see the BLE descriptors as well.

But while looking at the sniffed packets I realized the machine was only advertising its existence through BLE, then (probably) moving onto the more secure Bluetooth Classic communication for actually being controlled through the accompanying smartphone app.

A quick look through the code for the app (apk packages can mostly be unzipped) convinced us it was time to think of a simpler way for controlling this particular machine.

Second best - but winning

As a quicker method to get the POC rolling, a consultant suggested we try installing a macro app for Android. Specifically, he suggested we use Macrodroid, since he’s had good results using it in the past.

The process was quite simple, if a bit tedious. I found a disposable phone from an earlier project, installed the application for controlling the machine as well as Macrodroid, and started setting up a macro that would start the app, wait for a bit, then press the desired button on the second page, then delay again, and finally the third button.

For selecting textual buttons, Macrodroid offers a search functionality. For moving sliders around, I enabled Android’s Pointer Location in the Developer’s options, and set up moving the slider based on the actual display of the device I used.

Finally, I set up a webhook with some query parameters to trigger the macro on the device - and my part of the job was done.

I passed the webhook addresses and the parameters to the guy writing the rest of the machine’s controller, made sure the phone was always on and had its Bluetooth, location, and WiFi, always on (and with power), and went on to the next project.

Stay Tuned

In the next post, I’ll cover the next iteration of the machine, and how I had to use a (very) different approach to get it working. Stay tuned for a detailed walkthrough of the process and the final solution!

Customizing Jekyll With A Portfolio

2021-01-20T00:00:00+00:00

In my last post I detailed how I used Jekyll and the the new version (3) of the Minima theme, to style my site so that it looked similar to the WordPress team’s Twentythirteen theme.
In this post, I am going to outline how I created a new layout portfolio that could showcase any project of my making/tinkering/engineering/enginerding - be it a woodworking project, a mechanical contraption, or a Python script.

Design considerations

The portfolio layout started with a basic need. I wanted portfolio to keep the layout I was using on my WordPress site, which uses a paid plugin called GridKit. I was quite happy with the layout this plugin allowed me to have, but I wanted a bit more.
My WordPress portfolio using GridKit
First, I wanted to be able to display code snippets and projects’ readme’s. Second, I wanted to have a better understanding of how things were built. Third, though the plugin has got a free version, I found that it was not enough for what I wanted to display, and it was costing me money.
So, I set out to find how I could display a grid of pictures representing my projects, with a description scrolling from the top of each grid box describing the specific project on :hover, then on a click to open a modal displaying the project’s content - be it a README.md, a code snippet, or pictures.
GridKit portfolio modal

Finding solutions

Using Jekyll’s structure and YAML front matter

I started out with adding a collection to my _config.yml, then adding a new layout, _layouts/portfolio.html. A new page, portfolio.md was also added, as well as some markdown files in the _portfolio/ folder, describing the project and adding some content for the new layout to parse. The example below is of one the projects, and displays nicely a lot of the capabilities. It has a title, and some cover picture, and a text displaying the project in a few words. It also has tags, so the layout can group several projects sharing the same tag for the person viewing the portfolio to filter by, and a link to the project’s web page (in this case a github repository for the project). The last part of the yaml front matter is an array containing pictures, code, or markdown, to be rendered in the specific project’s modal.

---
cover: ble_slider_web_app_2.png
title: "Web-BLE control app + esp32 sketches"
tags: iot control
link: https://github.com/UriShX/ESP32_fader
modal:
  - md: https://raw.githubusercontent.com/UriShX/ESP32_fader/master/readme.md
  - img: ble_slider_web_app_2.png
  - img: ble_slider_web_app_1.png
---
A web-ble responsive web app to demonstrate control over BLE of ESP32 projects. 
Includes links to github repo of web app, and to a couple of ESP32 sketches for testing.

However, when displaying code, my layout needs a bit more information, so an object specifying the language to be displayed and whether line numbers should be rendered is also required. That is since code in one language can sometimes contain bits in some other language, such as a HTML page contained inside an embedded C code, for example.

modal:
  - code: 
      link: https://raw.githubusercontent.com/UriShX/portfolio/master/Roboclaw_control_over_ESP32_with_AP_for_control/roboclaw_esp32_w_AP_and_config.ino
      lang: cpp
  # - linenos: linenos

Adding modals

The two main characteristics of the GridKit portfolio I had on my WP site were its general layout, and that each project was displayed in its own modal window. Laying out the projects in a grid and animating their display is done through Jekyll’s own structure, and detailed below. But I had to come up with a way to display each project’s content in it’s own modal, which is not part of the theme I was using, Minima.

I started out looking for a ready-made solution, and my search parameters were threefold:

I wanted a modal library that will use the least amount of JavaScript - so, mostly implemented in CSS.
I wanted the modal library to be either pure CSS, or - preferably - in Sass.
I also did not want to include a whole CSS framework, mostly because I wanted to use the existing layout of the Minima theme.
And, naturally, I would prefer a library that will have ample examples, and is developed enough to use in production.

Searching Google got me to CSS-Modal, which checked all the boxes. All I had to do was figure out how to use it in my new layout. I copied it’s .scss files into Jekyll’s _sass/ folder (under a fitting subfolder, _sass/drublic-css-modal/), and it’s few JS utilities into Jekyll’s assets folder, under assets/script.

Importing and displaying remote code

As I wrote above, I wanted to add the ability to display projects that consist of code or markdown files. Simply put, not all of my projects have a visual element to them, but I still wanted to display them in a way that will be similar to the more visually- centric projects.
I wanted to use Jekyll’s statically built structure to my advantage, and import all the code snippets and MD files at build time. That way, the site contains all of it’s components, and can even be displayed off-line quite easily, which can be important when displaying my work in places with bad internet connection. In those cases, the site can be displayed directly from my phone’s cache.

For achieving that, I had to use a couple of Jekyll plugins.
The first one because Jekyll’s include and include_relative tags were designed to include HTML files in layouts, and not content - thus they allow only for including files from the current Jekyll project.
The second plugin was needed to allow passing parameters to Jekyll’s highlight tag, in a similar way to what’s possible with the include tags. Jekyll uses Rouge to format code snippets, and it does so quite intelligently, figuring out the code’s language from its’ syntax. Since some of my projects mix languages (as I wrote above, HTML code in an embedded C program, for eg.), or use non standard file extensions (such as .pde for Processing, or .ino for Arduino), I wanted to have the same control over the rendered language as for the source itself.

I searched RubyGems for Jekyll plugins that could quite fill my requirements, but couldn’t find any ‘out of the box’ solutions, so I set about adapting and writing some code.

For including files, I found jekyll-remote-include. This plugin can include a remote file in a similar way to Jekyll’s include tag already, but I wanted to add to it a capability. I wanted to be able to use Liquid to render my portfolio from a template, so each included remote source should be passed on as a variable. To achieve that, I forked the plugin’s repository, and added a begin/rescue block (which is Ruby’s way of calling a try/catch block). In this block the plugin simply tries to parse the URI from the given parameter, and if it fails, it tries to parse the URI as a context variable. That seems to work in a decent enough manner, so I opened a PR with my addition, which I hope will be merged to the plugin repo some day. In the meantime, my revised plugin can be found here.

As for passing variables to the highlight tag, I really couldn’t find what I was looking for as a plugin, so I set about writing my own, or rather re-writing Jekyll’s mainline implementation. My first implementation was pretty rough. I basically tried to use the same method as I did with the remote-include plugin, using a begin/rescue block, but that only got me part way to my goal. What it actually do is fail gracefully, by checking if the code is a context variable, and trying to parse it. If it cannot do that, the code simply continues as the mainline tag does. Once I figured that didn’t quite do what I was expecting it to do, I set about implementing a better model for selecting the route to go by.
Implementing a better model for detecting the passed parameters required a change to the regular expression in the tag’s code. I wanted the syntax to be as close to Jekyll’s implementation of Liquid, so my implementation is looking for curly braces to filter passed variables, and if it does not find any - it considers the parameter as a constant. That means backwards compatibility is maintained with Jekyll’s highlight tag, while allowing for variables to be passed. I opened a feature request for that in Jekyll’s repository, but I haven’t opened a PR yet. My implementation can be found on rubygems.org as jekyll-highlight-param.

Adding the plugins into a Jekyll site is quite simple, by directing the Gemfile to pull straight from Github:

# frozen_string_literal: true

source "https://rubygems.org"

gem "jekyll", "~> 4.1"
gem "minima", :github => 'jekyll/minima' #:git => "[email protected]:jekyll/minima.git" # github v.3.0, latest build is 2.5.1

group :jekyll_plugins do
    gem "jekyll-remote-include", :github => 'UriShX/jekyll-remote-include', :branch => 'context-variables' #:git => "[email protected]:UriShX/jekyll-remote-include.git", :branch => "context-variables"
    gem "jekyll-highlight-param", :github => 'UriShX/jekyll-highlight-param' #:git => "[email protected]:UriShX/jekyll-highlight-param.git"
end

Note, that to direct a plugin to a specific branch (like to my revised jekyll-remote-include plugin, which accepts variables, and is not yet merged to the main plugin repo), it is possible to pass :branch => to the gem.

building up the page

Building the page using Liquid

Building a portfolio from a collection and a bunch of markdown was a bit convoluted. I used quite a lot of flow control logic, checking whether the collection was indeed a portfolio, looking through the files in the _portfolio directory, pulling the content of each (the text describing the project), and creating a modal for each sort of project - code based, a markdown file, a picture gallery, a combination of those, or even none at all. I then arranged the projects (programatically) by order of their place in Jekyll’s _config.yml, and push the projects that were found in the _portfolio folder but were not in the _config.yml to be sorted alphabetically at the end of the portfolio.
Cover images and tags are also pulled from the project’s front matter, and by using some icons from Google’s Material Design Icons, (linked to in _includes/custom-head.html), links and modals are linked to each graphical representation of a project.

Most of the modal code for image galleries is based on the examples for the CSS modal library I used. With some Liquid control logic, and with the aid of the two plugins I wrote of above, creating a nice modal view of markdown and code files was not that difficult to implement also.

The whole portfolio layout can be found in my jekyll-portfolio repository, under _layouts/portfolio.html.

Layout using Sass

Since I’m using Minima v.3, it was quite easy to link the CSS modal library. All that’s needs to be done was linking the modal library’s .scss files, which I placed under _sass/drublic-css-modal, into Minima’s _sass/minima/custom-styles.scss, under an @import tag.

The rest of the portfolio styling can be found on lines 226 onward:

I then used the new CSS grid layout to style the layout of the .portfolio-wrapper class, in a similar way to how GridKit styles portfolios.

Some more styling for each .portfolio-block class to get the zoom-in and sliding animations for the text and links, darkening the background when a modal is open, as well as creating a satisfying layout for narrow screens such as phones, and that was it.

Filtering using JS

Finally, I needed a way to filter the projects by some key words. I used Jekyll’s tags to group them using Liquid in _layouts/portfolio.html, then display the tags on top of the portfolio blocks.

I then wrote a short JS script for querying the list of a href links created by the Jekyll layout, generating an array of links to projects, and by adding an event listener to the hashchange event, hiding or revealing the projects with the appropriate tags.

The script uses a mix of ES5 and ES6, though I believe it is quite easy to read and understand. I made a conscious decision to not provide support for older browsers, so using modern JS is easier for me.

Conclusions

This has been a really long post, so I think I’ll end here. The whole layout can be found on the repository I opened for it, called jekyll-portfolio.

Writing my own implementation for a portfolio layout in Jekyll was quite a learning experience. I still have some ways to go, as I still need to figure out a better way for sizing images better, and perhaps adding a PWA capability.

Besides that, the versioning for jekyll-highlight-param’s does not follow the semantic versioning scheme, and my additions to jekyll-remote-include need some refining.

Customizing Jekyll’s Default Theme

2020-12-16T00:00:00+00:00

Jekyll installs with a nice, clean theme, called Minima. While there are a lot of nice themes out there, I had both a clear view in my head of what I wanted to accomplish, and ample time thanks to CoVid19. So, armed with Minima’s source and Jekyll’s documentation, I decided to have a go with modifying the theme a bit, and in this post I’ll try to outline the way I went about it. I started off with the latest version of the Minima, even though it’s not published as a stable version, and is not what you get ‘out of the box’ when you get when you start a new site with the jekyll new <PATH> command. The main reason for that was that I had more of a desire for understanding how Jekyll themes were built, rather than a ‘getting the job done’ approach. So, I cloned the theme from its Github repository, and started to modify the way things looked and felt, to better fit my desired look and feel. If you’d like to follow along, you would need to change your Gemfile to point to Minima’s Github repository, like so:

gem "minima", :github => 'jekyll/minima' # github v.3.0, latest build is 2.5.1

Set your theme on your _config.yml, then run bundle from the command line.

Design goals

I wanted my site to maintain the general look and feel of my existing WordPress blog, which uses the WordPress team’s Twentythirteen theme.
My existing WordPress blog
Starting with the general layout of this theme, I wanted to have a header that will showcase the site’s name, some subtitle with the site’s motto, a logo, and to have all that in front of some colorful banner. I grabbed the banner straight from the WP theme I was using, re-designed my site’s logo, and started working on the layout.

Modifying Minima

Minima’s latest (unreleased) version, v.3.0, has got some nice features for easy customization. It imports some empty files while rendering both the theme’s default.html layout as well when compiling Sass (.scss) files. This complements Jekyll’s [theme building process]](https://jekyllrb.com/docs/themes/) nicely, as it allows the user to only modify what’s needed, without needing to copy and paste whole files. I found out that worked great with styling, but needed to make quite a few changes in the _includes HTML files, since there was no easy way for changing the page layouts themselves without that.
You can click on the filename in the headings to go to the file the paragraph describes.

Modifying `_includes`

`custom-head.html`

This file is really all you need to change if all you want is a slightly different look and feel, and some basic branding. This file is included in the <head> section in the generated site, so including a favicon, for example, is perfect here. I decided to change the fonts, so I grabbed the cdn link from Google fonts for two fonts I like (and used on my old site) - Alef and Tinos. These two fonts were originally selected since they both support Hebrew as well as English (and other European languages), though I don’t intend to implement Hebrew on this site in the near future. I just sorta got used to them in my old site, and decided to keep them.
Those fonts will simply override Minima’s default font, by reassigning some variables in _sass/minma/custom-variables.scss

`header.html`

Starting off from the top of the page, the branding for my site consists of a banner, which features my name, a subtitle, the site’s logo, and a navigation bar. Those are all inserted through Liquid tags, pulled from either Jekyll’s _config.yml, or from the Jekyll’s structure.
The banner itself is a css background, using Liquid to insert a relative URL from the site’s assets folder:

<header class="site-header" style="background-image: url('{{ site.banner | escape | prepend: "/assets/img/" | relative_url  }}');">

Then, inside a banner HTML class, the template checks if there’s a site.logo image in _config.yml, which (if it is configured) is inserted as a relative URL to the site’s assets folder again.

<div class="img-container">
    {% if site.logo %}
        <img alt="logo" src="{{ site.logo | escape | prepend: "/assets/img/" | relative_url }}"/>
    {% endif %}
</div>

After that, the site’s title and subtitle are also pulled from _config.yml:

<h1 class="site-title">
    <a rel="author" href="{{ '/' | relative_url }}">{{ site.title | escape }}</a>
</h1>
<h2>{{ site.subtitle | escape}}</h2>

The navigation bar remains unchanged in it’s structure from Minima’s.

`footer.html`

<li class="p-name">&#169; {{ site.author.name | escape }} {{ site.author.copyright-year | escape }}</li>

`social.html`

I changed the included assets/minima-social-icons.svg to include an icon for Hackaday.io. Technically, the icon I included is for Hackaday.com, but I think it’s the more easily recognizable of the two.
To add it into my social icons, I simply added a single line to the list of social icons, which displays the Hackaday icon with the appropriate link, if it is configured in _config.yml:

{%- if social.hackaday -%}<li><a rel="me" href="https://hackaday.io/{{ social.hackaday | cgi_escape | escape }}" title="{{ social.hackaday | escape }}"><svg class="svg-icon grey"><use xlink:href="{{ '/assets/minima-social-icons.svg#hackaday' | relative_url }}"></use></svg></a></li>{%- endif -%}

Adding the icon for Hackaday.io

I used ntembed’s Hackaday social media icon, and simply added the svg to Minima’s assets/minima-social-icons.svg using Inkscape. As I wrote above, the icon is actually for Hackaday.com, but that was good enough for my purposes.
I then added hackaday: UriSh (which is my username on Hackaday.io) to Minima v3’s social_links: category, under minima:, in Jekyll’s _config.yml. You can have a look at a sample _config.yml file here.

Modifying `_sass/minima`

Minima’s latest version (v.3.0) provides a few skins, to change it’s color scheme. It also, as I wrote above, gives the user hooks to modify the theme further. This is achieved by importing a couple of empty scss files, one right after some initial variable assignments, then another after most of the site’s styling is defined.
I set up the Minima’s skin to be skin:solarized, in _config.yml under minima:.

`custom-variables.scss`

In this file one can override Minima’s defaults by reassigning certain variables. It is also possible to add custom variables, which I needed for controlling the position and styling of the navigation bar’s look and feel.
I first overrode the default font for the site and adding a second font for headlines, using the fonts I linked to in _includes/custom_head.html, Alef and Tinos.
I added a couple of navbar definitions, for background color and hover effects, linking them to Minima’s skin definitions, and I also added a couple more breaks for size, since I found out my subtitle was too long to handle the banner / navigation bar properly.

`custom-styles.scss`

This file contains the styling for the site’s branding, as well as the styling for other layouts I will cover in future posts.
First, I wanted to style code on my site in a way that will contrast the styling of other text in my site, to make it a bit more discernible. I did that by importing a stylesheet I got from the repository recommended in Jekyll’s docs.

@import url('zenburn.css');

Next, I defined the headings’ font to use Tinos, and since I decided to override Minima’s skin code highlighter defaults, I also redefined the highlighted area to fit. After that, besides importing some useful mixins, comes the actual styling for the site’s header, banner, logo, and navbar.
Since this is quite a long list of changes from Minima’s header, I will only list a few, and if anyone’s interested in investigating further, the file is hosted here, or leave me a comment down below.
I set the area for the banner (the background image defined in header.html) under the class .site-header:

background-color: $brand-color-light;
background-size: cover;
background-position: center;
min-height: 14.1vw;

The min-height attribute is defined to fit the WxH ratio of the banner image I used.
Next, the .banner class defines the styling for the logo, title, and subtitle.
The .top-nav class then defines the navbar, including background, position, and hover behavior.
The class .site-title overrides Minima’s default by disabling float and hover decoration.
The .site-nav class and its subclasses control the behavior of the navbar further, including it’s alignment according to window size, and also highlights and disables the link for the current page. \

One thing I found important while styling my site was setting the correct breaks for window width. I added a couple more @media queries under .site-title:

@media screen and (min-width: $on-med-wide) {
    top: unquote('calc(max(14.1vw, 165px) - 1.6rem)');
}

@media screen and (min-width: $on-medium) and (max-width: $on-med-wide) {
    top: unquote('calc(max(14.1vw, 200.25px) - 1.6rem)');
}

to allow for better positioning of the navbar, while letting the banner grow in height to fit the long subtitle I chose.

Conclusion

Styling Minima v3 to follow an existing WP theme is quite straight forward mostly, though I did get into some ruts.
Mostly, it’s a matter of moving overriding existing definitions, and figuring out what class an element is in, in order to change it’s behavior in a couple of .scss files Minima imports by default.
Most of this work has been done in preparation for a more elaborate change I needed to implement, which is adding a portfolio page to this site. One of my next posts will be on that subject, as it was the driving force for moving from WP to Jekyll, even though it might not seem like an obvious decision.

You can find a ready template that includes the above changes I made to Minima in Jekyll-portfolio, which also includes the portfolio page and static commenting.

If anything above is not clear, or if you find you have better ways to do what I outlined here, or if you find the template repo useful, please leave a comment.

Adding Staticman To Jekylls Minima Theme

2020-10-21T00:00:00+00:00

After modifying the theme to fit my aesthetics, then adding a nice portfolio layout (which required writing and modifying a couple of plgugins), it was time to add a commenting system to my blog. Now, it’s not that I write a lot of blog posts, but for some projects I want to write some sort of how-to, or some sort of a thought process documentation - and for those, I really do wish for people to interact with me. Now, Minima already comes with a layout ready for adding Disqus comments. But I don’t really like relying on the free tier of a paid service, so I did a quick search and found Staticman, which is a self hosted solution. I think Staticman works in quite a clever way, using Jekyll’s own structure to save the comments as data files. Staticman’s back end is a self hosted endpoint which receives a POST request of the comment from a form displayed on each post. It then opens a Pull Request on the repository containing the blog with a new data file containing the comment - which in turn will be rendered with the related post when the site re-renders.
Staticman’s Github repository lists a couple of useful links for setting those up, which I followed. The first is Travis Downs’ tutorial, which is great, and the second is Michael Rose’s blog post documenting how he implemented Staticman in his own theme. I followed these two, adjusting as I went along to fit with my blog’s layout.
Required changes:

Include files comments.html, comment.html, and comment_form.html. If my memory serves me right, I copied these from Michael Rose’s blog post (MadeMistakes).
Rewrite JS to not require JQuery. That was done since that’s the only part in my site that uses JQuery, so that’s an unnecessary dependency. Since I don’t want this blog post to be entirely about that, I’ll just mention that I used the fetch API to send the comment to the Staticman endpoint, and you can find the code here.
Substitute colors in comment-styles.scss to work with Minima’s skins.
Add comments tag to _layouts/post.html as in adding the following to the bottom of the file:

  {%- if page.comments == true -%}
    {%- include comments.html -%}
  {%- endif -%}

Add the comments tag to the front matter of the posts I want to have comments on. ~~I also found that using the Heroku deploy button does not work correctly with my browser, so I followed issue #394 on the Staticman repo.~~ I used Heroku for the first deployment, but since the free tier is long gone, I did not use it for a good while. Since I recently got a VPS, I decided to deploy Staticman on it. I’ll document that process in the next section.

Deploying Staticman on a VPS

I looked online for resources on how to deploy Staticman on a VPS, using Dokku, and found this blog post. Reading through it, I decided I didn’t like the idea of setting up Node.JS and NPM, so I re-read the post and found there’s a link to another post, on another blog, that used the Docker image which Staticman ships with. So, I decided to document my own experience, to save myself the future headache of when I try to re-deploy it (which I know will happen sometime in the future, when I change VPS providers or something).
Here’s what I did:

Created a new app on my Dokku instance.
```
dokku apps:create staticman
```

Set up a URL for the app.

dokku domains:set staticman staticman-subdomain.domain.com

Set up letsencrypt for the app.

dokku letsencrypt:set staticman email [email protected]
dokku letsencrypt:enable staticman

Create a GitHub token, from my existing bot account (I used the same account for the Heroku deployment).
- Go to Settings -> Developer settings -> Personal access tokens -> Generate new token.
- Select the ‘Tokens (classic)’ option.
- Give the token a name, and select the repo scope.
- Copy the token.

Create a RSA key pair for the app.

It’s important to use the -m PEM flag, since the default is OPENSSH.

ssh-keygen -m PEM -t rsa -b 4096 -C "[email protected]" -f ~/.ssh/staticman_rsa.pem 
# copy the public key to the clipboard (pbcopy works on MacOS)
cat ~/.ssh/staticman_rsa.pem.pub | pbcopy

Add the public key to the GitHub bot account.
- Go to Settings -> SSH and GPG keys -> New SSH key.
- Paste the public key.
- Give the key a name.
- Click Add SSH key.
Set up the required environment variables.
- I scp‘d the private key to the Dokku instance from my local machine (scp ~/.ssh/staticman_rsa.pem user@ip:~) and then added it to the app.
```
dokku config:set staticman RSA_PRIVATE_KEY="$(cat staticman_rsa.pem)"
dokku config:set staticman GITHUB_TOKEN=your_github_token
```

Deploy the app (from my local machine).

git clone https://github.com/eduardoboucas/staticman.git
cd staticman
git remote add dokku dokku@yourserver:staticman
git push dokku master

Test the app exists.

curl https://staticman-subdomain.domain.com:port
# Should return 'Hello from Staticman version 3.0.0!'

Invite the bot to the repository.
- Go to the repository settings.
- Go to Manage access.
  - In my had to delete the bot’s access, since I had previously added it to the repository to work with Heroku.
  - Click Delete on the bot’s username.
- Click Invite a collaborator.
- Paste the bot’s username.
- Click Add.

Accept the invitation using the Staicman app.

curl https://staticman-subdomain.domain.com/v2/connect/github/your_github_username/your_repository_name
# Should return 'OK'
# If it returns 'Invitation not found' it might be that you've already accepted the invitation manually, or that your GITHUB_TOKEN is not right.

Add (or modify) the existing _config.yml file in the Jekyll blog to point to the correct Staticman endpoint.
- I decided to stick to the v2 endpoint, since I was using it before. From what I understand, the v3 endpoint requires a GitHub app, which I did not want to set up at this point in time.
```
staticman_url: https://staticman-subdomain.domain.com:port/v2/entry/your_github_username/your_repository_name/master/comments
```
Test the configuration.
- Add a comment to a post.
- If the comment does form does not seem to work, check the browser’s console for errors.
- In my case, I noticed that in the ‘network’ tab, I got a ‘500’ error. I checked for possible causes until I remembered I had to set the ‘reCpatcha’ keys in the Staticman app correctly. Since I didn’t remember how I initially set them, a great help here was this blog post, which I mentioned earlier, and another post I stumbled upon when trying to figure things up.
Set up the reCaptcha keys.
- Go to the reCaptcha admin console.
- Register a new site.
- Copy the site key to both the _config.yml file and the staticman.yml in the Jekyll blog.
- Copy the secret key, and go to https://staticman-subdomain.domain.com/v2/encrypt/[recaptcha-secret-key] in your browser to get the encrypted key.
- Copy the encrypted key to both the _config.yml file and the staticman.yml in the Jekyll blog.
Save and re-deploy the Jekyll blog.
Test the comment form again.
- If the comment form does not work, check the browser’s console for errors.
- If the comment form does work, check the repository for the new pull request.

I suppose that’s it. I really hope this post helps someone in the future, or at least helps me when I need to re-deploy Staticman. I’m really happy with how it turned out, and I’m looking forward to interacting with people on my blog. I’m also looking forward to writing more blog posts, and maybe even finishing some of the drafts I have lying around.

Uri Sh.

Enhancing the VPN Profile Switcher: Introducing Local DB Creation and a Self-Test Tool

1. Creating a Local Database with get_groups_and_countries_to_tsv.sh

2. Building a Self-Test Tool with vpn-status-checker-openwrt.sh

Looking Ahead: WireGuard Testing and Beyond

Updating the `vpn-profile-switcher.sh` to Include WireGuard Support

Overview of the Script

Key Updates for WireGuard Support

New Options in the Usage Function

Verification Functions

Fetching Recommendations

Configuration Handling

Creating and Enabling Entries

Restarting Services

Execution

Conclusion

Upgrading the VPN Profile Switcher Script for OpenWRT

Understanding the Script’s Purpose

Research and Preparation

Setting Up WireGuard

Script Details

Integrating Machines Using Android and Python - Part 3

Overcoming Dynamic Screen Challenges with ADB and Appium

Setting Up Appium for Development

Crafting the ADB Hierarchy Handler

Utilizing Helper Functions for Parsing

Wrapping Up and Testing

Conclusion

Openwrt On Raspberry Pi For Shell Script Development

Setting Up an OpenWRT Dev Environment on a Raspberry Pi 3 B+

Flashing OpenWRT

WiFi Hotspot and WAN Setup

Adding a WiFi WWAN

Why the WWAN?

Getting OpenVPN Running

SSH Convenience

Final Touches

Edit: 2024-07-24

Wrapping Up

Integrating Machines Using Android and Python - Part 2

First thoughts, again

Stay Tuned

Integrating Machines Using Android and Python - Part 1

What are my options, anyway?

The first machine

Initial probing

Second best - but winning

Stay Tuned

Customizing Jekyll With A Portfolio

Design considerations

Finding solutions

Using Jekyll’s structure and YAML front matter

Adding modals

Importing and displaying remote code

building up the page

Building the page using Liquid

Layout using Sass

Filtering using JS

Conclusions

Customizing Jekyll’s Default Theme

Design goals

Modifying Minima

Modifying _includes

custom-head.html

header.html

footer.html

social.html

Adding the icon for Hackaday.io

Modifying _sass/minima

custom-variables.scss

custom-styles.scss

Conclusion

Adding Staticman To Jekylls Minima Theme

Deploying Staticman on a VPS

1. Creating a Local Database with `get_groups_and_countries_to_tsv.sh`

2. Building a Self-Test Tool with `vpn-status-checker-openwrt.sh`

Modifying `_includes`

`custom-head.html`

`header.html`

`footer.html`

`social.html`

Modifying `_sass/minima`

`custom-variables.scss`

`custom-styles.scss`