selivan.github.io

Execute script before starting service in docker-compose

Sat, 28 Sep 2024 00:00:00 +0300

Sometimes you need to execute shell commands before starting service in docker compose.

You can create oneshot service using the image you need for those commands, entrypoint /bin/sh and shell commands in command option.

YAML multi-line syntax allows to conviniently place the script inside docker-compose.yml. Use set -x to get verbose logs of script execution. Don’t forget that the last command should have exit code 0.

Then you make the main service dependent on oneshot service, with option condition: service_completed_successfully. Unlike default dependency, this one requires service to successfully finish execution, not to be runnig. Other possible conditions are service_started(default) and service_healthy(healthcheck ok).

docker-compose.yml:

services:
  create-mysql-socket-dir:
    image: "busybox:stable"
    restart: "no"
    entrypoint: /bin/sh
    command:
    - "-c"
    - |
      set -x
      mkdir -p /var/run/mysqld
      chown ${MYSQL_UID}:${MYSQL_GID} /var/run/mysqld
    volumes:
      - /var/run:/var/run
  mysql:
    image: percona/percona-server:5.7
    depends_on:
      create-mysql-socket-dir:
        condition: service_completed_successfully
    user: "${MYSQL_UID}:${MYSQL_GID}"
    volumes:
      - /var/run/mysqld/:/var/run/mysqld/
    command: "--socket=/var/run/mysqld/mysqld.sock"
  ...

Simple bash debugger using trap DEBUG

Sat, 21 May 2022 00:00:00 +0300

Inspired by this post by @tminnigaliev.

Besides traps(handlers) for signals, bash have 4 special traps:

EXIT to run on exit from the shell.
RETURN to run each time a function or a sources script finishes.
ERR to run each time command failure would cause the shell to exit if set -e is used.
DEBUG to execute before every command.

The last one allows to create a simple debugger inside a bash script:

function _trap_DEBUG() {
    echo "# $BASH_COMMAND";
    while read -r -e -p "debug> " _command; do
        if [ -n "$_command" ]; then
            eval "$_command";
        else
            break;
        fi;
    done
}

trap '_trap_DEBUG' DEBUG

Now before executing every command, it is printed and we have a command prompt. There we can print any command to execute in the script context or an empty line to continue.

Also it is possible to run another script in this debugging mode without modifying it. The only catch is that bash functions and scripts inlined by source do not inherit DEBUG, RETURN and ERR traps. We can use set -T to allow inheriting DEBUG and RETURN traps.

Here is such a simple debugger with some bells and whistles added: selivan/bash-debug

Also it is worth mentioning that a gdb-like debugger for bash scripts exists: bashdb. It has way more functionality but requires building from source and installation.

WSL fix using random private subnets

Mon, 12 Jul 2021 00:00:00 +0300

WSL is a nice way to work with Linux development environment from Windows. It works pretty decently after version 2, that switched to using proper virtualization instead of translating syscalls and other weird magic.

Unfortunately, it has one serious problem: subnet for WSL is selected randomly from all available private subnets, preferably from 172.16.0.0/12 range. So you turn on your notebook, start WSL, connect to your work VPN and oops - that subnet is already used 🤷‍♂️.

Issue on Github to add subnet configuration support in WSL is opened since 2019. People in comments contacted developers and their position is that this by design, to make WSL networking transparent for newbies, and this won’t be changed.

Funny thing, VirtualBox selects random private subnet for host-only networking adapters. But these subnets are fixed after initial creation and easily can be changed later. Newbies are happy, advanced users are happy, and nobody is having problems. But that’s not how WSL developers like to do it.

WSL selects random private subnet on first start, subnet will not be changed after wsl --shutdown, only reboot will help. But it does ignore subnets that are already in use. In comments to the issue people proposed to create dummy network interfaces using all subnets except what we need WSL to start using. @jgregmac made this script: github.com/jgregmac/hyperv-fix-for-devs.

For some reason it does not work for me, so I made my own version. It brings up dummy network interfaces, starts WSL and then brings them down.

First, I already have VirtualBox on my machine. So I will use it’s host-only network interfaces to protect my subnets.

VirtualBox:

File - Host NetworkManager
- Create - IPv4: 172.16.0.1 Network Mask: 255.240.0.0 DHCP Server: [ ]
- Create - IPv4: 10.0.0.1 Network Mask: 255.0.0.0 DHCP Server: [ ]

⚙Settings - Network & Internet - Change Adapter Options

Rename interfaces to something more meaningful, like “Vbox-WSL-fix-10” and “Vbox-WSL-fix-172-16”.

Create script C:\Users\<YOUR USERNAME>\bin\WSL-subnet-fix.ps1:

# Start dummy interfaces protecting required private subnets

netsh interface set interface "Vbox-WSL-fix-10" enable
netsh interface set interface "Vbox-WSL-fix-172-16" enable

# start WSL, that is forced to select subnet not overlapping with protected subnets

wsl ip a

# Stop dummy interfaces protecting required private subnets

netsh interface set interface "Vbox-WSL-fix-10" disable
netsh interface set interface "Vbox-WSL-fix-172-16" disable

Winkey+R - taskschd.msc

Create New Task

General
- Name: WSL-subnet-fix
- (*) Run only when user is logged in
- [x] Run with highest privileges
Triggers
- At log on: Specific user: your user
Actions
- Start a program
  - Program: powershell.exe
  - Arguments: -windowstyle hidden -file "C:\Users\<YOUR USERNAME>\bin\WSL-subnet-fix.ps1"

Voila, you can use WSL and don’t have it screw up your work VPN.

If you need fixed static IP for your WSL machine, you may look at this solution by @protang. It adds secondary fixed private subnet to WSL network interface.

Change TLS/SSL version and cypher parameters for Linux service using openssl library

Fri, 14 May 2021 00:00:00 +0300

Sometimes you need to change TLS/SSL parameters for a service using libssl library from openssl, but the service config does not accept that parameters. In this example, I had to change rsyslog forwarder parameters to send logs to the target that wasn’t playing nice with TLS 1.3 and modern encryption protocols.

libssl and applications using it take configuration parameters from configuration file set by environment variable OPENSSL_CONF or from default file /etc/ssl/openssl.cnf.

Openssl documentation is not the easiest one to read, but man 5ssl config and some googling got me what I wanted.

/etc/ssl/openssl.no-tls13.cnf:

# This definition stops the following lines choking if HOME isn't
# defined.
HOME                    = .

openssl_conf = default_conf

[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
MaxProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=1

/etc/systemd/system/rsyslog.service.d/override-openssl-params.conf:

[Service]
Environment="OPENSSL_CONF=/etc/ssl/openssl.cnf.no-tls13"

And finally applying the new configuration:

systemctl daemon-reload
systemctl restart rsyslog

Systemd broken network.online target workaround

Wed, 23 Dec 2020 00:00:00 +0300

man systemd.special says:

Units that strictly require a configured network connection should pull in network-online.target (via a Wants= type dependency) and order themselves after it.

Despite that, I ran into a problem: service that requires outside network to be immediately available fails to start with configuration like described in manual:

[Unit]
Wants=network-online.target
After=network-online.target

Here is an ugly workaround for that:

[Unit]
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=sh -c "while ! ip -4 r | grep ^default; do echo waiting for ipv4 default route to appear; sleep 0.5; done"
ExecStart=<actual service here>

If you want not just default route available, but certain IP address:

ExecStart=sh -c "while ! ping -c 1 -W 0.5 <IP>; do true; done"

Syncronize time by NTP before starting any services in Linux

Wed, 23 Dec 2020 00:00:00 +0300

Servers often have wrong clock on startup. NTP services, like ntp, chrony and systemd-timesyncd try to correct clock gradually to avoid weird bugs in software. Therefore, if server has a large clock offset on startup, it works with incorrect clock for several minutes.

In my experience, AWS instances may have clock error up to 5 minutes on startup. Server writing log timestamps 5 minutes in the past or in the future is not always a good idea.

Solution is to force NTP time syncronization once before starting any other services. I prefer to use chrony: it can act both as always runnig NTP client and one-time syncronization tool; chronyc clearly reports syncronization status, making it easy to monitor; it is recommended by AWS.

All other services in multi-user.target will start after time is syncronized or failed to syncronize after given timeout.

/etc/systemd/system/ntp-sync-once.service :

[Unit]
Description=Quick sync NTP one time and exit

Wants=network-online.target
After=network-online.target

Before=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=True
# Ugly workaround for not working properly network-online.target
ExecStartPre=sh -c "while ! ip -4 r | grep ^default; do echo waiting for ipv4 default route to appear; sleep 0.5; done"
# -t <timeout in seconds>  timeout after which chronyd will exit even if clock is not syncronized
ExecStart=/usr/sbin/chronyd -q -t 20

[Install]
WantedBy=multi-user.target

network-online.target sometimes is not working as expected, ExecStartPre line is a workaround for that.

Letsencrypt: auto generate bundle certificates for Haproxy

Wed, 02 Dec 2020 00:00:00 +0300

I was surprised to not get instant answer googling “letsencrypt automatically generate bundle certificates for haproxy”.

So here it is, to save 5 minutes of your time.

Generates bundle(fullchain + privkey) certificates for everything in /etc/letserncrypt/live/*/fullchain_and_privkey.pem, does not update generated files if not necessary.

NOTE: renew hooks are not triggered when you run certbot certonly, so for the first time you should run it manually.

Using Ansible with bastion or jump host with variables

Fri, 30 Oct 2020 00:00:00 +0300

Sometimes due to network configuration or security reasons you can’t access a host directly, but should use intermediate host, so-called bastion or jump host.

My presious article on this topic, using ssh config file: Using Ansible with bastion host.

Another option is using variables. This approach is more flexible: it is easier to define different bastion hosts for different hosts and groups with variables.

Simplest option is to directly mention bastion host address and user:

group_vars/all/ansible_ssh.yml:

ansible_ssh_common_args: "-o ProxyCommand=\"ssh <bastion user>@<bastion address> -o Port=<bastion ssh port> -W %h:%p\""

And of course don’t forget to reset that variable for the bastion host itself:

host_vars/bastion_host/ansible_ssh.yml:

ansible_ssh_common_args: ""

Downside of simple option is duplicating bastion host coordinates in variables and in inventory. More complex setup to avoid the duplication:

group_vars/all/ansible_ssh.yml:

ansible_ssh_proxy_command: >-
  {% if bastion_host is defined and bastion_host != '' %}
  ssh {{ hostvars[bastion_host]['ansible_ssh_user'] }}@{{ hostvars[bastion_host]['ansible_ssh_host'] }}
  -o Port={{ hostvars[bastion_host]['ansible_ssh_port'] | default(22) }}
  -W %h:%p
  {% endif %}

ansible_ssh_common_args: >-
  {% if bastion_host is defined and bastion_host != '' %}
  -o ProxyCommand="{{ ansible_ssh_proxy_command }}"
  {% endif %}

# Default bastion host for all hosts
bastion_host=bastion1

And you still have to mention that bastion host itself doesn’t need this:

host_vars/bastion_host/ansible_ssh.yml:

bastion_host: ""

Smooth deploy of a loaded PHP web application with php-fpm and nginx, without downtime or errors

Thu, 09 Apr 2020 00:00:00 +0300

UPDATE 2020-05-10 Added the cachetool option, thanks to reddit users ds11 and SevereHeight for mentioning it.

Generally, the smoothest way to deploy a loaded web app without downtime or erros is by adding instances(hosts, containers) with the new version to balancer and then removing old ones, when they have served all running requests. But sometimes it is easier to switch app versions on instances without re-configuring balancer.

Here is how to do that with php-fpm and nginx.

After deploying the application(with deployer or whatever tool you prefer) you have directories with old and new application versions, shared files(like logs), and symlink current pointing to the new version. Like this:

/var/www/example.com:

releases/
    2020-04-08-12-00-00/
    2020-04-08-15-00-00/
shared/
    logs/
current -> releases/2020-04-08-15-00-00

Using the current symlink directly in nginx config may cause some troubles.

php-fpm is not designed to have app files suddenly change/disappear. Some of requests in progress when switching symlink may return 500 errors.

I suggest instead to use the absolute path to the required app version in nginx configs:

/etc/nginx/sites-enabled/example.com:

include /var/www/example.com/root_dir.nginx.conf;
root $root_dir;

/var/www/example.com/root_dir.nginx.conf:

set $root_dir /var/www/example.com/2020-04-08-15-00-00/public;

File root_dir.nginx.conf is updated on deploy, then nginx is reloaded. Nginx reload is done very carefully, all requests in progress are completed by old workers with the old config version, pointing to the old application version. Users won’t experience any errors or slowdown.

Now the only thing left to do is to clear the old code from php opcache. You can do that with opcache_reset() function. This function should be called from a PHP process using the fpm pool you want to reset the opcache for.

First option is to create a php file and make it accessible from localhost:

/var/www/localhost/opcache_reset.php:

<?php
header('Content-Type:text/plain');
try {
    if (opcache_reset()){
        print "SUCCESS: opcache_reset\n";
    } else {
        throw new Exception("ERROR: opcache is disabled\n", 500);
    }
} catch ( Exception $e) {
    http_response_code($e->getCode()?:500);
    print $e->getMessage();
}

/etc/nginx/sites-enabled/localhost:

server {
    listen 127.0.0.1:80;
    location / { return 403; }
    location = /opcache_reset {
        include fastcgi_params.conf;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        fastcgi_pass  unix:/var/run/php/phpXX-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $document_root/opcache_reset.php;
    }

Second option is using cachetool. It can connect directly to php-fpm unix socket or tcp port, no need to serve special php files. There is a ready deployer recipie to use it.

Finally, here is deployer config to do all that, example extending the symfony4 recipie with separate php file:

deploy-example-com.php:

namespace Deployer;
require 'recipe/symfony4.php';

desc('generate nginx config and reload');
task('nginx:update_config_and_reload', function () {
    run('echo "set \$root_dir /public;" > /root_dir.nginx.conf');
    run('sudo --non-interactive nginx -t'); // avoid reloading with incorrect config
    run('sudo --non-interactive systemctl reload nginx.service');
});

desc('call php-fpm opcache_reset from localhost');
task('php-fpm:localhost:opcache_reset', function () {
    run('curl http://localhost/opcache_reset');
});

after('deploy:symlink', 'nginx:update_config_and_reload');
after('nginx:update_config_and_reload', 'php-fpm:localhost:opcache_reset');

inventory('inventory.yml');
set('deploy_path','/var/www/example.com/');
set('repository', '[email protected]:my/example-com.git');
... // some other perameters

And example with cachetool:

deploy-example-com.php:

namespace Deployer;
require 'recipe/symfony4.php';
require 'recipe/cachetool.php';

desc('generate nginx config and reload');
task('nginx:update_config_and_reload', function () {
    run('echo "set \$root_dir /public;" > /root_dir.nginx.conf');
    run('sudo --non-interactive nginx -t'); // avoid reloading with incorrect config
    run('sudo --non-interactive systemctl reload nginx.service');
});

after('deploy:symlink', 'nginx:update_config_and_reload');
after('nginx:update_config_and_reload', 'cachetool:clear:opcache');

inventory('inventory.yml');
set('deploy_path','/var/www/example.com/');
set('repository', '[email protected]:my/example-com.git');
set('cachetool', '/var/run/php/phpXX-fpm.sock');
... // some other perameters

Note that deploy user should have rights to reload nginx service with sudo and to check nginx config(run nginx -t).

Removing all unnecessary bloatware from Xiaomi MIUI 11/12 (Android 9/10) without root

Tue, 25 Feb 2020 00:00:00 +0300

TLDR: If you just want the list of bloatware app package names for Xiaomi MIUI - scroll to the end of the artice.

UPDATE 2020-05-09: Added SIM Toolkit description.
UPDATE 2020-07-04 Added some apps, thanks to @thecrazyblack.
UPDATE 2020-11-23 Updated list of apps, thanks to new comments by @toast254, @swiesmann, @MarcelloDM, @satnited.

Xiaomi phones have impressive parameters for given price, but they come with a lot of unnecessary software. It eats battery and memory, sometimes shows annoying advertisement, and may have security issues. You can not uninstall this pre-installed apps like usual ones, and you can not even disable them from settings like in earlier Android versions.

Here is how to remove or disable unnecessary software without rooting phone. Works for MIUI 11 and 12 (based on Android 9 and 10), should work for other phones for recent Android versions. Thanks to this very useful but severely under-voted stackoverflow answer.

NOTE: I do not guarantee that this instructions won’t break your phone, blow it to flaming pieces or cause a sentient machines rebellion against humanity. You were warned.

To manage phone from command-line via USB you need adb - Android Debug Bridge, part of Android platform-tools. You can download the recent one for your OS here.

If you are on Windows, you also need USB drivers for your device. You may also try Universal Adb Driver. If you are using chocolatey package manager, you can get both like that:

choco install -y adb universal-adb-drivers

⚙️ Settings - About phone(or My Device). Tap “MIUI version” multiple times. Now Developer Options are unlocked.
⚙️ Settings - Additional settings - Developer Options - [✔️] USB debugging.
Connect the phone to your computer via USB. Choose “File Transfer” mode instead of default “No data transfer”.
Open console in a directory where you unpacked platform tools.
./adb devices
Phone will prompt to add your computer’s key to allowed, agree to it.
./adb shell you have a shell on your device.

Now you need app package names, like com.miui.yellowpage for Mi Yellow Pages.

⚙️ Settings - Apps - Manage Apps. Tap on application, then tap info(ℹ️) button in the right corner. There you can see “APK name”, that’s what we need.

There are 2 options: disable app and uninstall app. I prefer disabling them, it’s easier to enable them back if you’ve broken something.

# Disable app
pm disable-user app.package.name
# Re-enable it
pm enable app.package.name

# Uninstall app
# Sometimes uninstall command may not work without -k option on un-rooted devices
# -k: Keep the data and cache directories around after package removal. 
pm uninstall --user 0 app.package.name
# Install uninstalled system app
pm install --user 0 $(pm dump app.package.name | awk '/path/{ print $2 }')
# Another way to install uninstalled system app
pm install-existing app.package.name

More details here: pm commad manual.

To be able to install apps back, you need to enable

⚙️ Settings - Additional settings - Developer Options - [✔️] Install via USB

On Xiaomi phone to enable this setting you need to sign in into Mi Account. You may just use your Google account to sign into it and then sign-out when you don’t need it anymore:

⚙️ Settings - Mi Account - sign-out.

Here is a list of Xiaomi and Google apps that I find unnecessary:

UNSAFE TO DISABLE/UNINSTALL

`com.xiaomi.finddevice`	Result: endless boot loop, some time after it will ask to erase the phone and start over. Guess how I learned that?
`com.miui.securitycenter`	Result: phone reboots only in recovery mode
`com.android.contacts`	Result: you lose the phone icon
`com.mi.android.globalminusscreen`	Xiaomi App Vault. Result: if you are logged in with Mi account, device becomes locked, to unlock you should bring it to Xiaomi Services Center. Settings -> Home Screen crashes Settings app. See comment by @satnited.

Xiaomi:

`com.miui.analytics`	Mi Analytics (Spyware?)
`com.xiaomi.mipicks`	GetApps - app store like Google Play from Xiaomi. The most annoying one, periodically shows advertisement.
`com.miui.msa.global`	MIUI Ad Services - also responsible for showing ads.
`com.miui.cloudservice` `com.miui.cloudservice.sysbase` `com.miui.newmidrive`	Mi Cloud
`com.miui.cloudbackup`	Mi Cloud Backup
`com.miui.backup`	MIUI Backup
`com.xiaomi.glgm`	Games
`com.xiaomi.payment` `com.mipay.wallet.in`	Mi Credit
`com.tencent.soter.soterserver`	Authorize payments in WeChat and other Chinese services, useless if you don’t live in China.
`cn.wps.xiaomi.abroad.lite`	Mi DocViewer(Powered by WPS Office)
`com.miui.videoplayer`	Mi Video
`com.miui.player`	Mi Music
`com.mi.globalbrowser`	Mi Browser
`com.xiaomi.midrop`	Mi ShareMe
`com.miui.yellowpage`	Mi YellowPages. Some kind of caller id app.
`com.miui.gallery`	MIUI Gallery - if you use another gallery app WARNING: @nihalanand697 reports disabling it isn’t safe. But I had no problems after uninstalling it.
`com.miui.android.fashiongallery`	Wallpaper Carousel
`com.miui.bugreport` `com.miui.miservice`	Mi Bug Report - if you not using this feature
`com.miui.weather2`	MIUI Weather. I prefer another weather app.
`com.miui.hybrid` `com.miui.hybrid.accessory`	Quick apps.
`com.miui.global.packageinstaller`	MIUI package installer. Without it Play Store app will be used. It shows ads, but I like that you can manage app permissions before starting it.
`com.xiaomi.joyose`	?? Some junk

Google:

`com.google.android.gms.location.history`	Location History
`com.google.android.videos`	Google Movies
`com.google.android.music`	Google Music
`com.google.android.apps.photos`	Google Photos
`com.google.android.youtube`	Youtube - I prefer to use a browser
`com.google.android.apps.tachyon`	Google Duo - video calling app
`com.google.ar.lens`	Google Lens - identify things on camera
`com.google.android.googlequicksearchbox`	Google search box - I prever to use a browser or widget
`com.google.android.apps.wellbeing`	Digital wellbeing
`com.google.android.apps.googleassistant`	Google Assistant

Facebook:

What the @#$%? I just got a fresh phone, didn’t install any Facebook apps and I still have a bunch of Facebook services eating my battery and memory.

`com.facebook.katana`	Facebook mobile app
`com.facebook.services`	Facebook Services
`com.facebook.system`	Facebook App Installer
`com.facebook.appmanager`	Facebook app manager

Default Android Apps

`com.android.browser`	Default Browser - not necessary if you use Firefox or Chrome
`com.android.wallpaper.livepicker`	Wallpaper live picker
`com.android.dreams.basic`	Screen saver
`com.android.dreams.phototable`	Screen saver
`com.android.providers.downloads.ui`	Downloads UI. Periodically show notifications about files you downloaded. I find no use for it.

Miscelanneous Promoted Apps

`com.netflix.partner.activation`	Some Netflix stuff
`com.zhiliaoapp.musically`	TikTok
`ru.yandex.searchplugin`	Yandex Search
`com.yandex.zen`	Yandex Zen
`com.ebay.mobile` `com.ebay.carrier`	Ebay Store
`ru.ozon.app.android`	Ozon Store
`com.alibaba.aliexpresshd`	Aliexpress Store
`sg.bigo.live`	Some social media
`ru.auto.ara`	auto.ru app

And some additional steps to disable Xiaomi ads and collecting data:

⚙️ Settings - Passwords & Security - Authorization & revocation. Revoke authorization from msa(MIUI System Ads) application. Not necessary if you already disabled com.miui.msa.global.
⚙️ Settings - Passwords & Security - Privacy. Disable “User Experience Program” and “Send diagnostic data automatically”.
⚙️ Settings - Passwords & Security - Privacy - Ad services. Disable “Personalized ad recommendations”.

SIM Toolkit

Some Russian mobile operators use SIM card built-in application to promote paid services. They show pop-up windows with OK/Cancel buttons. Hit the wrong button - and you are suddenly subscribed to some freaking SMS horoscope with daily fee.

`com.android.stk`	SIM Toolkit
`com.android.stk2`	SIM Toolkit for the second SIM card

Note, that some mobile operators may use SIM toolkit for useful things, and you will lose that functions. In my experience I have never seen anything useful there.

Another good articles on de-bloating MIUI: