Assaf Morami

Hosting your own PPA repository on GitHub

2019-05-02T00:00:00+00:00

Publishing your own Debian packages and hosting it on a GitHub repo is pretty easy. This is a quick HowTo.

A PPA repo can be as simple as one directory

.
└── my_ppa
    ├── my_list_file.list
    ├── InRelease
    ├── KEY.gpg
    ├── Packages
    ├── Packages.gz
    ├── Release
    ├── Release.gpg
    ├── package-a_0.0.1_amd64.deb
    ├── package-a_0.0.2_amd64.deb
    ├── package-b_0.1.0_amd64.deb
    ├── package-b_0.1.1_amd64.deb
    ├── ...
    └── package-z_1.0.0_amd64.deb

A working example can be found in https://github.com/assafmo/ppa.

You can name my_ppa and my_list_file.list whatever you like. I used those names because it’s hard to name things.

Also don’t forget to replace ${GITHUB_USERNAME} with your GitHub user name and ${EMAIL} with your email address.

0. Creating a GitHub repo with your deb packages

Create a GitHub repo. We’ll call it my_ppa. Then go to https://github.com/${GITHUB_USERNAME}/my_ppa/settings, and under GitHub Pages select Source to be master branch.

Any HTTP server will work just fine, but GitHub pages is free, easy and fast.

Now clone the repo and put all of your debian packages inside:

git clone "[email protected]:${GITHUB_USERNAME}/my_ppa.git"
cd my_ppa
cp /path/to/my/package-a_0.0.1_amd64.deb .

1. Creating a GPG key

Install gpg and create a new key:

sudo apt install gnupg
gpg --full-gen-key

Use RSA:

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1

RSA with 4096 bits:

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (3072) 4096

Key should be valid forever:

Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

Enter your name and email:

Real name: My Name
Email address: ${EMAIL}
Comment:
You selected this USER-ID:
"My Name <[email protected]>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O

At this point the gpg command will start to create your key and will ask for a passphrase for extra protection. I like to leave it blank so when I sign things with my key it won’t promp for the passphrase each time.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key B58FBB4C23247554 marked as ultimately trusted
gpg: directory '/home/assafmo/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/assafmo/.gnupg/openpgp-revocs.d/31EE74534094184D9964EF82B58FBB4C23247554.rev'
public and secret key created and signed.

pub rsa4096 2019-05-01 [SC]
31EE74534094184D9964EF82B58FBB4C23247554
uid My Name <[email protected]>
sub rsa4096 2019-05-01 [E]

You can backup your private key using:

gpg --export-secret-keys "${EMAIL}" > my-private-key.asc

And import it using:

gpg --import my-private-key.asc

2. Creating the `KEY.gpg` file

Create the ASCII public key file KEY.gpg inside the git repo my_ppa:

gpg --armor --export "${EMAIL}" > /path/to/my_ppa/KEY.gpg

Note: The private key is referenced by the email address you entered in the previous step.

3. Creating the `Packages` and `Packages.gz` files

Inside the git repo my_ppa:

dpkg-scanpackages --multiversion . > Packages
gzip -k -f Packages

4. Creating the `Release`, `Release.gpg` and `InRelease` files

Inside the git repo my_ppa:

apt-ftparchive release . > Release
gpg --default-key "${EMAIL}" -abs -o - Release > Release.gpg
gpg --default-key "${EMAIL}" --clearsign -o - Release > InRelease

5. Creating the `my_list_file.list` file

Inside the git repo my_ppa:

echo "deb [signed-by=/etc/apt/trusted.gpg.d/my_ppa.gpg] https://${GITHUB_USERNAME}.github.io/my_ppa ./" > my_list_file.list

This file will be installed later on in the user’s /etc/apt/sources.list.d/ directory. This tells apt to look for updates from your PPA in https://${GITHUB_USERNAME}.github.io/my_ppa.

That’s it!

Commit and push to GitHub and your PPA is ready to go:

git add -A
git commit -m "my ppa repo is now hosted on github"
git push -u origin master

Now you can tell all your friends and users to install your PPA this way:

curl -s --compressed "https://${GITHUB_USERNAME}.github.io/my_ppa/KEY.gpg" | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/my_ppa.gpg >/dev/null
sudo curl -s --compressed -o /etc/apt/sources.list.d/my_list_file.list "https://${GITHUB_USERNAME}.github.io/my_ppa/my_list_file.list"
sudo apt update

Then they can install your packages:

sudo apt install package-a package-b package-z

Whenever you publish a new version for an existing package your users will get it just like any other update.

How to add new packages

Just put your new .deb files inside the git repo my_ppa and execute:

# Packages & Packages.gz
dpkg-scanpackages --multiversion . > Packages
gzip -k -f Packages

# Release, Release.gpg & InRelease
apt-ftparchive release . > Release
gpg --default-key "${EMAIL}" -abs -o - Release > Release.gpg
gpg --default-key "${EMAIL}" --clearsign -o - Release > InRelease

# Commit & push
git add -A
git commit -m update
git push

Sources

Some more progress and ETA tricks

2018-06-16T00:00:00+00:00

In an earlier post I wrote about pv -d. Here are some more tricks that can be even easier to use.

Reading to stdout with pv instead of cat

Reading files to stdout can be done with pv instead of cat. pv will also print transfer rate, ETA and a progress bar to stderr.

For example, counting packets in a pcap file:

➜  ls -lh
total 1.0G
-rw-rw-r-- 1 assafmo assafmo 1.0G Jun 10 20:51 my.pcap
➜  pv my.pcap | tcpdump -r /dev/stdin -qn | wc -l
reading from file /dev/stdin, link-type EN10MB (Ethernet)
512MiB 0:00:10 [26.3MiB/s] [================>                 ] 50% ETA 0:00:09

Result:

➜  pv my.pcap | tcpdump -r /dev/stdin -qn | wc -l
reading from file /dev/stdin, link-type EN10MB (Ethernet)
1023MiB 0:00:19 [44.8MiB/s] [================================>] 100%
8635943

Using `pv -s` in between pipes

Placing pv in between pipes will make it read from stdin and print to stdout (just like cat), and also print the transfer rate to stderr.

By using the -s flag we can tell pv the total size of the data being transferred, thus pv will also print progress and ETA to stderr.

For example, merging multiple pcap files together:

➜  ls -lh
total 3.0G
-rw-rw-r-- 1 assafmo assafmo 1.0G Jun 10 20:51 1.pcap
-rw-rw-r-- 1 assafmo assafmo 1.0G Jun 10 20:51 2.pcap
-rw-rw-r-- 1 assafmo assafmo 1.0G Jun 10 20:51 my.pcap
➜  joincap *.pcap | pv -s $(du -bc *pcap | tail -1 | cut -f 1) > merged.pcap
1.16GiB 0:00:04 [ 275MiB/s] [============>                    ] 38% ETA 0:00:06

Note: du -bc *pcap | tail -1 | cut -f 1 gives us an approximation for the size of merged.pcap. This is good enough, as pv -s won’t crash if the data is bigger or smaller, it’ll just be wrong (exit before or go over 100%).

Using `parallel --bar` to count done jobs

Finally, GNU parallel can print how many jobs are done and an ETA with the --bar flag.

For example, counting packets in multiple pcaps:

➜  ls *pcap | parallel --bar 'tcpdump -r "{}" -qn | wc -l > "{}".count'
66% 2:1=1s my.pcap

Mounting partitions from a dd image

2018-05-09T00:00:00+00:00

UPDATE: Some users on HN [1] and reddit [2][3][4] pointed out that the fdisk method is outdated. The same result can be achieved without using offsets with:

DEV=$(sudo losetup --partscan --find --read-only --show x.dd)
mkdir partition_1 partition_2
sudo mount -t ntfs -o ro "${DEV}p1" partition_1
sudo mount -t ntfs -o ro "${DEV}p2" partition_2

The old way - using fdisk

When x.dd is a disk clone created with one of the dd commands, first we’ll try to detect partitions:

$ fdisk -l x.dd
Disk x.dd: 94.1 MiB, 98705408 bytes, 192784 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xfc23b344

Device Boot Start    End Sectors  Size Id Type
x.dd1          63  96389   96327   47M  7 HPFS/NTFS/exFAT
x.dd2       96390 192779   96390 47.1M  7 HPFS/NTFS/exFAT

This means that our x.dd image has two partitions, both NTFS. The offset of the first partition is 63 units, and the offset of the second partitions is 96,390 units. Each unit is of size 512 bytes.

In order to mount those partitions, we’ll use the mount command with the offset option:

mkdir partition_1 partition_2
mount -t ntfs -o ro,offset=$((512*63)) x.dd partition_1
mount -t ntfs -o ro,offset=$((512*96390)) x.dd partition_2

Working offline with Yarn

2018-04-11T00:00:00+00:00

NPM is not very friendly when working with Node.js in an offline environment. Multiple times I found myself npm installing on an internet machine just to copy node_modules to the offline environment and commit it with the entire project.

For a while npmbox worked for me, but issues like trying to reach the internet would randomly appear and cripple my workflow.

I also tried Sinopia, but couldn’t consistently publish new or updated packages to it.

Enter Yarn

With Yarn I’m able to consistently install packages in an offline environment. Their original blog post is helpful, but I met some edge case it doesn’t cover.

This is my process for using Yarn in an offline environment.

Configuring `yarn-offline-mirror`

On the internet machine:

yarn config set yarn-offline-mirror ~/yarn-offline-mirror/

On the offline machine:

yarn config set yarn-offline-mirror ~/yarn-offline-mirror/

Note: On the offline machine ~/yarn-offline-mirror/ can also be a shared folder or a git repository.

Creating a new project

On the internet machine:

mkdir new-project/
cd new-project/
yarn add [email protected] [dep2...]

Then copy new-project/yarn.lock, new-project/package.json and ~/yarn-offline-mirror/ to the offline machine.

(rm -rf new-project/ is ok now.)

On the offline machine:

mkdir new-project/
cp /path/to/imported/{yarn.lock,package.json} new-project/
cp -n /path/to/imported/yarn-offline-mirror/* ~/yarn-offline-mirror/
cd new-project/
yarn --offline

Adding packages to an existing project

On the internet machine:

mkdir new-packages/
cd new-packages/
yarn add [email protected] [dep2...]

Then copy new-packages/yarn.lock, new-packages/package.json and ~/yarn-offline-mirror/ to the offline machine.

On the offline machine:

Append the imported yarn.lock to the existing yarn.lock. I found that without this step Yarn sometimes fails to find the new packages in the offline cache:
```
cat /path/to/imported/yarn.lock >> existing-project/yarn.lock
```

Update package.json with the new dependencies. This means merging both dependencies fields together. An ugly one-liner I tend to use:

cat existing-project/package.json <(cat existing-project/package.json /path/to/imported/package.json |
        jq '.dependencies' |
        jq -s 'add | {dependencies: .}') |
    jq -s add |
    sponge existing-project/package.json

Update yarn-offline-mirror:

cp -n /path/to/imported/yarn-offline-mirror/* ~/yarn-offline-mirror/

Install the new packages. This step also fixes existing-project/yarn.lock after we messed with it in step 1.
```
cd existing-project/
yarn --offline
```

I found the if I skip steps 1 and 2, and in step 4 I do yarn add --offline <dep1> [<dep2>...] then Yarn might not find the new packages in the cache and fail. This bug still exists in version 1.5.1. I believe it is related to these GitHub issues: [1][2][3][4][5][6].

Installing packages globally

Yarn discourages using global packages, so it’s hard by design to install them.

Find out where is the global installation location [7]:
```
yarn global bin
```
(Or set it with yarn config set prefix <file_path>)

Add it to your path. E.g.:

echo 'export PATH=$PATH:'"$(yarn global bin)" >> ~/.bashrc
source ~/.bashrc # reload

Similar to Creating a new project, but with a few subtle differences:

On the internet machine:
```
mkdir new-cli/
cd new-cli/
yarn add [email protected] [cli2...]
```
Then copy new-cli/yarn.lock and ~/yarn-offline-mirror/ to the offline machine.

(rm -rf new-cli/ is ok now.)

On the offline machine:
```
cp /path/to/imported/yarn.lock .
cp -n /path/to/imported/yarn-offline-mirror/* ~/yarn-offline-mirror/
yarn global add --offline [email protected] [cli2...]
rm -f ./yarn.lock
```
Note: In this context we don’t care about package.json. We only need to make sure that Yarn can find yarn.lock in the current directory and that ~/yarn-offline-mirror/ has the needed dependencies.

This is not a silver bullet

There are probably more edge cases I still haven’t met. I wish the docs around yarn.lock were better, Because playing with this file solved most of my problems.

For example when adding packages to an existing project, I still don’t understand why appending the new lock file to the existing one solves the resolution problem. According to the docs this step is unnecessary. And sometimes it is, but some other times it isn’t.

Overall I’m very satisfied with Yarn and I have a feeling it is only going to get better.

Better progress and ETA using the pv command

2017-08-02T00:00:00+00:00

Sometimes we need to run a task that takes a lot of time to finish. Since version 1.5.1, the pv command [1][2] has this neat option pv -d $PID that will monitor open file descriptors of the process $PID.

The pv command can detect when a process is reading a file from beginning to end, and it will kindly print transfer rate, ETA and a progress bar.

For example, We have a large CSV file that we want to load into a SQLite table.

➜  ~ ls -lh my.csv
-rw-r--r--  1 assafmo assafmo  16G Aug  2 08:41 my.csv
➜  ~ sqlite3 my.db
SQLite version 3.16.2 2017-01-06 16:32:41
Enter ".help" for usage hints.
sqlite> .mode csv
sqlite> .import ./my.csv my_table

This process can take a lot of time and we get no indication from sqlite3 when it will finish.

We need to find out what process preforms the reading and get its PID. This can be cat, sed, awk and many others, but in our case it’s sqlite3.

Using pv -d $(pidof sqlite3) gives us a nice progress bar and ETA for sqlite3 reading our file.

➜  ~ pv -d $(pidof sqlite3)
/home/assafmo/my.db:    0 B 0:00:07 [   0 B/s] [<=>          ]
/home/assafmo/my.csv: 152MiB 0:00:07 [1.96MiB/s] [>     ]  0% ETA 2:14:37
/home/assafmo/my.db-journal:    0 B 0:00:07 [   0 B/s] [<=>      ]

Similarly, we can get progress and ETA when copying files with rsync -P -h instead of cp.

Assaf Morami

Hosting your own PPA repository on GitHub

A PPA repo can be as simple as one directory

0. Creating a GitHub repo with your deb packages

1. Creating a GPG key

2. Creating the KEY.gpg file

3. Creating the Packages and Packages.gz files

4. Creating the Release, Release.gpg and InRelease files

5. Creating the my_list_file.list file

That’s it!

How to add new packages

Sources

Some more progress and ETA tricks

Reading to stdout with pv instead of cat

Using pv -s in between pipes

Using parallel --bar to count done jobs

Mounting partitions from a dd image

The old way - using fdisk

Working offline with Yarn

Enter Yarn

Configuring yarn-offline-mirror

On the internet machine:

On the offline machine:

Creating a new project

On the internet machine:

On the offline machine:

Adding packages to an existing project

On the internet machine:

On the offline machine:

Installing packages globally

On the internet machine:

On the offline machine:

This is not a silver bullet

Better progress and ETA using the pv command

2. Creating the `KEY.gpg` file

3. Creating the `Packages` and `Packages.gz` files

4. Creating the `Release`, `Release.gpg` and `InRelease` files

5. Creating the `my_list_file.list` file

Using `pv -s` in between pipes

Using `parallel --bar` to count done jobs

Configuring `yarn-offline-mirror`