Dev Notes - Articles

Branch Management With Git Worktree

Mon, 27 Jun 2022 00:00:00 +0100

If you need to frequently switch between Git branches, the worktree feature can be a useful way of working since it allows you to check out more than one branch at a time.

It avoids the necessity of either committing or stashing changes before switching to a different branch because each branch is managed in a new working tree - i.e. worktree branches exist as a separate directory trees on your filesytem.

You can add a new worktree at any path - I find it convenient to add .worktrees to a global .gitignore and add worktree branches to a .wortrees subdirectory of the main project root.

Why Bother?

You’re deep in refactoring on a feature branch and you need to review a co-worker’s PR for a hotfix urgently.

You could add & commit your changes, but they are currently not in a satisfactory state and you are aiming to keep the project commit history as clean as possible. You could stash changes, change to the PR branch, and pop the stash when finished, but this is messy and can be confusing if you’re managing multiple stashes.

Adding the PR as a new worktree branch allows you to simply change directory into the new PR branch without worrying about disturbing complex ongoing changes in the branch you’re working on.

Basic Example: Add New Branch as a Worktree

git worktree add <path>

To add a hotfix worktree, from the project root:

git worktree add .worktrees/hotfix

This creates a new branch named hotfix - the final component of .worktrees/hotfix - and checks it out at the .worktrees/hotfix path.

Add a second worktree branch:

git worktree add .worktrees/feature-1

The project file structure now looks like this:

.
├── README.md
└── .worktrees
    ├── feature-1
    │   └── README.md
    └── hotfix
        └── README.md

The top level README.md is in the original branch.

At this point, running git branch from the top level of the project shows:

  dev
+ feature-1
+ hotfix
* master

This shows a branch dev that is not a worktree, along with two worktree branches feature-1 and hotfix marked with the + symbol.

The current checked-out branch master is marked with a *.

If you move into hotfix - cd .worktrees/hotfix, git branch will now show:

  dev
+ feature-1
* hotfix
+ master

In each case, the current branch is colorised and marked with an asterisk. The plus symbol indicates that you can access/work on the branch by moving in to the relevant directory.

In the example above, the dev branch does not have a linked worktree.

Work on Existing Branch in a New Worktree

To set up an existing branch as a worktree, run git worktree add <path> <branch>. For example:

git worktree add .worktrees/my-existing-branch my-existing-branch

A new working tree is created at .worktrees/my-existing-branch and my-existing-branch is checked out at this location.

Remove Worktrees

Run git worktree remove worktree-name - where worktree-name is the final component of the worktree path. If you’re unsure of the worktree name, git worktree list will list worktrees in the current repo:

git worktree list
/tmp/tmp.e4JD71KDKU                       0f4b170 [master]
/tmp/tmp.e4JD71KDKU/.worktrees/feature-1  3471bba [feature-1]
/tmp/tmp.e4JD71KDKU/.worktrees/hotfix     565c127 [hotfix]

To remove the hotfix worktree:

git worktree remove hotfix

The hotfix worktree is now removed, but the branch is still available should you still need it.

If you just delete the worktree directory without using git worktree remove, don’t panic - Git has you covered. Garbage collection will eventually remove the metadata files for the (now non-existent) worktree.

Working on Branches

To work on a worktree branch, move into the correct directory. Make changes as required and use git add and git commit as usual. When you’re finished, simply return to the original directory.

Fetch a PR GitHub and Set Up as a Worktree

Save this script as an executable file named gitfetch-pr in your PATH:

#!/usr/bin/env bash
# /path/gitfetch-pr
set -euo pipefail

function usage {
	cat <<- EOF
	Provide the PR reference number from within a Git project:
	$(basename $0) <PR reference number> 
	EOF
}

[[ $# != 1 ]] && { usage; exit 1; }
[[ -d "$PWD/.git" ]] || { echo "No .git directory detected."; usage; exit 2; }
[[ "${1}" =~ ^[0-9]+$ ]] || { echo "The PR reference should be an integer."; usage; exit 3; }

git fetch origin "pull/${1}/head:PR${1}"
echo "Done."

You can now run gitfetch-pr 42 to fetch the commit associated with pull request #42 and set this up as a branch PR42.

To set this up as a worktree, run:

git worktree add .worktrees/PR42 PR42

You can view the PR42 commit by navigating to .worktrees/PR42.

You could also set this up as a git alias rather than a shell script, as demonstrated in the following example.

Fetch & Checkout a GitLab Merge Request

This can be achieved with a git alias. Add the following to ~/.gitconfig:

[alias]
	# Allows: `git mr origin 5` to fetch & checkout MR 5 from origin
	mr = !sh -c 'git fetch $1 merge-requests/$2/head:mr-$1-$2 && git checkout mr-$1-$2' -

To use, run git mr origin 42 fetches the specified merge request into a local mr-upstream-42 branch.

To use this as a worktree:

git worktree add .worktrees/mr42 mr-origin-42

This adds the branch mr-origin-42 to the worktree directory .worktrees/mr42.

Using Standard Branch Comparison

You can still compare across branches and pull specific hunks as you would usually do. For example, if working in the hotfix worktree you can view (and diffput/diffget) differences between the current file in the current branch and the same file in the feature-1 branch using vim fugitive:

:Gvdiff feature-1:%

Alternatively, open a split window in your editor, open the same file in a different worktree and pull changes using the editor’s fucntionality.

References

Git Worktree Documentation

Cargo Add for Dependency Management

Sat, 11 Jun 2022 00:00:00 +0100

The cargo add [OPTIONS] <DEP> is a useful command that adds dependencies to a Cargo.toml manifest file.

To install:

cargo install cargo-edit

Usage

In the root of your project run:

cargo add serde

This will add the dependency and update your Cargo.toml file.

For full instructions, run cargo add --help.

Git Setup

If you have overridden your access to a remote git service to use SSH rather than HTTPS, you may need to tweak your ~/.gitconfig.

The problem:

cargo add serde                
    Updating 'https://github.com/rust-lang/crates.io-index' index
Error: failed to acquire username/password from local configuration                                                                                                                                                                                                                                                          
                                                                               
Caused by:                                    
    failed to acquire username/password from local configuration

The solution - add this to ~/.gitconfig:

# existing github.com override
[url "git@github.com:"]
	insteadOf = https://github.com/

# fine-tune the override to avoid issues with cargo-edit
[url "https://github.com/rust-lang/crates.io-index"]
	insteadOf = https://github.com/rust-lang/crates.io-index

Rules of Borrowing in Rust

Fri, 15 Apr 2022 00:00:00 +0100

This is a personal note/reminder on borrowing rules in Rust.

Rules for Variables

Each value has a variable which is it’s owner.
There can only be one owner of a value at a time.
When the owner goes out of scope, the value is dropped.

Rules of Borrowing

Unlimited borrows for read-only borrows: let a = &x
For a read only borrow, the original data is immutable for the duration of the borrow
You can only have a single borrow at a time for mutable borrows: let a = &mut x

Example

/// Notes ownership and borrowing.
fn main() -> Result<(), &'static str> {
    let mut a = [1,2,3,4];
    println!("{:?}", a);
    {
        let b = &mut a[0..2];
        // You can't access a at this point because it has been mutably borrowed
        // from. The following line won't compile, with the error message:
        // `cannot borrow `a` as immutable because it is also borrowed as mutable`:
        // println!("a: {:?}", a);
        println!("b: {:?}", b); // b: [1, 2]
        b[0] = 42;
        println!("b: {:?}", b); // b: [42, 2]
    }
    // a is accessible again:
    println!("a: {:?}", a); // a: [42, 2, 3, 4]
    Ok(())
}

Add Methods To An Existing Type in Go

Sun, 20 Jun 2021 00:00:00 +0100

Code re-use in object-oriented Go is achieved by means of composition rather than inheritance.

Composition means that an object contains another object (“has a …”) rather than extending another object(“is a …”).

Instead of overriding functions, function calls are delegated to the internal object.

Add Method to An Existing Type

You can modify a type locally by type definition or embedding.

Type Definition

Create an alias to the external type.

Provides direct access to fields of the external type
DOES NOT provide access to methods of the external type


// Parent is a type defined in package-name with a field FamilyName and a method GetFamilyName
type child package-name.Parent

name := child.FamilyName // OK

name := child.GetFamilyName() // Won't work unless the method is defined specifically on child

Embedding

Embedding is a Go feature that eases type composition.

If a type A is included as a nameless parameter in type B, then the methods defined on type A are accessible through type B.

The compiler uses the concept of promotion to export properties & methods of the embedded type to the embedding type.

If a type contains a field of another type, but no name is assigned to that field, it is an embedded field.

Fields or methods on the embedded field are promoted to the containing struct.

If types are listed within a struct without giving them field names, they are embedded. This promotes the methods of the embedded types - they are available to the containing type. The local type then:

Has access to all fields of the parent
Has access to methods of the parent - they are promoted to the local type
If the embedded type is a pointer, it must be initialised separately
You can’t embed a non-struct type within a struct and get access to the parent methods

References

Use Delve to Run and Debug a Single Unit Test in Go

Mon, 19 Apr 2021 00:00:00 +0100

If you want to run a single unit test in the Delve debugger for Go, here’s how you do it on the command line:

dlv test github.com/path/to/package/testsuite -- -test.run ^Test_Random$

The command is run from within path/to/package/testsuite - the start point for the debugger will be the TestRandom function.

Once the debugger builds the test binary and launches, you can set breakpoints relative to the start of the function. For example:

b TestRandom:5 sets a breakpoint on the 5th line of the function.

I’m fairly new to Go, and debugging from witin the go-vim plugin allows the same functionality using the :GoDebugTestFunc command with the cursor on the required function.

So far, I prefer multiplexing with Tmux, with my code in Vim in one pane and the debugger in the other.

References

Comment on Delve Github issue #636

Virtualenv & Virtualenvwrapper Setup in Python 3 on Ubuntu Focal Fossa

Tue, 19 Jan 2021 00:00:00 +0000

This article outlines Virtualenv and Virtualenvwrapper installation on Python 3 on Ubuntu.

A virtual environment in Python is an isolated working copy of Python and associated modules that allows you to work on a specific project without fear of affecting other projects or your system as a whole.

Virtualenv

virtualenv creates a folder which holds all the necessary modules for a project.

A new virtual environment is created by:

cd project_directory
virtualenv my_project

You could also keep all virtual environments in one directory, and activate your environment by referencing this.

An environment is started by sourcing a file bin/activate located in the isolated environment. For example:

. /path_to_environments/my_project/bin/activate

This changes your prompt to reflect the virtual environment - in this case: (my_project).

Once you have used pip to install modules within the environment, you can easily reproduce the specific context by creating a requirements.txt manifest file with the pip freeze command:

pip freeze > requirements.txt

You can add this file into version control, and use it to set up a new virtualenv as required:

# Set up a new virtualenv
# Install dependencies in the requirements.txt
pip install -r requirements.txt

Install Virtualenv

Install pip3 if necessary:

sudo apt install python3-pip

Add an alias for the pip3 command. Add to .bashrc if you want to make this permanent:

alias pip='pip3'

Install Virtualenv:

pip install virtualenv

Virtualenvwrapper

virtualenvwrapper is a set of shell functions that serve as extensions tovirtualenv.

The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

Install Virtualenvwrapper

pip install virtualenvwrapper

Add the following to .bashrc or .profile:

# Next line not necessary unless you want to specify a different location - ~/.virtualenvs is the default
export WORKON_HOME=$HOME/.virtualenvs

# $PROJECT_HOME tells virtualenvwrapper where to place project working directories.
# This is required for the `mkproject` command.
export PROJECT_HOME=$HOME/Devel

# Source the script, making functions available in the shell
source /usr/local/bin/virtualenvwrapper.sh

Virtualenvwrapper Quickstart

Run workon: display a list of environments (which will be empty on first run)
Run: mkvirtualenv temp: a new environment temp is created and activated
Run: workon: the new temp environment is included
deactivate to return to the system installed Python

References

virtualenvwrapper docs

LUKS Encrypt Hard Drive with Cryptsetup on Ubuntu 20.04

Tue, 01 Dec 2020 00:00:00 +0000

This article outlines how to LUKS encrypt a secondary drive on Ubuntu 20.04 Focal Fossa using cryptsetup on the command line.

Find the Unmounted Disk

Once you have physically connected the disk, find the unmounted disk in the system using lsblk:

lsblk

# Typical output:
NAME                    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                       8:0    0 465.8G  0 disk
├─sda1                    8:1    0   512M  0 part
├─sda2                    8:2    0   488M  0 part  /boot
└─sda3                    8:3    0 464.8G  0 part
  └─sda3_crypt          252:0    0 464.8G  0 crypt
    ├─ubuntu--vg-root   252:1    0 448.8G  0 lvm   /
    └─ubuntu--vg-swap_1 252:2    0    16G  0 lvm
      └─cryptswap1      252:3    0    16G  0 crypt [SWAP]
sdb                       8:16   0 931.5G  0 disk
└─sdb_crypt             252:4    0 931.5G  0 crypt /media/secure-hdd-2
sdc                       8:32   0   1.8T  0 disk
├─sdc1                    8:33   0 994.9G  0 part  /media/datadrive
└─sdc2                    8:34   0 868.2G  0 part
  └─sdc2_crypt          252:5    0 868.2G  0 crypt /media/secure-hdd
sdd                       8:48   0 931.5G  0 disk
loop0                     7:0    0  83.8M  1 loop  /snap/core/3604
loop1                     7:1    0 178.2M  1 loop  /snap/atom/112
loop2                     7:2    0 174.2M  1 loop  /snap/atom/116
loop3                     7:3    0  28.1M  1 loop  /snap/telegram-latest/4
loop4                     7:4    0  83.8M  1 loop  /snap/core/3748
loop5                     7:5    0  81.3M  1 loop  /snap/core/3887
loop6                     7:6    0 178.2M  1 loop  /snap/atom/111

In this case you can see sdd is the target drive - the size type matches, and it is unmounted (as expected).

Double check disk info:

sudo hdparm -i /dev/sdd

# Output:

/dev/sdd:

 Model=Samsung SSD 860 EVO 1TB, FwRev=RVT01B6Q, SerialNo=S3Z9NB0JC01523A
 Config={ Fixed }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=0
 BuffType=unknown, BuffSize=unknown, MaxMultSect=1, MultSect=1
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=1953525168
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: unknown:  ATA/ATAPI-2,3,4,5,6,7

 * signifies the current active mode

If this checks out, you have the device reference.

Alternatively, list all drives in short format:

 sudo lshw -short -C disk
H/W path         Device     Class          Description
======================================================
/0/1/0.0.0       /dev/sda   disk           500GB Samsung SSD 850
/0/2/0.0.0       /dev/sdb   disk           1TB ST1000DM010-2EP1
/0/3/0.0.0       /dev/sdc   disk           2TB WDC WD20EZRZ-00Z
/0/4/0.0.0       /dev/sdd   disk           1TB Samsung SSD 860

Create a Partition

If the disk does not have an existing partition, create one.

sudo fdisk /dev/sdd

Follow instructions, hit all defaults to create one large partition…it will be /dev/sdd1.

If the disk is already partitioned, you can use an existing partition. Note that all data will be over-written.

LUKS Encryption

Next step is to LUKS encrypt the target partition - in this case, /dev/sdd1:

cryptsetup -y -v luksFormat /dev/sdd1

-v: verbose output
-y: Forces double entry from the user when interactively setting the passphrase - ask for it twice and complain if both inputs do not match

Create a Mapping

The encrypted partition is accessed by means of a mapping.

To create a mapping for the current session:

# Creates a mapping called backupSSD
sudo cryptsetup luksOpen /dev/sdd1 backupSSD

# Check:
ls -l /dev/mapper/backupSSD
lrwxrwxrwx 1 root root 7 Dec 17 15:48 /dev/mapper/backupSSD -> ../dm-6

# Check mapping status
sudo cryptsetup -v status backupSSD
[sudo] password for david:
/dev/mapper/backupSSD is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 256 bits
  device:  /dev/sdd1
  offset:  4096 sectors
  size:    1953519024 sectors
  mode:    read/write
Command successful.

This mapping is not persistent. If you want to open the disk/partition automatically on boot, you will need to amend /etc/crypttab to set up a mapped device name.

Crypttab

The file /etc/crypttab contains descriptive information about encrypted filesystems. Each filesystem is described on a separate line.

Fields on each line are separated by tabs or spaces. The order of records in crypttab is important because the init scripts sequentially iterate through crypttab doing their thing.

# /etc/crypttab
# Mapping name in field 1.
# UUID of partition in field 2
# Full path to keyfile in field 3
backupSSD UUID=4f942e15-ff00-4213-aab1-089448b17850 /root/.keyfiles/hdd-1.key luks,discard

The first field, target, describes the mapped device name. It must be a plain filename without any directory components. A mapped device which encrypts/decrypts data to/from the source device will be created at /dev/mapper/target by cryptsetup.

The second field, source device, describes either the block special device or file that contains the encrypted data. Instead of giving the source deviceexplicitly, the UUID is supported as well, using UUID=<luks_uuid>.

The third field, key file, describes the file to use as a key for decrypting the data of the source device. Note that the entire key file will be used as thepassphrase; the passphrase must not be followed by a newline character.

It can also be a device name (e.g. /dev/urandom), note however that LUKS requires a persistent key and therefore does not support random data keys.

If the key file is the string “none”, a passphrase will be read interactively from the console. In this case, the options precheck, check, checkargs and triesmay be useful.

The fourth field, options, describes the cryptsetup options associated with the encryption process. At minimum, the field should contain either the string luksrespectively tcrypt or the cipher, hash and size options.

Options are in the format: key=value [,key=value …]. The supported options are described below.Note that all four fields are mandatory and that a missing field will lead to unspecified behaviour.

–Man page: man crypttab

LUKS Header

Check it using the luksDump command. Data shown is anonymised (junk):

sudo cryptsetup luksDump /dev/sdd1

LUKS header information for /dev/sdd1

Version:       	1
Cipher name:   	aes
Cipher mode:   	xts-plain64
Hash spec:     	sha1
Payload offset:	4096
MK bits:       	256
MK digest:     	ad f8 5d b7 ce 4e 54 f6 17 79 a8 ce f5 07 12 ce 82 b1 94 28
MK salt:	cd 44 e4 6a db a0 b4 ba d1 9d c5 da 1b 97 d1 a2
		c8 a5 17 ac 7b 97 50 42 67 84 fe 74 a6 1f 5f d5               	
MK iterations: 	274632
UUID:          	2e72f987-ffa1-c0c0-d144-098f45d86100

Key Slot 0: ENABLED
	Iterations:         	978243
	Salt:               	1e b5 1f 35 7b 46 24 34 ec ab a2 7d 5a 74 ba 38
	                      	87 7e c9 57 b4 d0 2a ab b7 e1 13 32 92 65 b2 c2
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: ENABLED
	Iterations:         	4456344
	Salt:			39 68 d2 4e 04 51 a3 c2 c9 88 e8 71 a5 a9 0f d4 
	                      	17 3f 88 0c 39 2c f5 22 35 e5 19 48 3c 1b 47 49
	Key material offset:	264
	AF stripes:            	4000
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Amongst other interesting things, the output will show the size of the master key (MK bits). A 256 bit master key will equate to a LUKS header size of 1,052,672 Bytes See the cryptsetup FAQ

Format Partition

Do not omit this step - or the partition won’t mount.

sudo mkfs.ext4 /dev/mapper/backupSSD -L "Extra SSD 1TB"

…where -L denotes the partition label.

Mount Point

Create a mount point. Give the mount point appropriate ownership. E.g.:

sudo mkdir /media/secure-ssd
sudo chown $USER:$USER /media/secure-ssd

Mount

To mount, you need to reference the mapper not the partition:

sudo mount /dev/mapper/backupSSD /media/secure-ssd

Auto Mount

Mount at boot.

Edit /etc/fstab to reference the mapper to the decrypted volume

# Samsung 860 1TB EVO SSD path installed 17-02-2018
# mapper              mount-point       type mount-options dump fs_passno
/dev/mapper/backupSSD /media/secure-ssd ext4 defaults 0 2

Mount-options refer to filesystem related parameters, formatted as a comma-separated list. This contains at least the type of mount (ro or rw), plus any additional options appropriate to the filesystem type (including performance-tuning options). Basic filesystem-independent options are:

defaults - use default options: rw, suid, dev, exec, auto, nouser, and async.
noauto - do not mount when “mount -a” is given (e.g., at boot time)
user - allow a user to mount
owner - allow device owner to mount
comment - or x- for use by fstab-maintaining programs
nofail - do not report errors for this device if it does not exist.

The sixth field (fs_passno) is used by fsck to determine the order in which filesystem checks are done at boot time.

The root filesystem should be specified with a fs_passno of 1. Other filesystems should have a fs_passno of 2. Filesystems within a drive will be checked sequentially, but filesystems on different drives will be checked at the same time to utilize parallelism available in the hardware. Defaults to zero (don’t fsck) if not present.

Decrypt using a key

I set up additonal drives to mount & decrypt automatically by means of a keyfile held in my encrypted boot drive. This means that I just decrypt the primary disk at start up, with a passphrase as usual. The keyfile is then used to decrypt additional disks automatically. Keyfiles are secure since the drive holding the keyfile is encrypted.

Cryptsetup allows you to specify up to 8 keyslots - passwords or keyfiles. When you add these, they are hashed and added to key-slots in the LUKS header at the start of the device. When you first run luksFormat, the initial password you supply is hashed and stored in key-slot 0 of the LUKS header. You can easily add an additional passphrase, and this can take the form of a keyfile. This means that the volume can be decrypted either with the initial passphrase or the keyfile.

To add a password to a LUKS partition, you need an unencrypted copy of the master key - so if the partition is not initialized, you will be prompted for the original passphrase. Because more than one password is possible under LUKS setups, the wording of the password prompt may seem confusing - it says: “Enter any passphrase”. This means enter any valid existing password for the partition.

Create a keyfile:

# Make an appropriate directory
sudo mkdir /root/.keyfiles

# Make a keyfile of randomly sourced bytes
sudo dd if=/dev/urandom of=/root/.keyfiles/hdd-1.key bs=1024 count=4

# Make this read-only by owner (in this case, root):
sudo chmod 0400 /root/.keyfiles/hdd-1.key

Set up a keyfile for the LUKS partition (in this case, /dev/sdd1:

sudo cryptsetup luksAddKey /dev/sdd1 /root/.keyfiles/hdd-1.key

To automatically mount at boot, the mapping in /etc/crypttab should reference the keyfile:

# Modify line to /etc/crypttab if the keyfile has not been added
backupSSD UUID=4f942e15-ff00-4213-aab1-089448b17850 /root/.keyfiles/hdd-1.key luks,discard

Unmount & Secure

umount /dev/mapper/backupSSD
cryptsetup luksClose backupSSD

Decrypt & remount:

cryptsetup luksOpen /dev/sdd1 backupSSD
# mount /dev/mapper/backupSSD /media/mountpoint

Good image of LUKS header

Check Passphrase

sudo cryptsetup luksOpen --test-passphrase /dev/sdX && echo correct

# Prompts for password, echoes "correct" if successful

# Alternative - specify slot
cryptsetup luksOpen --test-passphrase --key-slot 0 /dev/sdX && echo correct

Backup Header Files

sudo cryptsetup luksHeaderBackup --header-backup-file ~/backup-dir/LUKS-headers/my-hostname/sda3 /dev/sda3
sudo cryptsetup luksHeaderBackup --header-backup-file ~/backup-dir/LUKS-headers/my-hostname/sdb /dev/sdb
sudo cryptsetup luksHeaderBackup --header-backup-file ~/backup-dir/LUKS-headers/my-hostname/sdc2 /dev/sdc2
sudo cryptsetup luksHeaderBackup --header-backup-file ~/backup-dir/LUKS-headers/my-hostname/sdd1 /dev/sdd1

Restore Header Files

# General form:
# cryptsetup luksHeaderRestore --header-backup-file <file> <device>

sudo cryptsetup luksHeaderRestore --header-backup-file ~/backup-dir/LUKS-headers/my-hostname/sda3 /dev/sda3
sudo cryptsetup luksHeaderRestore --header-backup-file ~/backup-dir/LUKS-headers/my-hostname/sdb /dev/sdb
sudo cryptsetup luksHeaderRestore --header-backup-file ~/backup-dir/LUKS-headers/my-hostname/sdc2 /dev/sdc2
sudo cryptsetup luksHeaderRestore --header-backup-file ~/backup-dir/LUKS-headers/my-hostname/sdd1 /dev/sdd1

Breaking LUKS Encryption

This article describes a method for breaking LUKS encryption that involves dumping the contents of memory while running cryptsetup luksOpen, and then finding the hashed password. Once an adversary has a copy of the hashed password, they can run it through a password cracker. The method relies on the fact that the hashed password is loaded into memory by cyrptsetup as part of the authentication process. I haven’t checked this vulnerability, but it certainly sounds plausible.

To prevent such an attack, use a complex password. My personal preference is for a memorised > 12 character mneumonic passsphrase drawn from a possible keyspace of 94 (upper & lowercase, number, special character), with additional padding characters to increase overall length up to 22. If you encrypt disks with LUKS and then select “password123” as your passphrase, you may as well not bother.

Appendix

lsblk is a straightforward and concise way to get all available drives & partitions. LUKS partitions/drives have the FSTYPE crypto_LUKS:

david@ubuntu:$ sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL,UUID

NAME                 FSTYPE        SIZE MOUNTPOINT              LABEL         UUID
sda                              465.8G
├─sda1               vfat          512M /boot/efi                             xxx
├─sda2               ext2          488M /boot                                 xxx
└─sda3               crypto_LUKS 464.8G                                       xxx
  └─sda3_crypt       LVM2_member 464.8G                                       xxx
    ├─ubuntu--vg-root
    │                ext4        448.8G /                                     xxx
    └─ubuntu--vg-swap_1
                     swap           16G                                       xxx
      └─cryptswap1   swap           16G [SWAP]                                xxx
sdb                  crypto_LUKS 931.5G                                       xxx
└─sdb_crypt          ext4        931.5G /media/secure-hdd-2     Barracuda 1TB xxx
sdc                                1.8T
├─sdc1               ext4        994.9G /media/datadrive        Public HD     xxx
└─sdc2               crypto_LUKS 868.2G                                       xxx
  └─sdc2_crypt       ext4        868.2G /media/secure-hdd       Secure HD     xxx
sdd                              931.5G
└─sdd1               crypto_LUKS 931.5G                                       xxx
  └─backupSSD        ext4        931.5G /media/secure-ssd       Extra SSD 1TB xxx
loop0                squashfs     83.8M /snap/core/3604
loop1                squashfs    178.2M /snap/atom/111
loop2                squashfs     83.8M /snap/core/3748
loop3                squashfs    174.2M /snap/atom/116
loop4                squashfs    178.2M /snap/atom/112
loop5                squashfs     81.3M /snap/core/3887
loop6                squashfs     28.1M /snap/telegram-latest/4

Available columns (for –output):

NAME: device name
KNAME: internal kernel device name
MAJ::MIN major:minor device number
FSTYPE: filesystem type
MOUNTPOINT: where the device is mounted
LABEL: filesystem LABEL
UUID: filesystem UUID
PARTTYPE: partition type UUID
PARTLABEL: partition LABEL
PARTUUID: partition UUID
PARTFLAGS: partition flags
RA: read-ahead of the device
RO: read-only device
RM: removable device
HOTPLUG: removable or hotplug device (usb, pcmcia, …)
MODEL: device identifier
SERIAL: disk serial number
SIZE: size of the device
STATE: state of the device
OWNER: user name
GROUP: group name
MODE: device node permissions
ALIGNMENT: alignment offset
MIN:-IO minimum I/O size
OPT:-IO optimal I/O size
PHY:-SEC physical sector size
LOG:-SEC logical sector size
ROTA: rotational device
SCHED: I/O scheduler name
RQ:-SIZE request queue size
TYPE: device type
DISC:-ALN discard alignment offset
DISC:-GRAN discard granularity
DISC:-MAX discard max bytes
DISC:-ZERO discard zeroes data
WSAME: write same max bytes
WWN: unique storage identifier
RAND: adds randomness
PKNAME: internal parent kernel device name
HCTL: Host:Channel:Target:Lun for SCSI
TRAN: device transport type
SUBSYSTEMS: de-duplicated chain of subsystems
REV: device revision
VENDOR: device vendor

References

Guide on LUKS cryptsetup
Labelling of partitions/volumes
Using fdisk to manage partitions on Linux
FAQ for cryptsetup - essential reading
Create an encrypted LUKS disk
Encrypt external drives with LUKS
Breaking LUKS encryption
Useful article from RedHat on recovering lost LUKS key/passphrase (spoiler - you probably can’t recover if the drive is encrypted and the password is strong)
Backup LUKS headers

Resources

Bitcoin Pay To Script Hash

Wed, 18 Nov 2020 00:00:00 +0000

Transaction outputs do not need to be sent to a Bitcoin public key - they can be sent to a script hash.

This process, known as Pay To Script Hash (P2SH) allows complex spending conditions to be attached to a UTXO in a way that does not increase the size of the transaction and which places the burden of complexity on the receiver.

P2SH involves:

The receiver generating a set of spending conditions - the <redeemScript>
The receiver generating a Bitcoin address by hashing the <redeemScript> and encoding this in an address format
The sender sending funds to the address, with funds locked according to the conditions set in step 1
Spending the transferred funds - the receiver must provide a valid <redeemScript> the hash of which matches that represented by the address, after which the <redeemScript> is executed as a locking script.

Intro

All transaction outputs create a cryptographic puzzle that must be solved in order to spend the output. The puzzle and the solution to the puzzle are represented by two scripts:

<scriptPubKey>: The locking script which specifies the conditions which must be met for the output to be spent
<scriptSig>: This contains a signature that unlocks the <scriptPubKey> - if the transaction is Pay To Public Key Hash, this is a cryptographic proof that the spender controls the private key associated with the address public key

In a Pay-To-Script-Hash (P2SH) transaction, the conditions to redeem the transaction are encoded in the <scriptSig> rather than in the <scriptPubKey>. The <scriptPubKey> contains the hash and opcodes necessary to check that the spenders <redeemScript> matches the original script used to generate the Bitcoin address. Once this has been checked, the <redeemScript> (present in the <scriptSig>) is permitted to execute.

In this way, the conditions required to spend the UTXO are shifted to the receiver of the funds (i.e. the input in the spending transaction).

This has several benefits:

The sender can send without concerning themselves with the spending conditions specified by the receiver - the sender focuses solely on transferring funds to the receiver
For multi-signature transactions, funds are sent without exposing the public keys of the recipient(s) - this increases privacy since these public keys are not exposed until the recipient spends the funds
The receiving address is short compared to a full redeemScript - typically a 34 character base58Check encoded address
UTXOs do not contain long locking scripts - the burden is shifted to transaction inputs which are stored in the blockchain, reducing the memory footprint of the UTXO set

The purpose of pay-to-script-hash is to move the responsibility for supplying the conditions to redeem a transaction from the sender of the funds to the redeemer.

The benefit is allowing a sender to fund any arbitrary transaction, no matter how complicated, using a fixed-length 20-byte hash that is short enough to scan from a QR code or easily copied and pasted. — BIP 0016

Locking Steps

The receiver decides the spending conditions for the funds that they will receive. These conditions are bounded by what is possible in a regular Bitcoin script.

Create a multi-sig script. For example, 2 of 3 multisig:

OP_2 <pubKey 1> <pubKey2> <pubKey3> OP_3 OP_CHECKMULTISIG

This is the <redeemScript>.

The hash of the <redeemScript> is computed by SHA256 hashing the serialized byte format, followed by RIPEMD160 hashing the result.

The Address

The receiver creates an address by converting the hash of the <redeemScript> into a base 58 Bitcoin address by Base58Check. From the serialized byte format of the script the process is as follows:

SHA256 hash the serialized bytes of the <redeemScript>
RIPEMD160 hash the result
Prepend a version byte: 0x05 for mainnet, 0xc4 for testnet/regtest
Create a checksum by double SHA256 hashing the RIPEMD160 hash, and appending the first four bytes of this value to the RIPEMD160 hash
Encode the RIPEMD160 + checksum hash into base 58.

The scriptPubKey of the transaction output will be:

OP_HASH160 <redeemScriptHash> OP_EQUAL

Worked Example: 2 of 3 Multisig as a P2SH

Create a 2-of-3 Multisignature address and send coins to the address.

Step 1: Construct the Multisig Script

The Multisig script requires the public keys that will be used to lock the output, as well as the total number of public keys used ($n$) and the threshold number of keys required for the funds to be unlocked ($m$).

To create the multisig:

# Make an array of public keys to be used - not strictly necessary, but handling keys
# in this way makes the command much cleaner:
pubkeys[0]=02c3eb5147ead06273e48b71fb2ce3e0f1484b3222c75b2039beb720c480ecb00c # first
pubkeys[1]=02b6d8bd0cd403f488df6dccc5281c61029052a31d6ce57a0d6dcb64890a4ea814 # second
pubkeys[2]=0234965dbaff057b7a1fd08b8e40b3117c23abaa05345ab3cedf129c0c50b17e5f # third

# Create a multisig redeem script using these keys. The jq utility is used to pass the full array pubkeys
# into the command in the required JSON format:
bitcoin-cli -regtest createmultisig 2 $(jq -nc '$ARGS.positional' --args "${pubkeys[@]}")

# If you'd rather not use jq, escape the quotations within the JSON array:
bitcoin-cli -regtest createmultisig 2 "[\"${pubkeys[0]}\", \"${pubkeys[1]}\", \"${pubkeys[2]}\"]"

# These commands are effectively identical. The result is:
{
  "address": "2N2Zxm7Kf3v4DKMSWW3zYz5Szi8tmGCvtKx",
  "redeemScript": "522102c3eb5147ead06273e48b71fb2ce3e0f1484b3222c75b2039beb720c480ecb00c2102b6d8bd0cd403f488df6dccc5281c61029052a31d6ce57a0d6dcb64890a4ea814210234965dbaff057b7a1fd08b8e40b3117c23abaa05345ab3cedf129c0c50b17e5f53ae",
  "descriptor": "sh(multi(2,02c3eb5147ead06273e48b71fb2ce3e0f1484b3222c75b2039beb720c480ecb00c,02b6d8bd0cd403f488df6dccc5281c61029052a31d6ce57a0d6dcb64890a4ea814,0234965dbaff057b7a1fd08b8e40b3117c23abaa05345ab3cedf129c0c50b17e5f))#q6zdklmm"
}

The redeemScript is constructed by:

Setting the first byte to an opcode that pushes the number $m$ (the threshold) onto the stack - in this case 0x52 (an instruction to “push the number 2 onto the stack”)
The next byte is in the range 0x01-0x4b, which denotes that this value is the next number of bytes to be pushed to the stack
- In the case of a multisig redeem script, this byte is 0x21, which denotes that the following 33 bytes are to be pushed onto the stack
The next bytes to be added are the first public key
Steps 2 and 3 are repeated until all public keys have been added to the script
After the final key, the next byte is an opcode that pushes the number $n$ (the total number of public keys) onto the stack - in this case, 0x53 (an instruction to “push the number 3 onto the stack”)
Finally the opcode OP_CHECKMULTISIG is added - the value 0xae

The address for the multisig is computed by:

SHA256 hash the input bytes - the input bytes are the bytes of the entire redeem script as described above
RIPEMD160 the result of step 1
Prepend the network byte to the hash
Perform SHA256 twice on the concatenated (network byte || RIPEMD160 hash) - the checksum is the first four bytes of the result.
Append the 4 byte checksum to the concatenated (network byte || RIPEMD160 hash), resulting in 25 bytes
Base58 encode the result

Storage in Chainstate DB: UTXO

The scriptPubKey that is stored in the chainstate database is the value of the hashed redeemScript:

66445308b05d3eec5aab83dceb5cc87a759bd7d5

Because we know the script type, the scriptPubKey can be assembled by adding the appropriate opcodes:

OP_HASH160  Push next 0x14 bytes    data                                        OP_EQUAL    
a9          14                      66445308b05d3eec5aab83dceb5cc87a759bd7d5    87

Compute address

Get address from UTXO:

extract hash from DB
Prepend network designator byte (0xc4 for regtest)
Compute checksum and append
Base58 encode

Network Byte data Checksum c4 66445308b05d3eec5aab83dceb5cc87a759bd7d5b7 86084f

Expressed as Base58: 2N2Zxm7Kf3v4DKMSWW3zYz5Szi8tmGCvtKx

Process of Spending

The conditions specified in an output <scriptPubKey> must be met to spend the funds contained in that output.

The locking script - <scriptPubKey> is a hash of the original custom locking script (<redeemScriptHash>) prepended with the OP_HASH160 opcode and appended with the OP_EQUAL opcode.

The spending input must provide the original <redeemScript> along with the required number of signatures as a <scriptSig>.

Once it has been verified that the hash of the <redeemScript> in the <scriptSig> is equal to the hash contained in the <scriptPubKey>, the <redeemScript> provided in the <scriptSig> can be deserialized and processed. If the provided signatures match, the transaction is considered valid and the funds can be spent.

The mechanics of this involve the protocol concatenating <scriptSig> and <scriptPubKey> and running the entire script. If the final result evaluates to true, the transaction is valid. If at any point the script evaluation fails, the transaction is considered invalid and is discarded.

Step 1: Add scriptSig to Stack

The <scriptSig> consists of the serialized <redeemScript> and the required signatures. This is pushed onto the stack:

Stack

<redeemScript>

<sig 2>

<sig 1>

At this point, a copy is made of the stack - this measure is peculiar to P2SH scripts.

The original locking script (<scriptPubKey>) is now run:

OP_HASH160 0x14 <redeemScriptHash> OP_EQUAL

OP_HASH160: The top item (<redeemScript>) is popped off the stack, hashed with RIPEMD160(SHA256()) (the same hashing process used to generate the <redeemScriptHash> provided by the locking script) and pushed back onto the stack.
The next 0x14 bytes (the <redeemScriptHash>) are pushed onto the stack
OP_EQUAL is executed - the top two elements are popped from the stack and compared. If they are equal, the integer 1 is pushed onto the stack. Otherwise, a zero is pushed.

At this point, if the stack contains the value 1, the <redeemScript> supplied by the unlocking <scriptSig> matches the original script contained within the locking script, and execution can continue. Otherwise, execution stops at this point.

Step 2: Execution of redeemScript

Once it has been determined that the <redeemScript> matches the <redeemScriptHash>, the <redeemScript> provided by the <scriptSig> is deserialized and considered as a locking script to be run against the signatures. This is achieved by:

Operating on the copy of the stack that was made in step 1
Popping the <redeemScript> and deserializing it
Executing the <redeemScript> against the stack contents

References

Worked example, P2SH Multisig Transaction
SO Answer, David Harding
P2SH on Learn Me A Bitcoin

Access YAML Values from Frontmatter Using Python

Wed, 04 Nov 2020 00:00:00 +0000

Access nested (or not) YAML values using Python. Used within a Makefile to set variables from either Markdown frontmatter or YAML config files.

Uses the Python 3 yaml module’s load_all method - this is necessary if accessing YAML within a frontmatter block (e.g. at the start of a markdown file), since the normal frontmatter delimiters cause the Python module to treat the YAML block as the first in a series of YAML documents.

The function receives a list of YAML keys key_list ordered from highest to lowest level, allowing access to nested elements.

For example:

---
name: Rocky
address:
  line 1: 22 Acacia Ave
  line 2: London
---

To access the value “London”, run ./yaml_access.py file.md address "line 2".

To access the value “Rocky”, run ./yaml_access.py file.md name.

#!/usr/bin/env python3
# yaml_access.py

import sys, os, yaml

def main(filename, key_list):
    project_dir = os.path.realpath('.')
    filepath = os.path.join(project_dir, filename)
    
    with open(filepath, "r") as f:
        # Make a dict from the first YAML block
        data = next(yaml.load_all(f, Loader=yaml.FullLoader))
	
        # If there are multiple keys, we need to step down a hierarchy of keys
        # to reach the correct value. At each iteration, set data to be a new
        # simpler dict (or the final value if we're at the last key) by accessing
        # the value by it's key.  
        for key in key_list: data = data[key]
    print(data)

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2:])

Build Bitcoin Core in Ubuntu

Mon, 19 Oct 2020 00:00:00 +0100

This article describes how to clone, configure, make and install Bitcoin Core on a newly installed Ubuntu 20.04 Focal Fossa operating system.

If you’re not a native Linux user, you could consider building and running Bitcoin Core in a Linux virtual machine using something like VirtualBox or a suitable Docker image.

Install Required Build Tools

There are a number of software tools that are necessary for the build process.

Assuming a completely fresh Ubuntu install, you need to install the required build tools, as outlined in doc/build-unix.md:

sudo apt git install build-essential libtool autotools-dev automake pkg-config bsdmainutils python3

Clone Bitcoin Core

Clone the Bitcoin Core codebase and move into the project directory:

git clone https://github.com/bitcoin/bitcoin.git
cd bitcoin

Dependencies

Bitcoin Core has a number of dependencies - it relies on other pieces of software to work and these must be available when compiling Bitcoin Core binaries.

You can either:

Install dependencies system-wide using the Ubuntu package management system - see the following note for a caveat
Build them yourself and reference them appropriately during the Bitcoin Core build process

The second option may be best if you need to ensure compatibility with wallets generated by the distributed binaries, which are compiled with BerkeleyDB version 4.8.

This is incompatible with the version of BerkeleyDB distributed by Ubuntu - which is likely to be version 5.1 or greater. This means that if you use the Ubuntu package manager to install BerkeleyDB (using sudo apt install libdb-dev libdb++-dev) the wallet that you generate will be incompatible with wallets generated by the distributed binaries.

If you choose to do this, you’ll need to pass the --with-incompatible-bdb option to the configure script. I’ve also had problems in the past with incompatible Boost library versions - so if you’re building Bitcoin on a development machine, it might be better to build dependencies. If you do self-build dependencies, you avoid all incompatibility hassles. On the other hand, the build process for dependencies adds to the total build time considerably.

If you are building from self-compiled dependencies, skip the “install required dependencies” in the Bitcoin Core build-unix.md instructions.

Build Dependencies

To build dependencies for the current architecture & OS, move into the depends subdirectory and:

Install dependencies required for the build process
Run make

# Move into the depends subdirectory
cd bitcoin/depends

# Install the tools required in order to build dependencies 
# Some of the dependencies may already have been installed - if this is the case, they will be ignored by `apt` (or `apt-get`) so don't worry about it.
sudo apt install make automake cmake curl g++-multilib libtool binutils-gold bsdmainutils pkg-config python3 patch

# Bitcoin core provides a suitable Makefile that makes self-building of dependencies straightforward: 
cd ~/bitcoin/depends
make

At this point you may want to go and make a cup of tea because this step is going to take some time. Amongst other packages, this process downloads and builds:

Berekely DB 4.8.3
Boost 1.70.0
QT
QR Encode

Once the process has finished, you will have an architecture/OS specific directory in the depends subdirectory.

In the case of Ubuntu 20.04 on a modern PC with an x86 processor, this will likely be in a directory named x86_64-pc-linux-gnu.

To configure for building, move back up into the project directory:

cd ..

Configure

The configure command creates the necessary Makefiles. It allows you to configure(!) the build process.

To generate the configure script, move into the bitcoin core project directory and run:

./autogen.sh

The configure script guesses values for system dependent variables. If you have self-built dependencies you need to set the --prefix command option to the directory that was created when building dependencies (i.e. when make was run in the depends sub-directory). This tells the configure script to pick up libraries, tools and settings from the depends build: --prefix=PREFIX instructs the configure script to install architecture-independent files in PREFIX.

It also means that the generated binaries will be located in a bin subdirectory of the prefix dir.

# Run configure for self-built dependencies
# From the bitcoin core directory
./configure --prefix=$PWD/depends/x86_64-pc-linux-gnu

By default, the generated Makefile includes the compile flags -g -O2 - the build includes debugging symbols and is optimized for code size & execution time. You can change this by passing an option to ./configure:

# Compile without debugging symbols
./configure CXXFLAGS="-O2"

If the config script completes successfully, you are ready to run make to build the executables.

Make

Run make to build bitcoind and bitcoin-qt if dependencies are met.

If the make process is successful, the following binaries can be found in the src directory under the project root:

bitcoind
bitcoin-cli
bitcoin-tx
bitcoin-wallet

…and in the src/qt subdirectory:

bitcoin-qt

Make bitcoind & bitcoin-qt System Commands

This can be achieved by symlinking the executables in a directory that is present in your command search path: $PATH.

For our purposes, usr/local/bin is a good choice:

sudo ln -s ~/bitcoin/src/bitcoin{d,-cli} /usr/local/bin
sudo ln -s ~/bitcoin/src/qt/bitcoin-qt /usr/local/bin

Alternatively, run sudo make install in the project core directory after making the project.

Full Install Instructions, Installed Dependencies

If you’re not worried about wallet compatibility, the commands below are probably the easiest way to build a Bitcoin Core node.

Complete commands to install Bitcoin Core on a new Ubuntu 20.04 installation

# Update OS before starting
# -----------------------------------------------------------------------------------------------------------
sudo apt update && sudo apt upgrade

# Install Dependencies
# -----------------------------------------------------------------------------------------------------------
# Build requirements:
sudo apt install git build-essential libtool autotools-dev automake pkg-config bsdmainutils python3

# Install required dependencies
sudo apt install libevent-dev libboost-system-dev libboost-filesystem-dev libboost-test-dev libboost-thread-dev

# Install the BerkeleyDB from Ubuntu repositories:
sudo apt install libdb-dev libdb++-dev

# Optional: upnpc
sudo apt install libminiupnpc-dev

# Optional ZMQ:
sudo apt install libzmq3-dev

# For GUI:
sudo apt install libqt5gui5 libqt5core5a libqt5dbus5 qttools5-dev qttools5-dev-tools

# For QR Code support
sudo apt install libqrencode-dev

# Install Bitcoin
# -----------------------------------------------------------------------------------------------------------
git clone https://github.com/bitcoin/bitcoin.git

# Move into project directory
cd bitcoin

# Config
# -----------------------------------------------------------------------------------------------------------
# Generate config script
./autogen.sh

# Configure, with incompatible BerkeleyDB
./configure --with-incompatible-bdb

# If debugging symbols not required, amend compile flags:
./configure --with-incompatible-bdb CXXFLAGS="-O2"

# ...lot's of checking...

# Make
# -----------------------------------------------------------------------------------------------------------
make

# Install - sudo is required to install binaries in /usr/local/bin
sudo make install

Developer Notes

If you’re intending to work on Bitcoin Core, you should fork a copy on GitHub, before cloning and building your own fork.

It you do this, you’ll need to keep your fork in sync with upstream development.

Configure Remote

Check current remotes:

git remote

This will probably output origin, denoting that your repo has a single remote.

Configure a Git remote for the upstream (original) repo:

git remote add upstream https://github.com/original-project/original-project.git

Check:

git remote

# Output:
origin
upstream

Sync Local Fork with Upstream

There are now three repositories:

local
origin (GitHub, Bitbucket etc)
upstream (The original repo)

Sync from upstream to local, then push changes to origin:

# fetch changes
git fetch upstream

# switch to master branch
git checkout master

# Merge changes from upstream into master
git merge upstream/master

# Push local changes to origin
git push

References

Notes on self-building dependencies

Construct C++ Objects From Stdin

Sat, 12 Sep 2020 00:00:00 +0100

Objects From Stdin: C++

This project is an example of how stdin data can be used to construct objects in C++ under Linux.

Data is input via stdin either through the terminal or by means of file redirection.

You could use this as boilerplate for importing data stored in a file into C++ data structures. In this case, data is imported on a line-by-line basis, with each line providing data for a separate object.

A vector of unique_ptr to objects is used to store objects.

Use Case

I put this project together to better understand input streams in C++. The aim was to accept input by means of file redirection or terminal input. I also wanted to switch to terminal input for additional interaction in the event of file redirection input of the main body of data.

One interesting result of this approach is that you can input data either by means of file redirection or by means of terminal input in the default stdin way.

File Redirection

Redirection in Linux involves changing standard input/output devices.

Usually, the standard input device stdin is the keyboard and the standard output device stdout is the screen.

This means that in C++ std::cin >> input; accepts input from the keyboard, and std::cout << output outputs data to the screen.

To input data from a file, redirection can be used:

./myProg < input.txt

In this case, the shell streams in the contents of input.txt to stdin instead of accepting keyboard input from the terminal.

Terminal Input

If you run the binary file without supplying data by means of file redirection, the user is required to supply data at the terminal. Data is entered:

Line-by-line, with each line representing a separate object (\n delineates objects).
Individual scores are separated by spaces.
Input is ended by entering Ctrl-D which is the Linux end-of-file signal.

Switching from File Redirection to Terminal

When input is redirected from a file, std::cin refers to the redirected file. If you need to get input from the terminal at a later stage in the programme, you can assign another input buffer using std::cin.rdbuf().

// Define an input stream as the terminal for the current process (/dev/tty)
std::fstream in("/dev/tty");

// Set the streambuf of std::cin as the streambuf of /dev/tty.
std::cin.rdbuf(in.rdbuf());

// stdin is now /dev/tty - keyboard input.

For a working example, see below:

#include <ostream>
#include "record.h" // Defines Record, a custom data type (class)

int main()
{
	// Create a vector of *Record unique pointers
	std::vector<std::unique_ptr<Record>> records;
	while (1) {
		// Make a unique pointer to a Record
		auto s = std::make_unique<Record>();

		// Input data to this object - reset object if EOF is indicated
		if (s->inputFromStdin() == -1) {
			s.reset();
			break;
		}

		// Move the object to the array
		records.push_back(std::move(s));
	}
	
	// Reset stdin to terminal	
	std::fstream in("/dev/tty");
	std::cin.rdbuf(in.rdbuf());
     
	for (auto& record : records) {
		std::cout << *record;
	}
	
	try {
		int index;

		// Output to tty
		std::streambuf *backup = std::cout.rdbuf();
		std::ofstream out("/dev/tty");
		std::cout.rdbuf(out.rdbuf());
		
		// This should print to screen even if output is redirected to file.
		std::cout << "Enter index: ";
		std::cin >> index;
		
		std::cout.rdbuf(backup); // Reset the original streambuf to std::cout
		Record s = *(records[0]);
		std::cout << "value: " << s[index] << '\n';
	} catch (const IndexOutOfBoundsException& e) {
		std::cout << "ERROR: index out of bounds." << '\n';
	}
}

GitHub Repo.

std::fstream is the input/output file stream class. The code above associates the /dev/tty (terminal device) stream with the variable in.

The next line sets the std::streambuf for std::cin to the std::streambuf of in which has been set to dev/tty.

The function std::ios::rdbuf() is used to get and set a stream buffer:

streambuf* rdbuf() const; // get - returns a pointer to the stream buffer of the object associated with stream
streambuf* rdbuf (streambuf* sb); // sets the object pointed at by sb, clears error flags.

Headers: <ios> <iostream>.

Streams & Stream Buffers

A stream is a communication channel between a programme and it’s environment.

Streams involve data flowing from a source to a destination - for example, data may flow from the programme to the screen, or may be read from the user’s keyboard into the programme.

A stream buffer is a memory buffer for the stream - the streambuf is an interface that defines a mapping from a stream to a device, file or memory.

The stream is responsible for converting data into an appropriate format.
The streambuf actually communicates the data.

References

Diffie Hellman Key Exchange

Tue, 18 Aug 2020 00:00:00 +0100

The fastest encryption schemes are symmetric - both participants have a copy of a secret key, which is used to encrypt and decrypt messages.

In order to set up such a scheme, Alice and Bob would need to share the secret key by means of a secure channel - and this is where symmetric encryption schemes can be problematic.

If Alice and Bob have a secure communication channel by which to transfer a secret key, it obviates to some extent the necessity for symmetric encryption - they could simply exchange their private message over the secret channel. If they don’t have a secure communication channel, they can’t securely exchange a private key and symmetric encryption is therefore not possible.

Diffie Hellman Key Exchange allows participants to independently compute a shared secret key such that it is computationally infeasible for an adversary to determine the key. Once the “secret” key has been computed by the participants, they can communicate using symmetric encryption.

Neither participant chooses the secret key - but by the end of the algorithm, they each share a copy.

Algortihm

Shared Values (Public)

Participants Alice and Bob agree on two shared values, q and g.

q: A large prime number.
g: Primitive root of q.

These values are public - they are available to any potential adverary.

Private (Secret) Keys

Alice and Bob each selects a large random number X that serves as their individual private key.

They do not share their private keys - the security of the system is dependent upon these values remaining secret.

Alice’s private key: Xa
Bob’s private key: Xb

Public Keys

Each particpant computes their Public key by raising g to the power of their private key, modulo q:

Y = g^X modulo q

Alice’s public key: Ya = g^Xa modulo q
Bob’s public key: Yb = g^Xb modulo q

These public keys are then shared. Public sharing of these keys does not affect the security of the system.

Compute Shared Key

Each participant computes a shared private key by raising the other participant’s Public key to the power of their own Private key.

This results in each participant having the same secret key - but computation of the secret key is not computationally feasible without the particpant’s Private key that they used to generate the Public key.

Summary of Values

Value	Alice	Bob
Random number: private	Xa	Xb
Computed public key	Ya = g^Xa mod q	Yb = g^Xb mod q
Keys swapped	Ya, Yb	Yb, Ya
Computed shared key	K_ab = Yb^Xa	K_ba = Ya^Xb

Because:

Ya = g^Xa mod q

and

Yb = g^Xb mod q

For Alice:

(g^Xb)^Xa = g^{Xb . Xa}

For Bob:

(g^Xa)^Xb = g^{Xa . Xb}

This works because powers of powers are commutative:

(a^b)^c = a^bc = a^cb = (a^c)^b

Summary of the Private & Exchanged Values

Participant	Private Key	Shared Public Key	Shared Key: computed individually
Alice	Xa	g^Xa mod q	g^{Xb . Xa}
Bob	Xb	g^Xb mod q	g^{Xa . Xb}

Alice and Bob now share a key that is known uniquely to themselves.

They each used their individual private key along with the other participants “public” key.

So long as the public keys are computed with the same values of g and q, the key that they each compute independently will be identical.

Security: Discrete Logarithm Problem

It is hard to compute the private key from the public key.

For example, given Alice’s public key (Ya = g^Xa mod q), it is hard to determine the private key Xa from g^Xa. This is known as the discrete logarithm problem.

Given g^e ≡ c mod q, it is difficult to compute e when g (the original generator), c(a public key) and q (the modulus) are known.

Pattern Matching in Rust During Variable Assignment

Thu, 02 Jul 2020 00:00:00 +0100

The match keyword in Rust is something like a C switch statement - but it can also be used during variable assignment:

fn main() {

    // Match is an expression as well as a statement. This example shows how
    // a variable is set based on an Option enum.
    // ------------------------------------------
    let msg_option = Some("+++MELON MELON MELON+++");   // Some variant
    let msg_option_empty = None;                        // None variant

    // The variable msg is assigned a value dependent upon the Option variant	
    let msg = match msg_option {
        Some(m) => m,   // m is the unwrapped data from the option 
        None => "Out of Cheese Error",
    };
    println!("msg = {}", msg);
    
    // As above, but handling the None variant
    let msg_2 = match msg_option_empty {
        Some(m) => m,   // In this case we won't enter this branch
        None => "Out of Cheese Error", // msg_2 set to this value
    };
    println!("msg_2 = {}", msg_2);
}

// Output:
// msg = +++MELON MELON MELON+++
// msg_2 = Out of Cheese Error

This is a really handy idiom given that functions in Rust often return an Option enum.

You could achieve the same thing using unwrap_or():

fn main() {

    let msg_option = Some("+++MELON MELON MELON+++");   // Some variant
    let msg_option_empty = None;                        // None variant
    let default_msg = "+++Please Reinstall Universe and Reboot+++";

    let msg = msg_option.unwrap_or(default_msg);
    println!("msg = {}", msg);
    
    // As above, but handling the None variant
    let msg_2 = msg_option_empty.unwrap_or(default_msg);
    println!("msg_2 = {}", msg_2);
}

// Output:
// msg = +++MELON MELON MELON+++
// msg_2 = +++Please Reinstall Universe and Reboot+++

Using C++ Vectors in C Functions

Sun, 28 Jun 2020 00:00:00 +0100

You can use C utilities in C++ code. For example, you may be using openssl to perform hashing algorithms.

In this case, the openssl code is C rather than C++. It can’t operate on vectors of bytes - instead it uses basic C style data structures like const void * - an array of void pointers.

Example: `SHA256_update()`

The SHA256_update() function is part of the openssl library. The signature of this function is: int SHA256_Update(SHA256_CTX *c, const void *preimageBytes, size_t len);.

It serves to fill the bytes of c with the hash of preimageBytes.

Note that 2nd parameter is a const void *. This means that the caller must provide a pointer to data rather than a std::vector or a vector iterator.

The caller can either pass v.data() or a pointer to the first element in the vector: &v[0].

Code Example

// main.cpp
#include <iostream>
#include <vector>
#include <iomanip>
#include <openssl/sha.h>
#include <openssl/ripemd.h>

#define HASH_LEN 32

int sha256(const uint8_t preimageBytes[], size_t len, std::vector<uint8_t>& res)
{
	SHA256_CTX sha256;
	SHA256_Init(&sha256);
	SHA256_Update(&sha256, preimageBytes, len); 
	int ret = SHA256_Final(res.data(), &sha256);
	return ret;
}

int main()
{
	std::vector<uint8_t> preimageBytes{0xde, 0xad, 0xbe, 0xef};
	std::vector<uint8_t> hash(HASH_LEN);
	sha256(preimageBytes.data(), preimageBytes.size(), hash);
	
	for (auto& el : hash)
		std::cout << std::hex << std::setfill('0') << std::setw(2) << (int)el;
	std::cout << '\n';

	return 0;
}

To compile:

g++ -W -Wall -std=c++17 -g -o prog main.cpp -lssl -lcrypto

Pseudo Random Numbers in a Range and Modulo Bias

Mon, 08 Jun 2020 00:00:00 +0100

It is sometimes necessary to obtain a pseudo-random number from a specific range. For example:

Simulating a fair die roll, where the value should be between 1 and 6 inclusive
Randomly selecting a word from a word list - for example, generating a secure passphrase

This can be achieved by using modular arithmetic whereby numbers are limited to a certain range, wrapping around when the range is exceeded.

Selecting From a Range

Suppose you have a random number generator that generates numbers with a sufficient degree of randomness in the range [0, 16) - i.e. 16 integers, from 0 to 15 inclusive.

You need a random number in the range [1, 6].

To achieve this, you could reduce the range of the random number by modular division:

Determine an appropriate modulus, m such that m = upper bound - lower bound + 1 (in this case, 6 - 1 + 1 = 6)
Compute the remainder (mod m) of the random number returned
Add this to the lower bound (in this case, 1) to get a random number in the required range

The diagram below shows each possible random number in our 16 number range in black, with the remainder mod 6 shown in blue:

From this diagram, you can see that the probability of returning a number of 1 to 6 is NOT evenly distributed. You can easily see probabilities in this case by simply counting how many times they occur and dividing by the total number of possible values:

Mod 6 Value + 1, Random Number 0-15	Probability
1	3/16
2	3/16
3	3/16
4	3/16
5	2/16
6	2/16

In this case the values 5 and 6 have a (slightly) lower probability of being returned, even if the original random number (between 0 and 15) is truly random. This effect is known as modulo bias.

Random Bytes

Modern computers operate effectively at the level of the byte (8 bits) - under Linux, you might get a pseudo-random number by fetching a byte from the /dev/urandom or /dev/random devices - these are basically the Linux kernel API for accessing randomness.

This random byte represents a decimal number between 0 (an empty byte) and 255 (wherein all bits are set).

To continue with the fair die analogy, consider how we might return a number in the range [1, 6] for all possible values of a single byte:

This time the absolute difference in probabilities is smaller, because the possible range for our original random number is greater - there are 256 possibilities compared to 16:

Mod 6 Value + 1, Random Number 0-255	Probability
1	43/256
2	43/256
3	43/256
4	43/256
5	42/256
6	42/256

Correcting for Modulo Bias

We can correct modulo bias by discarding any random number in excess of the maximum exact multiple of our modulus.

In other words:

Determine the modulus m such that we return numbers in the required range [lower bound, upper bound]: upper bound - lower bound + lower bound
Take the number of possible random values - this will be the maximum possible value + 1 to account for the possibility of zero
Determine the remainder of dividing the number of possible random numbers by m
Subtract this excess from the maximum possible random number

The result is a threshold above which we discard random numbers. If the random number exceeds this threshold, we re-roll our random number generator until we return a value that is within the range. This ensures that the final results are evenly distributed.

Taking the 16 value example above, we would discard random values greater than or equal to 12:

16 (mod 6) ≡ 4
16 - 4 = 12
If the random integer is > 11 we should reject it

Doing the same thing for the one-byte example, 251 is our threshold above which we should reject values.

Example: Python

This example uses the range-restricted random number to pseudo-randomly select a word from a wordlist:

#!/usr/bin/env python3
import sys
import os
from subprocess import check_output

class RandomInRange:
    """
    Class that provides a pseudo-random number within a given range,
    corrected for modulo bias.
    """
    def __init__(self, lower, upper):
        self.lower = lower
        self.upper = upper
        self.random_src_path = '/dev/urandom'
        self.set_parameters()

    def set_parameters(self):
        max_bytes = {
            1: (1 <<  8) - 1, # Max required random number is 2⁸ - 1
            2: (1 << 16) - 1, # Max required random number is 2¹⁶ - 1
            3: (1 << 24) - 1, # Max required random number is 2²⁴ - 1
            4: (1 << 32) - 1  # Max required random number is 2³² - 1
            }
        
        # Compute modulus m that will be used to restrict range
        self.m = self.upper - self.lower + 1
        try:
            assert(self.m < max_bytes[4]), "Max value exceeded."
        except AssertionError as error:
            print(error)
            print("The range magnitude {} is too large.".format(self.m)) 
            sys.exit(1)

        # Determine number of pseudo random bytes needed and the related max random value
        for i in range(1, 4):
            if self.m <= max_bytes[i]:
                self.n_bytes = i
                self.max_random = max_bytes[i]
                break

        self.excess = (self.max_random % self.m) + 1
        self.max_allowed = self.max_random - self.excess
        
    def random_number(self):
        while True:
            data = os.urandom(self.n_bytes)
            data = int.from_bytes(data, "big")
            if data < self.max_allowed:
                break
        return (data % self.m) + self.lower

class RandomWords:
    """
    Class that builds a list of randomly-selected words.
    """
    def __init__(self, n):
        # Number of words
        self.n = n
        # Initialise an empty list for the randomly selected words
        self.random_words = list()
        # The word list to select from (Ubuntu/Debian)
        self.words = '/usr/share/dict/cracklib-small'
        # Number of lines in the wordlist file
        self.keyspace = sum(1 for line in open(self.words))
        # Initialise a RandomInRange object
        self.r = RandomInRange(1, self.keyspace)
        self.make_list()

    def make_list(self, n=None):
        """
        Randomly select words from a wordlist and append them to a list.
        Uses a modular range-reduced random number to make the selection
        without modulo bias.
        """
        n = n if n else self.n
        for i in range(0, n):
            r_num = self.r.random_number()
            self.random_words.append(self.get_one_line(r_num).decode('utf-8').rstrip())

    def get_one_line(self, line_number):
        """
        sed is a reasonable way to get a line from a file indexed by line number.
        If there is a more Pythonic way of doing this, please leave a comment.
        """
        return check_output([
            "sed",
            "-n",
            "%sp" % line_number,
            self.words
            ])

    def print_words(self):
        for w in self.random_words:
            print(w)

def main():
    r = RandomWords(24)
    r.print_words()

if __name__ == '__main__':
    main()

Example: Bash

#!/bin/bash
#
# Copyright 2020 David Egan
# 
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# 
# http:#www.apache.org/licenses/LICENSE-2.0
#
# Random Number Within a Specific Range
# -------------------------------------
# To obtain a random number within a particular (inclusive) range within a Bash script:
# * Source this file
# * Run `random_number_in_range <lower bound> <upper bound>`
# * The `$random_number` variable in your calling script will have an
# appropriate random number
#
# If you run `random_number_in_range` with no parameters, you will be prompted to enter
# a lower and upper value.
# -----------------------------------------------------------------------------------------------------------
RANDOM_SOURCE="/dev/urandom"
ONE_BYTE_MAX=$(( (1 « 8) - 1 ))
TWO_BYTE_MAX=$(( 1 « 16 - 1 ))
THREE_BYTE_MAX=$(( (1 « 24) - 1 ))

function set_n_bytes {
	lower=$1
	upper=$2
	m=$(($upper - $lower + 1))
	if (( $m < $ONE_BYTE_MAX )); then
		n_random_bytes=1
		max_random=$ONE_BYTE_MAX
	elif (( $m > $ONE_BYTE_MAX && $m < $TWO_BYTE_MAX )); then
		n_random_bytes=2
		max_random=$TWO_BYTE_MAX
	elif (( $m > $TWO_BYTE_MAX && $m < $THREE_BYTE_MAX )); then
		n_random_bytes=3
		max_random=$THREE_BYTE_MAX
	elif (( $m >= $THREE_BYTE_MAX )); then
		echo "Too big for this script"
		exit 1
	fi

	mod=$(( $upper - $lower + 1))
	excess=$(( ($max_random % $mod) + 1 ))
	max_allowed=$(( $max_random - $excess ))
}

function random_in_range {
	if [[ $excess != 0 ]]; then
		while (( 1 )); do
			random_number=$(od -N${n_random_bytes} -An -i ${RANDOM_SOURCE})
			if (( $random_number <= $max_allowed )); then
				break
			fi
		done
	else
		random_number=$(od -N${n_random_bytes} -An -i ${RANDOM_SOURCE})
	fi
	random_number=$(( ($random_number % $mod) + $lower ))
}

function set_inputs {
	if [[ $# -eq 0 ]]; then
		echo "Enter lower (inclusive) bound:"
		read lower
		echo "Enter upper (inclusive) bound:"
		read upper
	else
		lower=$1
		upper=$2
	fi
	set_n_bytes $lower $upper
}


function random_number_in_range {
	set_inputs $1 $2
	random_in_range
}

function test_run {
	. tests
	set_inputs 1 6
	test_diceroll
	set_inputs 1 4
	test_4_sided_diceroll
}

[[ $1 == "test" ]] && test_run

NOTE: If you copy and paste the above code it will error - I used digraphs for the left-shift operators because markdown syntax highlighting thinks << should start a heredoc, even inside a $(( )) block.

If you want a copy of this script you can get it here.

A Note on Randomness

Remember that your source of “Randomness” is very important. For most purposes, we can use Pseudo random number generators such as /dev/urandom and /dev/random on Linux.

Anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin. For, as has been pointed out several times, there is no such thing as a random number — there are only methods to produce random numbers, and a strict arithmetic procedure of course is not such a method.

John Von Neumann

Because computers are deterministic, they can only ever produce pseudo-randomness.

References

GitHub repo for this post with examples using Bash and Python to generate randomly selected words
Randomness in C++ - I included a “diceroll” example here that simulates a diceroll using C++ and the Linux kernel randomness API
Fast alternative to modulo reduction - if speed is important for your application, read this article!

Image by 955169 from Pixabay

Notes on GNU Make

Thu, 04 Jun 2020 00:00:00 +0100

The GNU make utility is a tool that maintains groups of programmes make can determine which pieces of a large programme need to be compiled and/or recompiled, and generally take care of compilation and recompilation in an efficient way.

You can use make with any programming language whose compiler can be run from the command line. Beyond compiling programmes, make can describe any task that involves updating files automatically when other files change.

The Makefile

The Makefile is used to describe relationships between files in the programme, and the commands necessary for updating files.

In C programmes, the executable file is updated (“linked”) from “object” files, which are in turn made by compiling source files.

Once you’ve written a suitable Makefile, you just need to run make in the programme root directory to perform all necessary recompilations.

If you don’t specify a Makefile with make -f <makefile>, make will look for the makefiles:

GNUmakefile
makefile
Makefile

in that order. It is recommended to call the makefile Makefile.

The makefile contains one or more rules that specify how a target is produced from prerequisite files. make updates a target if the prerequisite files that it depends upon have been modified since the target was last changed, or if the target does not exist.

Rules

A make rule is comprised of a target specification and it’s prerequisite files, all on one line and with the target separated from the prerequisites by a colon.

After the space-separated list of prerequisites, the rule has a newline and a list of commands that should be executed required to rebuild the target from the prerequisites.

Commands may be added over multiple lines, but each line that contains a command must start with a TAB character.

Example Makefile

The following makefile has 3 targets: myProgramme, moduleOne & moduleTwo:

myProgramme: moduleOne.o moduleTwo.o
	gcc -o myProgramme moduleOne.o moduleTwo.o

moduleOne.o: moduleOne.c moduleOne.h standaloneHeader.h
	gcc -std=gnu=gnu99 -pedantic -Wall -c moduleOne.c

moduleTwo.o: moduleTwo.c moduleTwo.h standaloneHeader.h
	gcc -std=gnu=gnu99 -pedantic -Wall -c moduleTwo.c

Special Make Variables

Inside an action, special variables are available for matching filenames:

$@: The full target name of the current target
$?: Dependencies that are newer than the current target
$*: Returns text corresponding to % in current target
$<: Returns the name of the first dependency
$^: Name of all dependencies, space separated

This allows your makefile to be more concise:

Example Makefile 2

Example C++ makefile for the following project structure:

main.cpp: includes grade.h & util.h
util.cpp: includes util.h
util.h: contains declarations
grade.cpp: includes grade.h
grade.h: contains declarations

# Set a variable for flags to avoid repetition
CXXFLAGS = -W -Wall -pedantic -std=c++17 -g
CXX = g++ ${CXXFLAGS}
NAME = grades

# The target has the dependencies util.o, main.o & grade.o
$(NAME): util.o main.o grade.o
	$(CXX) -o $@ $^

# For any .c file, build an object file by applying the actions on the following line.
# This is a pattern rule, a type of implicit rule. Specifies one target and one dependency.
# Causes one invocation of $(CXX) for each target.
%.o: %.c
	$(CXX) -c $<

# util.c and grade.c include header files. Their object files are generated by the implicit
# rule above, but the following lines specify additional dependencies:
util.o: util.h
grade.o: grade.h

# Remove the executable and all object files
.PHONY: clean
clean:
	-rm $(NAME) *.o

Note that the pattern rule (%.o: %.c) involves matching file names - the % can match any non-empty substring, whereas the characters match only themselves. The rule says: “Compile .c files into corresponding .o files”.

In this example, main.o, util.o and grade.o will be made from main.c, util.c and grade.c respectively.

Variables for Implicit Rules

CFLAGS: Extra flags to give to the C compiler.
CPPFLAGS: Extra flags to give to the C preprocessor and programs that use it (the C and Fortran compilers).
CXX: Program for compiling C++ programs, default g++.
CXXFLAGS: Extra flags to give to the C++ compiler.

Set Variables

Assign variables within a Makefile.

In a Makefile, a variable is a name assigned to a value, which is a string of text. Values are substituted into targets, prerequisites, commands etc.

# Set a variable to hold the absolute path of the Makefile
# Getting `firstword` strips off any includes
ABS_PATH := $(dir $(realpath $(firstword $(MAKEFILE_LIST))))

# Alternatively:
ABS_PATH := $(shell pwd)

See this article for a more complete description.

Variables can be either recursively expanded (defined using the = operator) or simply expanded (defined using the := operator).

See this article for more information.

Example: C

# Set common parameters
CC = gcc
CFLAGS = -std=gnu99 -W -Wall -pedantic -g

# Specify necessary targets/tasks
all: maxSeq.o test-subseq.o test-subseq

# Compile maxSeq.c into maxSeq.o
maxSeq.o: maxSeq.c
        $(CC) $(CFLAGS) -c maxSeq.c

# Compile test-subseq.c into test-subseq.o
test-subseq.o: test-subseq.c
        $(CC) $(CFLAGS) -c test-subseq.c

# Link object files test-subseq.o and maxSeq.o into an executable test-subseq
test-subseq: test-subseq.o maxSeq.o
        $(CC) $(CFLAGS) -o test-subseq test-subseq.o maxSeq.o

clean:
        rm maxSeq.o test-subseq.o

Example: C++ Makefile to Create Multiple Executables

In this example, each *.cpp file is compiled and linked into a distinct executable:

# Makefile to compile and link all C++ files in the current directory, such that each C++
# file results in a unique executable.
#
# This is useful for experimentation, testing and small-scale projects.
# 
# Copyright (c) David Egan 2020
# SPDX-License-Identifier: GPL-2.0-or-later

WARNINGS = -pedantic -Wall -Wfatal-errors -Wextra -Wno-unused-parameter -Wno-unused-variable
BIN_DIR = bin
OBJS_DIR = $(BIN_DIR)/objs
CXXFLAGS = $(WARNINGS) -std=c++17 -g
LDFLAGS =
CXX = g++ ${CXXFLAGS}

# Handle all .cpp files together - allows more to be added to the project without
# changing the Makefile
SRCS = $(wildcard *.cpp)

# String substitution to generate appropriate names for executable files
EXECUTABLE_FILES = $(SRCS:%.cpp=$(BIN_DIR)/%)
# Note that the following is an alternative way of performing string substitution:
# EXECUTABLE_FILES = $(patsubst %.cpp,$(BIN_DIR)/%,$(SRCS))

# Output for debugging purposes
$(info EXECUTABLE_FILES = "$(EXECUTABLE_FILES)")

# There should be a corresponding object file for each .cpp file
OBJECT_FILES = $(SRCS:%.cpp=$(OBJS_DIR)/%.o)

.PHONY: all clean
all: $(EXECUTABLE_FILES)

# This pattern is triggered by the `all` rule. This is a builtin rule
# that builds executables from object files.
$(BIN_DIR)/%: $(OBJS_DIR)/%.o
	@$(CXX) $(LDFLAGS) -o $@ $^

# Builtin rule to generate object files from .cpp files
$(OBJS_DIR)/%.o: %.cpp
	$(info Building object file for "$(<)")
	@mkdir -p $(@D)
	@$(CXX) $(CXXFLAGS) -o $@ -c $<

# Additional dependencies
$(OBJS_DIR)/selection-sort.o: util.h
$(OBJS_DIR)/merge-sort.o: util.h

clean:
	rm $(OBJS_DIR)/*.o
	rm -f $(BIN_DIR)/*

Project Structure:


.
├── bin
│   ├── merge-sort
│   ├── objs
│   │   ├── merge-sort.o
│   │   └── selection-sort.o
│   └── selection-sort
├── Makefile
├── merge-sort.cpp
├── selection-sort.cpp
└── util.h

2 directories, 8 files

Output Variables

The internal info function allows you to output variables to stdout:

$(info EXECUTABLE_FILES = "$(EXECUTABLE_FILES)")
# outputs the value of `EXECUTABLE_FILES`

String Substitution

$(patsubst pattern,replacement,text) finds whitespace separated words in text that match pattern and replace these with replacement. The pattern may contain a % character, which acts as a wildcard matching any number of characters within a word - escape literal % characters in a pattern with a \.

Example:

# For each "word" in $(SRCS) remove .cpp and prepend a directory path
EXECUTABLE_FILES = $(patsubst %.cpp,$(BIN_DIR)/%,$(SRCS))

You can use substitution references to get the same effect:

EXECUTABLE_FILES = $(SRCS:%.cpp=$(BIN_DIR)/%)

Substitution references substitute the value of a variable with the specified alterations, in the form:

$(VAR:a=b)

This means take the value of VAR and replace every a at the end of a word with b. Wildcards can also be used, as shown in the example above.

References

Set Default Values in Rust

Thu, 07 May 2020 00:00:00 +0100

Rust provides many ways to set a default value - or to fallback gracefully in the event of an error. This article focuses on how to handle Option types.

The Option<T> type represents an optional value. An Option<T> can either contain Some() value, or None (which is self-explanatory). Specifically, Option<T> is an enum with two variants:

None: indicating lack of value or failure
Some(value): A tuple struct that wraps a value of type T

Unwrapping an Option

If the Option is a Some variant, the value can be accessed using the unwrap() method:

let x: Option<u8> = Some(42);
println!("x unwraps to give the value {}", x.unwrap());
// x unwraps to give the value 42

Trying to unwrap() a None variant causes a panic!, which is not a particulary elegant way of terminating early:

let y: Option<u8> = None;
println!("unwrapping y will cause a panic! {}", y.unwrap()); 
// thread 'main' panicked at 'called `Option::unwrap()` on a `None` value'...

Unwrap, Or?

If you use the unwrap_or() method to unwrap an Option<T>, you get to specify a fallback/default value.

This ensures that the programme does not panic and is useful if the only appropriate response to the None variant is a default value:

fn main() {
    // Simulate function returning Option
    let x: Option<u8> = Some(42);	// Some variant
    let y = None;			// None variant
    
    let a = x.unwrap_or(13);
    let b = y.unwrap_or(13);
    
    println!("a = {}", a);
    println!("b = {}", b);
}

// Output:
// a = 42
// b = 13

Another example that uses Option<&'static str>:

fn main() {
    // Simulate function returning Option of Some() variant:
    let message: Option<&'static str> = Some("+++MELON MELON MELON+++");

    // Simulate function returning Option of None variant:
    let empty_message = None;
    
    let a = message.unwrap_or("+++Divide By Cucumber Error. Please Reinstall Universe And Reboot +++");
    let b = empty_message.unwrap_or("+++Divide By Cucumber Error. Please Reinstall Universe And Reboot +++");
    
    println!("a = {}", a);
    println!("b = {}", b);
}

// Output:
// a = +++MELON MELON MELON+++
// b = +++Divide By Cucumber Error. Please Reinstall Universe And Reboot +++

Let Match

A match statement can be used to match against an Option<T> pattern, which allows for more complex handling of each variant.

This example accomplishes the same thing as the one above - as such it’s not that useful, but is presented as a comparison:

fn main() {

    // Match is an expression as well as a statement. This example shows how
    // a variable is set based on an Option enum.
    // ------------------------------------------
    let msg_option = Some("+++MELON MELON MELON+++");   // Some variant
    let msg_option_empty = None;                        // None variant

    let msg = match msg_option {
        Some(m) => m,   // m is the unwrapped data from the option
        None => "Out of Cheese Error",
    };
    println!("msg = {}", msg);
    
    // As above, but handling the None variant
    let msg_2 = match msg_option_empty {
        Some(m) => m,   // In this case we won't enter this branch
        None => "Out of Cheese Error", // msg_2 set to this value
    };
    println!("msg_2 = {}", msg_2);
}

// Output:
// msg = +++MELON MELON MELON+++
// msg_2 = Out of Cheese Error

The following example shows how we might handle Option return from a function.

In this case, if the function returns the None variant, we print a message and end the programme early:

fn main() {
    let i = 5;
    //let input: Option<u8> = Some(i);
    let input: Option<u8> = None;
    
    // If the Option returned by square() has Some() value, set s to this value.
    // Otherwise, return early.
    let s = match square(input) {
        Some(ans) => ans,
        None => {
            println!("square() did not return a value.");
            return
        },
    };
    // We didn't return early, so s has a value. There is no need to unwrap.
    println!("{operand} * {operand} = {}", s, operand = i);
}

/// Check if arg has a value. If it does, set val to the value.
/// Otherwise, return None. Once we have a val, return the square.
fn square(arg: Option<u8>) -> Option<u8> {
    let val = if let Some(val) = arg {
        val
    } else {
            return None;
    };
    Some(val * val)
}

The if let Idiom

In the square() function in the example above, you could write the guard clause as:

let val = if let Some(val) = arg { val } else { return None; };

// If arg Option destructures, set val to it's value. Otherwise, enter the else block.

This has the effect of setting val to the value of arg if it is the Some() variant. If arg is the None variant, the function returns early with the returned Option being the None variant.

The if let idiom combines a pattern match and an if statement.

If you just want to operate on the value held by the Option when it is of the Some() variant:

fn main() {
    let n = Some(42); // The Option
    
    // If you just want to operate on the value held by the `Option`.
    // If the option is the `None` variant, do nothing.
    // "If n is the `Some()` variant, bind it's value to r and use this in the
    // following block."
    if let Some(r) = n {
        // We only enter this block if n is the Some() variant 
        print_num(r);
    }
    
    // This has the same result as the above if let block:
    match n {
        Some(r) => print_num(r),
        _ => {},
    }
}

Failure Case/Fallback Required

fn main() {
    let n = None;//Some(42);
    let fallback = 17;
    
    // If a failure case is required.
    // "If n is the `Some()` variant, bind it's destructured value to r and pass this
    // to the following block. Otherwise, enter the else block." 
    if let Some(r) = n {
        print_num(r);
    } else {
        print_num(fallback);
    }
    
    // This has the same result as the above:
    match n {
        Some(r) => print_num(r),
        _ => print_num(fallback),
    }
}

Save Destructured Value into a Variable with a Default

The following example shows how to save the destructured value into a variable (in the event of the Option being the Some() variant) with a default value as a fallback:

fn main() {
    let n = None;//Some(42);
   
    // Save the destructured value into a variable.
    // If the Option variant is Some(), destructure it's value and save it into
    // the variable `result`. In the case of the `None` variant, set result to
    // the default value of 13.
    let result = if let Some(r) = n { r } else { 13 };
    
    // Equivalent to the above:
    let result_2 = match n {
        Some(x) => x,
        None => 13,
    };
    
    println!("result = {}", result);
    println!("result _2 = {}", result_2);
}

References

Rust by Example: Option - has a good practical example that handles divide by zero
Rust by Example: Result - follows on from the above, with better handling of divide by zero.
Concise, clear description of if let

Contrasting C and JavaScript - Make an Array of Words From a String

Tue, 28 Apr 2020 00:00:00 +0100

This article contrasts how you would make an array of strings representing the “words” in an original sentence - in other words, build an array of substrings derived from an original string, delimited by space characters or the string end.

The objective is to write a function that mutates a passed-in array.

C Code

In C, function parameters are passed by value. The value of the parameter is copied into the function’s stack frame - any changes to the function parameter are only relevant within the scope of the function.

For a function to modify data and for such changes to persist in the outside scope, the parameter must be passed as a pointer. In this way, the memory address of the “outside” variable is copied into the stack frame of the function, where it can be de-referenced to access (and change) the actual data.

In the case of an array of strings, the variable to be changed in the function is already a pointer to a pointer. The base data type is a char, with a char* being a pointer to a char, which can be used to represent an array of char - or when null-terminated, a string. An array of strings therefore can be represented by a pointer to a char * - in other words, char **.

If char ** is our array of strings, to modify this in a function we need to pass in a pointer to char **, or char ***.

Inside the function, dereferencing char ***words once - *words provides the pointer to a pointer that is the array of strings we’re working on.

We need to make sure that the array has enough space to hold pointers to the required number of words. For this example, the required memory is allocated dynamically. This means that memory must be carefully tracked by the programmer and freed later. The code therefore includes a function to free the array, by firstly iterating over and freeing individual strings.

Because memory allocation (and reallocation) is relatively expensive, the function initially allocates space for up to 10 words. Whenever a new word is detected, we check to see if this memory allocation needs to be increased. If this is the case, a realloc() is performed, adding space for an extra 10 “word pointers”. We add an extra 10 rather than just a single extra pointer to avoid re-allocating memory too frequently.

When saving individual words as strings in the array, we know the exact number of characters in the string. We allocate memory for these using calloc(): char *tmp = calloc(wordSize + 1, sizeof(*tmp));. This function initialises all memory to zero, so if we initialise with one more member than we need for the word characters, the final character will be 0 - which serves as a null terminator and saves us the bother of null-terminating the string later.

Note that all memory allocations are checked for success - if calloc() or realloc() return NULL we trigger a graceful exit of the programme.


#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <stdlib.h>

const char str[] = "Out of cheese error +++ redo from start";

void printArray(char **arr, size_t n);
void allocFail(const char *msg);
void freeArray(char **arr, size_t n);

size_t setArray(char ***words, const char *str)
{
	size_t len = strlen(str);
	size_t nAllocatedWords = 10;
	*words = calloc(nAllocatedWords, sizeof(*words));
	if (*words == NULL) {
		allocFail("calloc() failed.");
	}
	size_t i = 0;
	size_t wordCount = 0;
	while (i < len) {
		while (isspace(str[i]))
			i++;
		size_t j = i;
		// count until next space.
		while (j < len && !isspace(str[j]))
			j++;			
		// i is the index of the first letter of word
		// j - 1 is the index of the last.
		if (i != j) {
			wordCount++;
			if (wordCount > nAllocatedWords) {
				nAllocatedWords += 10;
				char **tmp = NULL;
				tmp = realloc(*words, sizeof(**words) * nAllocatedWords);
				if (!tmp) {
					allocFail("realloc failed.");
				}
				*words = tmp;
			}
			size_t wordSize = j - i;
			char *tmp = calloc(wordSize + 1, sizeof(*tmp));
			if (!tmp) {
				allocFail("calloc failed.");
			}
			(*words)[wordCount - 1] = tmp;
			strncpy(tmp, &str[i], j - i);
			i = j + 1;
		}
	}
	return wordCount;
}

int main()
{
	char **arr = NULL;
	size_t n = setArray(&arr, str);
	printArray(arr, n);
	freeArray(arr, n);
	return 0;
}

void printArray(char **arr, size_t n)
{
	for (size_t i = 0; i < n; i++) {
		printf("%s\n", arr[i]);
	}
}

void allocFail(const char *msg)
{
	fprintf(stderr, "%s\n", msg);
	exit(EXIT_FAILURE);
}

void freeArray(char **arr, size_t n)
{
	for (size_t i = 0; i < n; i++) {
		free(arr[i]);
	}
	free(arr);
}

JavaScript Code

This is considerably shorter:

#!/usr/bin/env node

const str = "Out of cheese error +++ redo from start";

let res = [];
var setArray = (str, arr) => {
	arr.push(...str.split(" "));
} 

setArray(str, res);
res.forEach((el) => console.log(el));

Inside the setArray() function, we use the split() method to split the string into an array of strings, delimited by a space character.

This is then unpacked to individual elements by the spread syntax operator .... These values are pushed into our array. It is possible to mutate an array within a function like this in JavaScript.

Debugging C and C++ with GDB

Thu, 16 Apr 2020 00:00:00 +0100

Using a debugger allows you to investigate the state of your programme. If you’re working in C or C++, familiarity with a debugger like GDB is more or less essential.

GDB is the GNU debugger - it is a tool that allows you to see what is going on “inside” your target programme while it is executing - or what a programme was doing at the moment it crashed.

Compiling

To use GDB, compile with debugging symbols: these are extra pieces of data that help the tool determine the layout of code and data in memory.

The -g option instructs gcc to compile with debugging symbols. Use -ggdb3 for the maximum amount of debugging information. If compiling in multiple steps, include -ggdb3 at each step.

Run GDB

Once the programme has been compiled with debugging symbols, run gdb from the command line, providing the name of your executable as the sole argument: gdb ./my-prog. If your programme requires arguments, you don’t enter them at this stage.

Commands: Moving Through the Programme

Run

run or r: Run the programme, or possibly restart it. The programme will run until it encounters a condition that causes it to stop - a crash, breakpoint etc.

Start

start: Begin (or restart) the programme. Similar to run, but start is the equivalent of setting a temporary breakpoint at the beginning of the main procedure. The start command stops the programme (e.g. to accept more commands) as soon as execution enters the main function - main() in C and C++.

Step

step or s: Advance the programme one step - step will go into called functions.

next or n: Advance the program one line. Unlike step, if the current line of code is a function call, GDB will execute the entire called function without jumping into the function.

Repeat Commands

Hit enter without entering a command to repeat the last command.

Command Line Arguments

To pass command line arguments, you can either add them after the start or run (r) command (e.g. r someArg anotherArg), or you can use set args to set command line arguments.

Examine Variables

The print and display commands allow you to see what value an expression evaluates to.

Print

print or p: Evaluates an expression and prints the result. Print a variable x: p x to show the value of x at the current position in the programme.

If the expression has side-effects they will affect the state of the program - if you do p x = 3, x is set to 3 and printed.

Format print:

/x: Integer in hexadecimal format
/d: Integer in signed decimal
/u: Integer in unsigned decimal
/o: Integer in octal
/t: Integer in binary (“t” == “two”)
/a: Print as an address

E.g. print variable i in hex format: p/x i.

See here for more options.

When you print the value of an expression, GDB remembers the value in internal variables which are named $1, $2 etc. Gdb shows the internal variable when it prints the value e.g., $1 = 42.

The $ variables can be used in subsequent expressions.

Print from Array

You can print multiple elements from an array - put @number after an lvalue name and GDB prints number values starting at that index.

If A is an array, p A[0]@5 will print the first 5 elements.

Print all elements in array A of length len: p *A@len

Display

display or disp: Automatically display certain expressions each time GDB stops at a breakpoint or after a step. To display the value of i at each breakpoint: display i.

Investigating the State of Your Program

Inspect Stack Frames

GDB allows you to inspect the current set of stack frames and move up and down within them. The backtrace (bt) command lists all of the stack frames with the current one on top, and main on the bottom - showing the function calls that lead up to the current position. The backtrace also shows the line where each call was made.

Gdb uses the variables in the current scope when executing the print command. To inspect variables in other frames further up the stack, select different frames with the up and down command, which move scope up and down the stack.

If the programme stops in a failed assert GDB will stop deep inside the C library - in the code that handles assert. The code under examination is likely to be a number of stack frames up. Use the up command a few times until GDB enters the appropriate frame.

The info command provides information about various aspects of the programme. This command has various subcommands:

info frame describes the memory layout of the current frame
info types describes the types that are in the current program
info b information about breakpoints

There more info options - use help info for more information.

Controlling Execution

Kill GDB

To kill GDB and restart a fresh debugging session without restarting GDB: ki (kill) and enter ‘y’.

Next and Step

Start the programme with start, then step through execution line by line.

The next (n) and step (s) commands advance the state of the program line-by-line.

Breakpoints

A breakpoint instructs GDB to stop execution whenever the program reaches a particular line.

# Breakpoint inside function main() - break inserted just after opening curly brace:
b main
Breakpoint 1 at 0x4007de: file main.c, line 6.

# Breakpoint in file main.c on line 42:

b main.c:42
Breakpoint 2 at 0x4009b5: file main.c, line 43.

Breakpoints are set with the break (b) command, followed by either a line number or a function name (meaning to set the breakpoint at the start of that function).

When you set a breakpoint, GDB assigns it an ID number.

The programme runs (or continues if already started) until the breakpoint. When the breakpoint is encountered, GDB pauses execution and allows you to inspect the state of the programme at this point.

You can enter c to continue from the breakpoint, which jumps to the next breakpoint. You can also use n or s to step through the programme from this point.

By default, breakpoints are unconditional. GDB will stop the program and give you control anytime it reaches the appropriate line.

Conditional Breakpoint

Conditional breakpoints are very useful for looking inside loops and recursive reoutines.

Break on line 7 if i is as specified:

(gdb) break 7 if i==2

Breakpoint Info

Show info on currently set breakpoints:

info b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00000000004007de in main at main.c:6
2       breakpoint     keep y   0x00000000004009b5 in main at main.c:43

Delete all Breakpoints

delete, del or d and enter ‘y’ to confirm.

Delete Specific Breakpoint

Get the ID using info b then d x to delete breakpoint with id x.

Disable all Breakpoints

disable with no arguments to disable all breakpoints. To enable, enter enable x where x is the ID of the breakpoint in question.

Disable Specific Breakpoints

disable x

# disable multiple, space separated:
disable x y

Save and Reload Breakpoints

Save current breakpoints into a file bp-store.txt:

save breakpoints bp-store.txt

To reload:

source bp-store.txt

Backtrace

Run bt after programme interruption to see a backtrace - a list of function calls, representing the call stack. The top line is the frame that was current when the programme was interrupted.

Use the frame command to inspect a specified frame.

Watchpoints

A watchpoint stops execution when the value of an expression changes. The simplest use is to watch the value of a single variable - the command watch i causes a break whenever the value of i changes, printing the old & new values.

Watchpoints can be useful when debugging pointer-related issues.

If the alias (variable) associated with the watchpoint goes out of scope before changing, you can find the relevant GDB variable (e.g. $1) by using print and this (e.g. watch $1).

More info on watchpoints, or enter info gdb and search for watchpoint.

Signals

When a programme receives a signal, GDB will stop the programme and give you control.

SIGSEGV

Indicates a segmentation fault. The programme has attempted to access a restricted area of memory.

Execution stops the programme on the line where the segfault happened. You can inspect the state of the program (printing out variables) to see what went wrong.

SIGABRT

Programme calls abort() or fails an assert. GDB leaves you in control of the programme at the point where assert caused the abort. You may need to move through frames to get back into your own code to debug the fault.

SIGINT

Programme is interrupted - e.g. by the user pressing Control-c. If the programme is getting stuck in an infinite loop, run it in GDB and press Control-c. Once execution has halted use bt to view a backtrace. To examine a particular frame, use the frame command, e.g. (gdb) f 4 to inspect frame 4.

References

Debugging with GDB - online book
GDB Tutorial by Faye Williams - loads of useful GDB stuff on the same site
Watchpoints

Pointer Arithmetic & Multiple Dereferencing in C

Thu, 09 Apr 2020 00:00:00 +0100

It’s useful to really understand pointer arithmetic in C - though in most situations you’d probably be better off using the array subscript operator ([]).

The * dereferencing operator has precedence over the addition + operator.

For the following examples, we’ll use multiple indirection to create an abstract data structure doc to represent this string:

char str[] = "hello world. abc def.\nxyz";

…as a multidimensional array. The string could be represented as a document doc whereby:

doc: contains multiple paragraphs - sequences of characters delimited by newline (‘\n’) characters
paragraph: contains multiple sentences - sequences of characters delimited by fullstop (‘.’) characters
sentence: contains multple words -sequences of characters delimited by space (‘ ‘) characters
word: contains multiple characters, with no spaces and terminated by a NUL character

The underlying data type in this example is a char. This means:

Words are pointers to chars (an array of chars): char *
Sentences are pointers to words (an array of arrays): char **
Paragraphs are pointers to sentences (an array of arrays of arrays): char ***
The Document is a pointer to paragraph(s) - an array of arrays of arrays of arrays: char****.

If you’re trying to wrap your head around multiple indirection, I strongly recommend:

Creating a simple programme with a data structure like the one shown below.
Run the programme with GDB, and attempt to dereference the various elements, as shown at the end of this article.

Data Structure

If you’re more used to using the array index operator [], you can think of the data structure like this:

char *doc[2][5][10]

// alternatively, allocate space dynamically:
// char ****doc = NULL;

// Assume space has been allocated:
doc[0][0][0] = "hello";
doc[0][0][1] = "world";
doc[0][1][0] = "abc";
doc[0][1][1] = "def";
doc[1][0][0] = "xyz";

Resulting Data structure:

char *doc[][10][20] =
// Document
{
	// Paragraph 0
	{
		// Sentence 0
		{"hello", "world"},
		// Sentence 1
		{"abc", "def"}
	},
	// Paragraph 1
	{
		// Sentence 0
		{"xyz"}
	}
};

Dereferencing

Example

*(**doc + 1))

doc: This variable holds the address of the first highest level array - a pointer to first (zero index) paragraph
*doc: Dereference doc to get a pointer to the first sentence in the first paragraph.
**doc: Dereference this to get a pointer to the first word in the first sentence of the first paragraph. This happens next because * has precedence over +.
**doc + 1: Add 1 to this to move the pointer to the second word of the sentence.
*(**doc + 1): Dereference this to get a printable string - a pointer to a null terminated array of chars.

printf("%s\n", *(**doc + 1));
// Output: "world"

Example

*(*(*doc + 1) + 1)

*doc + 1: Dereference doc to get a pointer to first sentence in paragraph 0, then add 1 to get a pointer to sentence 2.
*(*doc + 1) + 1: Dereference the pointer to sentence to get a pointer to first word, add 1 to get a pointer to second word.
*(*(*doc + 1) + 1): Dereference this to print the string.

printf("%s\n", *(*(*doc + 1) + 1));
// Output: "def"

General Dereferencing Heuristic

//                          para index 0
//                          |
//                          |    sentence index 1
//                          |    |
//                          |    |    word index 0
//                          |    |    |
//                          |    |    |    character index 1
//                          ↓    ↓    ↓    ↓ 
char letter = *(*(*(*(doc + 0) + 1) + 0) + 1);
// equivalent:
char letter = doc[0][1][0][1];

More Examples

// GDB output
>>> p **(*(*doc+1)+1)
$112 = 100 'd'

>>> p *(*(*(*doc+1)+1)+1)
$113 = 101 'e'

>>> p *(*(*(*doc+1)+1)+2)
$114 = 102 'f'
>>>

char ****doc;

doc[0][0][0] == "hello"
doc[0][0][1] == "world"
doc[0][1][0] == "abc"
doc[0][1][1] == "def"

// GDB output
>>> p *(*(*doc+0)+0)
$117 = 0x602680 "hello"

>>> p *(*(*doc+0)+1)
$118 = 0x6026a0 "world"

*doc: Dereference doc once, giving a pointer to the first sentence in the first paragraph.
*doc + 1: Add 1 to this value - providing a pointer to the second sentence.
*(*doc + 1): Dereference this, giving a sentence (an array of words).
*(*doc + 1) + 0: Add zero to pointer - which still points to the first array of words (sentence).
*(*(*doc + 1) + 0): Dereference this to print, generating tdoche first word in the second sentence of the first paragraph

// GDB output
>>> p *(*(*doc+1)+0)
$119 = 0x6026c0 "abc"


>>> p *(*(*doc+1)+1)
$120 = 0x6026e0 "def"

Build Python from Source in Linux

Tue, 24 Mar 2020 00:00:00 +0000

Get the latest version of Python on Ubuntu

Download the latest version of Python from here: https://www.python.org/downloads/

# Download Python:
wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz

# Download the detached signature file:
wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz.asc

Verify Download

Checksums (MD5) are presented on the dowloads/releases/python-xxx/ page.

Check the md5sum manually by running:

md5sum Python-3.8.2.tgz

You can also automate the check by building a suitable manifest file and checking this:

# Get md5 checksum from website, filename corresponds to the download
cat > manifest
f9f3768f757e34b342dbc06b41cbc844  Python-3.8.2.tgz

# Ctrl-d to exit cat
md5sum -c checksum 2>&1 | grep OK 
# Output if correct:
Python-3.8.2.tgz: OK

The website also posts the filesize, so you should also check this.

Import Relevant GPG Key From Trusted Server

# Imports key sourced here: https://www.python.org/downloads/
gpg --keyserver keyserver.ubuntu.com --recv-keys B26995E310250568

Check that the downloaded file has been signed by this key:

# Reference the signature file
gpg --verify Python-3.8.2.tgz.asc
# Correct output:
gpg: assuming signed data in `Python-3.8.2.tgz'
gpg: Signature made Tue 25 Feb 2020 10:41:46 GMT using RSA key ID 10250568
gpg: Good signature from "Łukasz Langa (GPG langa.pl) <lukasz@langa.pl>"
gpg:                 aka "Łukasz Langa <lukasz@python.org>"
gpg:                 aka "Łukasz Langa (Work e-mail account) <ambv@fb.com>"
gpg:                 aka "[jpeg image of size 24479]"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: E3FF 2839 C048 B25C 084D  EBE9 B269 95E3 1025 0568

Extract

tar -xvf Python-3.8.2.tgz

This unpacks files into a directory (in this case, Python-3.8.2) which you should move into:

cd Python-3.8.2

Check out the README.rst.

Build

./configure --enable-optimizations
make
make test
sudo make install

This will install Python as python3.

Memoization when Computing Fibonacci Sequence in C

Mon, 23 Mar 2020 00:00:00 +0000

The objective of this exercise is to compute a Fibonacci sequence up to a target number of elements, saving the sequence as an array.

The Fibonacci sequence is the sequence of numbers such that each number is the sum of the two preceding numbers, starting from 0 and 1. For example, the first 6 Fibonacci numbers are:

0, 1, 1, 2, 3, 5

The exercise contrasts two methods:

Iterative, using a for loop and computing values in order
Recursive, using dynamic programming technique (memoization) to improve efficiency

Iterative

The iterative method is straightforward - loop from zero up to the target, setting the current element to:

0 if the current element is at index 0.
1 if the current element is at index 1.
The sum of the elements at the previous two adjacent indices otherwise.

void iterativeFibonacci(int *resArray, int n)
{
	for (int i = 0; i < n; i++) {
		resArray[i] = i == 0 || i == 1 ? i : resArray[i-1] + resArray[i-2];
	}
}

Recursive, with Dynamic Programming Caching

Simple recursive calls (in a tree structure) would involve multiple repeat calls performing the same calculation.

Computing the 4th number in the Fibonacci sequence would involve calling:

fib(4) once
fib(3) once
fib(2) twice
fib(1) three times
fib(0) twice

fib(4)
|     \
|      \
|       \
|        \
fib(3)	  fib(2)--
|    \     \      \
|     \	    \      \
|      \     \      \	
fib(2) fib(1) fib(1) fib(0)
|    \
|     \
|      \
fib(1)	fib(0)

For example, a naive recursive implementation in C looks like this:

int fib(int n)
{
	if (n < 2) {
		return n;
	}
	return fib(n - 2) + fib(n - 1);
}

The number of function calls grows out of proportion as you calculate higher numbers in the sequence: computing the 10th number in the Fibonacci sequence calls fib() 177 times - computing the 20th number calls fib() 21891 times. Each call to the function requires a stack frame, so this approach is quite inefficient both in terms of time and space complexity.

We can do better by storing results as we go - a dynamic programming technique called memoization.

Memoization

This can be implemented by using an array to hold successive numbers in the sequence. For each recursive call, we check to see if the value has already been computed by looking in the cache. If has been previously computed, we return this value. Otherwise, we perform the computation and add this to the cache.

#include <stdio.h>
#include <stdlib.h>

#define N 10
static size_t count = 0;

void iterativeFibonacci(int *resArray, int n)
{
	for (int i = 0; i < n; i++) {
		resArray[i] = i == 0 || i == 1 ? i : resArray[i-1] + resArray[i-2];
	}
}

/**
* Build a cache representing the Fibonacci sequence.
* Note that the second parameter is an array index which allows the cache to
* be checked for the required element. 
*/
int recursiveFibonacci(int *cache, int n)
{
	count++; // For analysis only

	// If the cache holds -1 at the required index, it has not yet been computed.
	if (cache[n] == -1) {
		cache[n] = recursiveFibonacci(cache, n - 1) + recursiveFibonacci(cache, n - 2);
	}
	return cache[n];
}

void printArray(int *arr, int n) {
	for (int i = 0; i < n; i++) {
		printf(i == 0 ? "[ %d" : ", %d", arr[i]);
	}
	puts(" ]\n");
}

int main(void)
{
	int r[N];
	iterativeFibonacci(r, N);
	printArray(r, N);
	
	// Initialise an array of N elements, each element set to -1
	// Note that this is a GNU extension to the GCC compiler
	int cache[N] = { [0 ... N-1] = -1 };

	// Set the first two elements in the sequence, which are known
	cache[0] = 0;
	cache[1] = 1;

	// The function receives a pointer to the cache array and the index of the last element.
	recursiveFibonacci(cache, N - 1);
	printArray(cache, N);
	printf("recursiveFibonacci() called %lu times.\n", count);
	return 0;
}

Using this method, computing the 20th number in the Fibonacci sequence requires 37 calls to recursiveFibonacci(), compared with the 21891 calls that are required if caching is not used. We also have the additional benefit of having the entire sequence up to and including our target in our cache array, for very little extra cost.

References

Binary Search in Rust

Sun, 22 Mar 2020 00:00:00 +0000

Binary search is an algorithmically efficient way of searching an ordered array. It works by comparing the target value with the middle element of the array. If the two are not equal, the array is partitioned and re-searched, this time excluding the elements that are either too high or too low.

For example, if the target value is 42, and the middle element of an ordered array is 100, the target cannot lie in the elements from the mid point to the end of the array. This allows us to reset the search bounds, eliminating half of the previous array at each step.

Binary search in Rust: Iiterative

Arguments: Array slice, array length, target integer to look up.

The initial value for low (index of the first array element) is 0.

The initial value for high (index of the last array element) is length - 1.

Algorithm

Determine the index of the middle element if length is odd or the lowest of the pair of middle elements if length is even.
The mid index is ⌊(high - low) / 2⌋ + low.
Compare the value at this index with the target.
If the values are the same, return the index.
- 4a. If the target > value, search elements at higher indices (because elements indexed by these have higher values). Set low to mid + 1
- 4b. If the target < value, search elements with lower indices (these have lower values). Set high to mid - 1
Keep doing steps 1 to 5 until the value of low is greater than that of high.
If the function has not returned an index by this point, the returned option will be None.

To fully search the array, the type of the high variable be such that the variable can hold a negative value.

Rust

Rust is really strict about types and casting - there is no implicit type conversion between primitives.

Because of this, and the fact that high must be able to handle negative numbers, computing mid_index firstly involves manipulating i8 integers to perform the arithmetic, before overtly casting the result into a usize type that can be used to index the array.

Note that the Option type is ideal for the function return value - it allows us to return (and handle) a None option if the target does not exist in the array. In languages without this capability, we might return -1 to indicate non-membership of the array - but this is not as overtly descriptive as None!

Code

The example below also includes a couple of tests.


// Binary search, iterative.
// Arguments: Array slice, array length, target integer to look up.

pub fn iterative(a: &[i32], len: usize, target_value: &i32) -> Option<usize> {
    let mut low: i8 = 0;
    let mut high: i8 = len as i8 - 1;

    while low <= high {
        let mid = ((high - low) / 2) + low;
        let mid_index = mid as usize;
        let val = &a[mid_index];

        if val == target_value {
            return Some(mid_index);
        }

        // Search values that are greater than val - to right of current mid_index
        if val < target_value {
            low = mid + 1;
        }

        // Search values that are less than val - to the left of current mid_index
        if val > target_value {
            high = mid - 1;
        }
    }
    None
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn correct_iterative() {
        let correct_arr = [
		1, 10, 20, 47, 59, 63, 75, 88, 99,
		107, 120, 133, 155, 162, 176, 188,
		199, 200, 210, 222
		];
        for i in 0..correct_arr.len() {
            assert_eq!(i, iterative(&correct_arr, correct_arr.len(), &correct_arr[i]).unwrap());
        }
    }
    #[test]
    fn incorrect_iterative() {
        let searched_arr = [
		1, 10, 20, 47, 59, 63, 75, 88, 99,
		107, 120, 133, 155, 162, 176, 188,
		199, 200, 210, 222
		];
        let incorrect_arr = [
		2, 22, 48, 58, 61, 73, 84, 90, 100,
		119, 132, 154, 160, 177, 187, 197,
		201, 211, 2242
		];
        for i in 0..incorrect_arr.len() {
            assert_eq!(None, iterative(&searched_arr, searched_arr.len(), &incorrect_arr[i]));
        }
    }
}

Calling code:

fn main() {
    let arr = [
		1, 10, 20, 47, 59, 63, 75, 88, 99,
		107, 120, 133, 155, 162, 176, 188,
		199, 200, 210, 222
		];
    let target: i32 = 47;
    if let Some(result) = rust_binary_search::iterative(&arr, arr.len(), &target) {
        println!("{} found at index {}", target, result);
    } else {
        println!("{} not found.", target);
    }
}

Similar Functionality in C++

Note that it returns -1 if the target value is not found in the array.

// Iterative algorithm: binary search
// 1. Set values for low and high indices - starting condition, low == 0, high == len -1.
// 2. Determine the mid index (or leftmost mid index if length is even)
// 3. If value at the mid index == target value, return the index.
// 4. If value at the mid index > target value, make a partition of elements with indices < mid index.
// 5. If value at the mid index < target value, make a partition of elements with indices > mid index.
// 6. Repeat until:
//	- low > high
int binarySearchIterative(int A[], int len, int targetVal)
{
	int high = len - 1;
	int low = 0, midIndex = 0;
	while (low <= high) {
		midIndex = ((high - low) / 2) + low;
		int val = A[midIndex];
		if (val == targetVal) {
			return midIndex;
		}
		if (val > targetVal) {
			high = midIndex - 1;
			continue;
		}
		if (val < targetVal) {
			low = midIndex + 1;
			continue;
		}	
	}
	return -1;	
}

Some or None - Handling Rust Options

Sat, 21 Mar 2020 00:00:00 +0000

The Rust Ownership Model

Sat, 21 Mar 2020 00:00:00 +0000

Rust manages heap memory through a system of ownership, whereby the compiler checks a set of rules at compile time.

Ownership features do not slow down the compiled code - if the code has compiled, it has satisfied all necessary ownership requirements.

Ownership Rules

Each value has a variable called it’s owner.
There can only be one owner at a time.
When the owner goes out of scope, the value is dropped.

If a value is heap allocated (i.e. it’s size is not known at compile time), copies are shallow by default and the original is invalidated. This is like a C++ move.

When a variable that includes heap-allocated data goes out of scope, the value is cleaned up by drop unless the data has been moved - i.e. another variable has taken ownership.

Rules of Borrowing

Unlimited borrows for read-only borrows: let a = &x
For a read only borrow, the original data is immutable for the duration of the borrow
You can only have a single borrow at a time for mutable borrows: let a = &mut x

Example Borrow

fn main() -> Result<(), &'static str> {
    let mut a = [1,2,3,4];
    println!("{:?}", a); // Line 1 output
    
    {
        let b = &mut a[0..2];
        // You can't access a at this point because it has been mutably borrowed
        // from. The following line won't compile, with the error message:
        // `cannot borrow `a` as immutable because it is also borrowed as mutable`:
        // println!("a: {:?}", a);

        println!("b: {:?}", b); // Line 2 output
        b[0] = 42;
        println!("b: {:?}", b); // lIne 3 output
    }

    // The borrow is scope-dependent - a is accessible again because the 
    // variable that borrowed it is out of scope:
    println!("a: {:?}", a); // Line 4 output
    Ok(())
}

Output:

[1, 2, 3, 4]
b: [1, 2]
b: [42, 2]
a: [42, 2, 3, 4]

Rust References

Two kinds:

Shared reference: &
Mutable reference: &mut

A reference cannot outlive it’s referent.

A mutable reference cannot be aliased.

Note that aliased is not yet defined.

The Rules of Referencing

At any given time, you can have either one mutable reference or any number of immutable references.
References must always be valid - dangling pointers won’t compile - the thing being referred to must be in scope.

Aliasing

Variables and pointers alias if they refer to overlapping regions of memory.

References

FizzBuzz in C - without the Modulus Operator

Sat, 21 Mar 2020 00:00:00 +0000

Code for this article can be found here.

FizzBuzz - iterate over a range of numbers and print:

“Fizz” if the number is divisible by 3.
“Buzz” if the number is divisible by 5.
“FizzBuzz” if the number is divisible by both 3 and 5.
The number if it is not divisible by either 3 or 5.

For the purposes of this exercise, use of the modulus operator % is disallowed. There’s no particular reason for this - it’s just an exercise.

Divisibility by Three

This can be determined by comparing the sum of odd binary digits (mod 3) with the sum of even binary digits (mod 3). If these values are equal, then the number is divisible by 3.

Example 1

Represented in binary digits, “even” bits marked:

↓   ↓   ↓
1 0 1 0 1 0

Sum of odd bits = 0 + 0 + 0 ≡ 0 mod 3

Sum of even bits = 1 + 1 + 1 ≡ 0 mod 3

These are equal, so 42 is divisible by 3.

Example 2

Represented in binary digits, “even” bits marked:

↓   ↓   ↓
1 0 1 0 1 1

Sum of odd bits = 0 + 0 + 1 ≡ 1 mod 3

Sum of even bits = 1 + 1 + 1 ≡ 0 mod 3

These are not equal, so 42 is divisible by 3.

This algorithm still requires you to calculate the modulus of the sum of bits. Because the operand will have limited size, we can compute this by repeated subtraction. For a 32 bit integer, the maximum size of the operand will be 16 (i.e. the sum of 16 bits) which requires a maximum 5 subtractions.

#include <stdio.h>
#include <stdint.h>
#include "integer-input.h"

#define BITS_IN_BYTE 8

/**
 * Determine mod 3 of input without using modulus operator.
 * Input should not be a large number - in this context it will
 * be at most 16 (assuming a 4 byte integer, this function will
 * receive at most the sum of 16 bits (i.e. 16)
 * */
uint8_t mod_3(int input)
{
	uint8_t result = 0;
	while (input >= 3) {
		input -= 3;
	}
	result = input;
	return result;
}

/**
 * Check that an integer x is divisible by 3 without using the modulus
 * operator.
 *
 * For a number to be divisible by 3, the sum of the odd bits mod 3 should equal
 * the sum of the even bits mod 3.
 * */
uint8_t divByThree(int x)
{
	unsigned mask_1 = 1u;
	unsigned mask_2 = 2u;
	int oddSum = 0;
	int evenSum = 0;
	// Because the mask shifts 2 bits to the left each iteration,
	// we only need to iterate half the number of bits times.
	for (size_t i = 0; i < (sizeof(x) * BITS_IN_BYTE) / 2; i++) {
		if (x & mask_1) {
			oddSum++;
		}
		if (x & mask_2) {
			evenSum++;
		}
		mask_1 <<= 2;
		mask_2 <<= 2;
	}
	return mod_3(oddSum) == mod_3(evenSum) ? 1 : 0; 
}

An alternative algorithm to determine divisibility by three:

Sum all decimal digits in the number.
Repeat the summation of digits until a single digit remains.
If this value is divisible by 3, the original number is divisible by 3.

Divisibility by Five

A number is divisible by five if it’s least significant decimal digit is either 5 or 0:

/**
 * A number is divisible by 5 if the least significant decimal digit is either 5 or 0.
 * */
uint8_t divByFive(int x)
{
	uint8_t finalDecimalDigit = x - ((x / 10) * 10);
	return finalDecimalDigit == 5 || finalDecimalDigit == 0 ? 1 : 0;
}

FizzBuzz

Both our divisibility functions return 1 if the number is divisible and 0 if not. This makes it straightforward to perform the necessary logic to implement FizzBuzz:

void fizzBuzz(int inputInteger)
{
	for (uint8_t n = 1; n <= inputInteger; n++) {
		if (divByFive(n) && divByThree(n)) {
			printf("%d: FizzBuzz\n", n);
			continue;
		}
		if (divByThree(n)) {
			printf("%d: Fizz\n", n);
			continue;
		}
		if (divByFive(n)) {
			printf("%d: Buzz\n", n);
			continue;
		}
		printf("%d\n", n);
	}
}

References

GitHub Repo for this article

Declare Default Values for Mutable Types in Python

Fri, 20 Mar 2020 00:00:00 +0000

If you want to specify a default for a function argument of an immutable type in Python, you assign the default in the function declaration.

You can’t assign a default value for an immutable type in this way. Instead, you can assign a default value None and check for this inside the function - if the passed-in argument is None, then assign a default:

#!/usr/bin/env python3
class Node:
    def __init__(self, children: dict=None, is_end=False, text=''):
        # If a dict is passed in, set self.children to this, otherwise
	# set it to an empty dict.
	self.children = children if children else dict()
        self.is_end = is_end
        self.text = text

    def print(self):
        print("is_end: {}, text: {}".format(self.is_end, self.text))
        for key in self.children:
            print("Key: {} Value: {}".format(key, self.children[key]))

def main():
    a = Node()
    b = Node({'a': 'value'}, True, 'hello!')

    a.print()
    b.print()

if __name__ == '__main__':
    main()

Convert decimal to hexadecimal in the Python REPL

Fri, 13 Mar 2020 00:00:00 +0000

Convert a space-separated string of decimal integers into a space separated string of hexadecimal integers in the Python REPL.

Starting string:

137 60 228 95 213 195 161 98 210 106

# Make a list of integers:
l = "137 60 228 95 213 195 161 98 210 106".split(' ')

# Make a list of hex strings, without leading 0x characters:
h = [str(hex(int(i)))[2:] for i in l]

# Put the list back into a string
hs = ' '.join(h)

# Output:
print(hs)

The final output is:

89 3c e4 5f d5 c3 a1 62 d2 6a

One liner (yikes!):

print(' '.join([str(hex(int(i)))[2:] for i in "137 60 228 95 213 195 161 98 210 106".split(' ')]))

Initialize a C Array with Values

Tue, 10 Mar 2020 00:00:00 +0000

Initialize all members of a C array to the same value.

For the GCC compiler, you can initialize an element to a particular value by specifying the array index and associated value.

For example:

...
int a[5] = { [1] = 42, [4] = 42 };

// Equivalent to:
int a[5] = { 0, 42, 0, 0, 42 };

You can also intialise a range of elements by using the format [first ... last] = value

This is a GNU extension.

Initialise All Elements to the Same Value

The following complete example shows how this technique can be used to initialize all elements in an array to the same value:

#include <stdio.h>
#define N 5

void printArray(int *arr, int n) {
	for (int i = 0; i < n; i++) {
		printf(i == 0 ? "[ %d" : ", %d", arr[i]);
	}
	puts(" ]\n");
}

int main()
{
	// Incorrect method
	int arr[N] = { 42 };
	printArray(arr, N);
	// Output: [ 42, 0, 0, 0, 0 ]
	
	int arr2[N] = { [0 ... N - 1] = 42 };
	printArray(arr2, N);
	// Output: [ 42, 42, 42, 42, 42 ]

	return 0;
}

References

GCC Documentation

Return a Result Type from the Main Function in Rust

Wed, 04 Mar 2020 00:00:00 +0000

The main() function in a Rust programme can return a Result type, which allows you to provide feedback to users as well as setting the appropriate exit codes for the programme.

Example

use std::num::ParseIntError;

fn main() -> Result<(), ParseIntError> {
    let number_str = "this can't be parsed as a number";
    let number = match number_str.parse::<i32>() {
        Ok(number)  => number,
        Err(e) => return Err(e),
    };
    println!("{}", number);
    Ok(())
}

The above programme prints Error: ParseIntError { kind: InvalidDigit } to stdout.

When the parse::<i32>() method returns an error:

The match expression enters the Err() branch
The error is passed into the branch
Finally another Err() is returned - the error is output to stderr and the function (and therefore the programme) is terminated.

Note that in Linux the exit status of running this programme is 1, denoting that the command did not complete successfully.

Here’s another example:

/**
* Contrived function, returns an error if the values are the same.
**/
fn get_largest(a: u32, b: u32) -> Result<u32, &'static str> {
    if a == b {
        return Err("equal values");
    }
    if a > b { Ok(a) } else { Ok(b) }
}

fn main() -> Result<(), &'static str> {
    let a = 42;
    let b = 42;
    let number = match get_largest(a, b) {
        Ok(number)  => number,
        Err(e) => return Err(e),
    };
    // If no error, use `number`.
    println!("{}", number);
    Ok(())
}

C++ Articles & Resources

Tue, 25 Feb 2020 00:00:00 +0000

Collection of links to C++ articles that I think are worth reading.

OOP

Express intentions (to users & compiler) with override.

Virtual, final and overrride in C++, Jonathan Boccara

Declaring Classes

Rationale for a particular way of declaring a class. Discusses the importance of the order of declarations, and consistency:

How I Declare My Class and Why, Howard E Hinnant

Move Semantics

Eveything you ever Wanted to Know About Move Semantics, Howard Hinnant

Rust Vectors

Sun, 02 Feb 2020 00:00:00 +0000

The vec! macro makes initializing vectors convenient:

fn main() {
    let expected = vec![
        0x77, 0xd6, 0x57, 0x62, 0x38, 0x65, 0x7b, 0x20,
        0x3b, 0x19, 0xca, 0x42, 0xc1, 0x8a, 0x04, 0x97,];
    print_vec(&expected);
}

Initialize a vector of size n = 5 setting each element to 42:

let mut my_vec = vec![42; 5]; // Note semicolon seperator

Get an index within a for loop using .iter().enumerate():

fn print_vec(v: &Vec<u8>) {
    print!("[");
    for (i, el) in v.iter().enumerate() {
        if i != 0 {print!(", ")}
        print!("{}", el);
    }
    print!("]");
}

Initialize a Vector with Values

fn main() {
    let expected = vec![
        0x77, 0xd6, 0x57, 0x62, 0x38, 0x65, 0x7b, 0x20,
        0x3b, 0x19, 0xca, 0x42, 0xc1, 0x8a, 0x04, 0x97,];
    
    print_vec(&expected);

    // Initialize a 5 element vector, each element set to 42
    let my_vec = vec![42; 5];
    print_vec(&my_vec);
}

Print Vector

fn print_vec(v: &Vec<u8>) {
    print!("[");
    for (i, el) in v.iter().enumerate() {
        if i != 0 {print!(", ")}
        print!("{}", el);
    }
    print!("]\n");
}

Initialize an Array

// Array of 32 u8, each element set to 0
let mut result: [u8; 32] = [0; 32];

Passing by Reference in Rust

Sun, 02 Feb 2020 00:00:00 +0000

Example of passing arguments by reference in Rust:

fn main() {
  // Define a mutable variable and a reference to it
  let mut n_main: usize = 100;
  let n_main_ref: &usize = &n_main;

  // Prints the return value of `increment_value`, original variable unchanged
  println!("{}", increment_value(n_main_ref)); // "101"
  println!("n_main = {}", n_main); // "n_main = 100"

  // This function receives a mutable reference, and changes the original value.
  amend(&mut n_main);
  println!("n_main = {}", n_main); // "n_main = 199"
}

// Accepts a mutable reference, changes the value it refers to, no return value
fn amend(n_inc_ref: &mut usize) {
    *n_inc_ref +=99;
}

// Accepts a reference, does not mutate it, returns same type as the passed-in reference
fn increment_value(n_inc_ref: &usize) -> usize {
  n_inc_ref + 1
}

Sync Forked Repo With Upstream

Thu, 23 Jan 2020 00:00:00 +0000

Once you’ve forked and cloned a repo the following describes how to sync the local fork with the upstream remote.

Configure Remote

Check current remotes:

git remote

This will probably output origin, denoting that your repo has a single remote.

Configure a Git remote for the upstream (original) repo:

git remote add upstream https://github.com/original-project/original-project.git

Check:

git remote

# Output:
origin
upstream

Sync Local Fork with Upstream

There are now three repositories:

local
origin (GitHub, Bitbucket etc)
upstream (The original repo)

Sync from upstream to local, then push changes to origin:

# fetch changes
git fetch upstream

# switch to master branch
git checkout master

# Merge changes from upstream into master
git merge upstream/master

# Push local changes to origin
git push

Bitcoin RPC Through SSH Tunnel

Sat, 18 Jan 2020 00:00:00 +0000

Bitcoin Core functions as a HTTP JSON-RPC server by means of the bitcoind programme. This means that Bitcoin Core can be controlled remotely by means of HTTP requests.

JSON-RPC is a lightweight remote procedure call protocol. It involves a client sending requests in JSON format to a server. The server (in this case, bitcoind) executes the required function and sends a serialized JSON object back to the client as a response.

Authentication in bitcond is by means of basic HTTP authentication. A username/password combination is set on the server, and requests must supply this data. Because the connection is not HTTPS/TLS, the username/password are exposed in the request. This isn’t a problem if the connection is made on the same computer that is running bitcoind but if the client accesses the server over the network, it opens a security vulnerability.

SSH tunneling fixes this by sending all data over an encrypted channel.

Why Use an SSH Tunnel?

SSH is a standard for secure remote login and file transfer over untrusted networks. SSH can be set up to use port forwarding to tunnel any TCP/IP port over SSH. Because data flows over an SSH connection, it is encrypted in transit. This makes SSH tunneling an obvious choice when sending and receiving sensitive data.

This example involves connecting to a remote bitcoind RPC-JSON server over an SSH tunnel. The application does not allow HTTPS connection. Without SSH tunneling (or similar encrypted channel) sensitive information such as authentication data, addresses and Bitcoin amounts would be sent over a public network in cleartext.

An SSH tunnel can be set up to forward traffic sent to a specified port on localhost to a specified port on a remote host. Forwarding a port from the client machine to the server machine in this way is known as Local Forwarding.

Setting Up the JSON-RPC Server

You can either pass command-line options when starting bitcoind to enable the RPC interface and control other necessary settings, or you can add configuration data in a config file.

The config file approach is probably the most practical, and is the method covered in this article.

When started, bitcoind looks for a configuration file named bitcoin.conf in the bitcoin data directory (under Linux, this is ~/.bitcoin by default). You’ll need to specify:

The IP/IP range that is allowed access
Authentication credentials

Allow/Restrict IP Address

By default, bitcoind only allows RPC connections from localhost. You can ease this restriction by specifying an IP address in the rpcallowip field in the ~/.bitcoin/bitcoin.conf configuration file.

Example bitcoin.conf allowing RPC access from the 192.168.122 subnet to the regtest network:

# /path/to/bitcoin.conf

# These settings apply to the regtest network only
[regtest]
rpcbind=0.0.0.0
rpcallowip=192.168.122.0/24 # Allow access fromm the 192.168.122 subnet
server=1 # Tells the bitcoin server to accept JSON-RPC commands
rpcuser=yourusername # Required for the JSON-RPC API
rpcpassword=yourpassword # Required
rest=1

The example shows a config block for the regtest network. To set up access for mainnet, you would provide a similar configuration block for [main].

Security of the JSON-RPC Server

The JSON-RPC server requires basic HTTP authentication. For example, to send a request using curl:

curl --user alice:password123 \
--data-binary '{"jsonrpc":"1.0","id":"curltext","method":"listunspent","params":[]}' \
-H 'content-type:text/plain;' http://192.168.122.18:18443/ \

In this case, the username is alice and the (obviously inadequate) password is ‘password123’.

The request is made to http://192.168.122.18:18443 - a virtual machine that shares a network with the client machine making the request. In this case, the local network IP address of the client machine is 192.168.122.1.

The request sends the method listunspent, which is processed by Bitcoin Core and the result returned as a JSON object.

Authentication Data is Sent in Plaintext

Authentication data (user name and password) is sent as base 64 encoded plaintext. This example shows the HTTP header packets including the authorization data.

0000:  504f 5354 202f 2048 5454 502f 312e 310d  POST / HTTP/1.1.
0010:  0a48 6f73 743a 2031 3932 2e31 3638 2e31  .Host: 192.168.1
0020:  3232 2e31 383a 3138 3434 330d 0a41 7574  22.18:18443..Aut
0030:  686f 7269 7a61 7469 6f6e 3a20 4261 7369  horization: Basi
0040:  6320 5957 7870 5932 5536 6347 467a 6333  c YWxpY2U6cGFzc3
0050:  6476 636d 5178 4d6a 4d3d 0d0a 5573 6572  dvcmQxMjM=..User
0060:  2d41 6765 6e74 3a20 6375 726c 2f37 2e34  -Agent: curl/7.4
0070:  372e 300d 0a41 6363 6570 743a 202a 2f2a  7.0..Accept: */*
0080:  0d0a 636f 6e74 656e 742d 7479 7065 3a74  ..content-type:t
0090:  6578 742f 706c 6169 6e3b 0d0a 436f 6e74  ext/plain;..Cont
00a0:  656e 742d 4c65 6e67 7468 3a20 3638 0d0a  ent-Length: 68..
00b0:  0d0a                                             ..

These packets were captured on the server hosting bitcoind, but they show how the data is transmitted over the network.

Note that in the above block YWxpY2U6cGFzc3dvcmQxMjM= represents the base64 encoded user:password combination - in this case, alice:password123. This is not encrypted - it’s simply a base64 representation of the ASCII characters. See for yourself by decoding here. Obviously this is a security vulnerability - an eavesdropper can easily determine the user name and password.

In the case of basic HTTP authentication like this, the connection is not secure unless the exchange takes place over HTTPS(TLS).

The Bitcoin RPC API does not allow connection via SSL - this functionality was dropped in 2015.

Luckily, we can easily fix this by using an SSH Tunnel to connect.

SSH Tunnel

If the computer hosting bitcoind is set up to communicate with a client machine by means of SSH, it is pretty straightforward to set up an SSH tunnel.

Set up SSH local port forwarding on the client machine that will access the bitcoind server - i.e. the computer that will send HTTP requests to the RPC server:

ssh -v -fNL 5555:192.168.122.18:18443 remote_user@192.168.122.18

To send traffic through the ssh tunnel, you now use the 127.0.0.1:5555 host:port combination.

The same data shown above now results in the following packet - approximately, since the data is encrypted and there is no way to tell which bytes constitute the HTTP header:

20:43:28.404868 IP _gateway.44348 > donnager.ssh: Flags [P.], seq 100:376, ack 45, win 296, options [nop,nop,TS val 5081969 ecr 985853552], length 276
0000:  4510 0148 828f 4000 4006 41ac c0a8 7a01  E..H..@.@.A...z.
0010:  c0a8 7a12 ad3c 0016 af75 982e c79e cd40  ..z..<...u.....@
0020:  8018 0128 f86f 0000 0101 080a 004d 8b71  ...(.o.......M.q
0030:  3ac2 ee70 fb68 ceb4 aa74 80d3 aef4 f56b  :..p.h...t.....k
0040:  3794 e39d 9468 888a 7aa4 44d4 377e b1b3  7....h..z.D.7~..
0050:  9a87 f91a d4eb 9650 cb7a 2906 ae71 fca8  .......P.z)..q..
0060:  bd0b c8ec bbe6 01f0 42d1 6c11 e560 ddae  ........B.l..`..
0070:  817e 2fa0 df1c 6f70 ca19 b5d9 bb8e 0dca  .~/...op........
0080:  f5ff 0d3e ab69 a43b 1cc6 cfb7 e8f2 1bca  ...>.i.;........
0090:  cae2 14c4 85a7 f06f 33be f097 f904 713b  .......o3.....q;
00a0:  13cf a482 19d4 b3a2 6ede 887d 2749 b897  ........n..}'I..
00b0:  ed27 3deb d8a2 a60e 96ee 074f 085f d134  .'=........O._.4

Python Example

It is fairly straightforward to wrap the SSH tunnel and request process in a Python programme, as this basic example shows:

SSH Tunnel

# ssh_tunnel.py
import subprocess
from threading import Thread

class SSHTunnel(Thread):
    def __init__(self, local_port, remote_port, remote_user, remote_host):
        Thread.__init__(self)
        self.local_port = local_port
        self.remote_port = remote_port
        self.remote_user = remote_user
        self.remote_host = remote_host
        self.daemon = True

    def run(self):
        if subprocess.call([
            'ssh',
            '-N',
            '-L',
            str(self.local_port) + ':' + self.remote_host + ':' + str(self.remote_port),
            self.remote_user + '@' + self.remote_host
            ]):
            raise Exception("ssh tunnel setup failed.")

Configuration Data

{
	"local_port": 5432,
	"remote_port": 18443,
	"remote_user": "alice",
	"remote_host": "111.222.43.232",
	"rpc_user": "alice",
	"rpc_password": "password123"
}

Request Through the Tunnel

#!/usr/bin/env python3
from ssh_tunnel import SSHTunnel
import requests, time, json, sys
from pygments import highlight, lexers, formatters

def main(config):
    with open(config) as file:
        config_data = json.load(file)
    tunnel = SSHTunnel(
            config_data["local_port"],
            config_data["remote_port"],
            config_data["remote_user"],
            config_data["remote_host"]
            )
    tunnel.start()
    time.sleep(1)

    headers = {'content-type': 'text/plain;'}
    data = '{"jsonrpc":"1.0","id":"curltext","method":"listunspent","params":[]}'
    response = requests.post(
            "http://127.0.0.1:{}/".format(config_data["local_port"]),
            headers=headers, data=data, auth=(config_data["rpc_user"], config_data["rpc_password"])
            )
    response_dict = response.json()
    my_vals = [{
        'txid': x['txid'],
        'amount': x['amount'],
        'address': x['address'],
        'vout': x['vout']} for x in response_dict['result']]
    print(highlight(json.dumps(my_vals, indent=2), lexers.JsonLexer(), formatters.TerminalFormatter()))

if __name__ == '__main__':
    if len(sys.argv) != 2:
        print("Please pass a path to a config file.")
    else:
        main(sys.argv[1])

GitHub repo for this example

References & Resources

Fast Modular Exponentiation

Thu, 12 Dec 2019 00:00:00 +0000

Modular exponentiation is used in public key cryptography.

It involves computing b to the power e (mod m):

c ← b^e (mod m)

You could brute-force this problem by multiplying b by itself e - 1 times and taking the answer mod m, but it is important to have fast (efficient) algorithms for this process to have any practical application.

In cryptography, the numbers involved are usually very large. Without an efficient algorithm, the process would take too long.

This article is educational - it is a summary of what I have learned about the process of modular exponentiation, with a few code implementations of a possible algorithm rather than a presentation of the most efficient methods.

Naive Exponentiation

THe most basic approach would be to multiply b by itself e - 1 times, and performing floor division on the result to obtain the result modulo m (which will be the remainder or residue of floor division of the result by the modulus). There are a few problems with this approach:

If b and e are large numbers, b^e will be enormous - which causes problems representing the resultant data as a native type in many languages/systems.
If b and e are large, a lot of multiplications are required. The universe might not last long enough for us to carry out the computation…

Slightly Less Naive Approach

For multiplication (mod m) congruence is maintained. In other words:

if a ≡ x (mod m) then a ∙ k ≡ x (mod m)

It follows that if we’re just concerned with congruences (i.e. the residue mod m), multiplying the congruences provides the same result as multiplying the factors and then taking the result modulo m:

If a ∙ b ≡ x (mod m), then a (mod m) ∙ a (mod m) ≡ x (mod m)

In terms of an exponentiation algorithm, multiplying the result modulo m at each step leads to much smaller numbers which spares computational resources.

Slightly Better Algorithm

For c ← b^e (mod m)

Start with 1, multiply by b, take the result mod(m), repeat e times.

In other words:

Start with c ← 1
Repeat e times: c ← c ∙ b mod m

Exponent is a Power of Two

If c ← b^e (mod m) and

e = 2^k

We can compute c using the “squares” method - this allows for fast computation of large positive integer powers of a number.

From rules of indices:

(b^e)^f = b^ef

For example, this allows a⁸, can be represented as ((a²)²)².

If you calculate a⁸ naively:

a⁸ = a ∙ a ∙ a ∙ a ∙ a ∙ a ∙ a ∙ a

…7 multiplications are required (the exponent - 1).

Alternatively, computing a⁸ as ((a²)²)² requires three multiplications:

a ∙ a = s₁ s₁ ∙ s₁ = s₂, where s₂ is equivalent to (a²)² s₂ ∙ s₂ = s₃, where s₃ is equivalent to ((a²)²)²

In this way, aⁿ requires no more than 2 log₂(e) multiplications, where e is the exponent.

So long as our exponent is a power of 2, and we multiply congruences at each stage, we have an efficient algorithm that can be converted to code:

Example Code: Exponent is a Power of 2

#!/usr/bin/env python3

def exponent_power_of_2(a, e_power2, mod):
    for i in range (0, e_power2):
        a = (a * a) % mod
    return a

Exponent is Not Necessarily a Power of Two

The previous algorithm gets us on the right track, but it has a major limitation - it only works if the exponent is a power of two.

We can however take the principle and generalise it so that it works for any number.

The idea is to take any number represented as the summation of powers of two - which as luck would have it, is exactly the way that modern computers represent numbers - and to create a running total of the required squares.

Example:

a¹¹ = a⁸ ∙ a² ∙ a¹

…Notice that 8, 2 and one are powers of 2 (3, 1 and 0 respectively).

We can get the answer by working through each power of two up to the maximum possible given the size of e, squaring the base at each stage and only adding the squares to the result if the given power of two is a factor in the exponent.

Algorithm

Given a base b, an exponent e and modulo m, compute b^e (mod m):

Create an integer (or long) variable called result and set this result equal to 1.
Check the least significant bit (2⁰) of the exponent e. If it is 1, set result equal to base.
Check each bit in the exponent by iteratively bitshifting and masking against 1 - this checks each position in order, starting from the second-least-significant bit (we have already considered the least most significant bit in stage 2.
Start a loop
At each iteration, set base equal to the value of the previous base squared, modulo m
At each stage, if the LSB of e is set, set result equal to the product of the previous result and the current base (which is the previous base squared, as described in stage 3), all modulo m
When the value of e is NULL, end the loop
The value of result is the product of all b to the power of two for all powers of two that constitute the exponent.

In summary, set the result to be 1. Starting from the least significant bit of the exponent, iteratively check each bit - which denotes that the particular power of two is a component of the exponent. Square the base modulo m at each stage. If the exponent bit is set, multiply the base with the current result, modulo m. The final result is the base raised to the exponent mod m - the product of a set of base raised to exponents that constitute the original exponent broken into powers of two.

Example Code

Examples applying the above algorithm in C and Python are shown below:

Example: C

// See: https://github.com/DavidCWebs/input-integer-C for integer inputfunction
#include <stdio.h>
#include "integer-input.h"

int fastExp(int b, int e, int m)
{
	int result = 1;
	if (1 & e)
		result = b;
	while (1) {
		if (!e) break;
		e >>= 1;
		b = (b * b) % m;
		if (e & 1)
			result = (result * b) % m;
	}
	return result;
}

int main()
{
	puts("Compute b to the power e modulo m");
	int b = 0, e = 0, m = 0;
	puts("Enter b:");
	getIntegerFromStdin(&b);
	puts("Enter e:");
	getIntegerFromStdin(&e);
	puts("Enter m:");
	getIntegerFromStdin(&m);
	printf("%d to the power %d ≡ %d (mod%d)\n", b, e, fastExp(b, e, m), m);
	return 0;
}

Example: Python

#!/usr/bin/env python3
def fast_exp(b, e, m):
    r = 1
    if 1 & e:
        r = b
    while e:
        e >>= 1
        b = (b * b) % m
        if e & 1: r = (r * b) % m
    return r

def main():
    b = int(input("Enter b:"))
    e = int(input("Enter e:"))
    m = int(input("Enter m:"))
    r = fast_exp(b, e, m)
    print("{} ^ {} ≡ {} (mod {})".format(b, e, r, m))

if __name__ == '__main__':
    main()

References

Code examples, GitHub
Modular Arithmetic, Khan Academy, with practice quizzes
Intro to modular exponentiation, You Tube, Mark’s Education Tutorials
Modular Exponentiation Made Easy, Randall Heyman YouTube

Double Negation Operator Convert to Boolean in C

Tue, 29 Oct 2019 00:00:00 +0000

In C and C++ the double negation operator can be (and often is) used to convert a value to a boolean.

Simply put, if int x = 42, !!x evaluates to 1. If x = 0, !!x evaluates to 0.

You could argue that it is not readable, but as of the current date I counted more than 5000 instances of this idiom in the Linux kernel. For this reason alone it’s worth being aware of it.

#include <stdio.h>

int main()
{
	int x = 42;
	// Following statements are equivalent:
	printf("x is %d, which is boolean %d.\n", x, !!x);
	printf("x is %d, which is boolean %d.\n", x, (x == 0 ? 0 : 1));
	printf("x is %d, which is boolean %d.\n", x, x != 0);
	// Output: "x is 42, which is boolean 1."

	x = 0;
	// Following statements are equivalent:
	printf("x is %d, which is boolean %d.\n", x, !!x);
	printf("x is %d, which is boolean %d.\n", x, (x == 0 ? 0 : 1));
	printf("x is %d, which is boolean %d.\n", x, x != 0);
	//Output: "x is 0, which is boolean 0."
	
	return 0;
}

Sync Forked Repo With Upstream

Fri, 04 Oct 2019 00:00:00 +0100

If you have forked a Git repository, you’ll need to sync it with the upstream repo so that your changes are applied to the current version of the project.

Configure Remote

Check current remotes:

git remote

This will probably output origin, denoting that your repo has a single remote.

Configure a Git remote for the upstream (original) repo:

git remote add upstream https://github.com/original-project/original-project.git

Check:

git remote

# Output:
origin
upstream

Sync Local Fork with Upstream

There are now three repositories:

local
origin (GitHub, Bitbucket etc)
upstream (The original repo)

Sync from upstream to local, then push changes to origin:

# fetch changes
git fetch upstream

# switch to master branch
git checkout master

# Merge changes from upstream into master
git merge upstream/master

# Push local changes to origin
git push

Compute Bitcoin Merkle Root

Sun, 01 Sep 2019 00:00:00 +0100

Merkle trees are an efficient way to verify that an element is in a set, without having to store the full set.

The leaf nodes (lowest level nodes) of the Merkle tree are made up of hashes of individual data members of the set. In the case of Bitcoin, individual data members are made up from transaction ids (TXIDs). TXIDs are the result of SHA256 hashing the bytes in a raw transaction twice.

Adjacent leaves are concatenated pairwise, and the hash of the concatenation constitutes the node’s parent.

Parent nodes are concatenated and hashed in a similar way to generate another level of parent nodes. This process is repeated until a single hash remains - the Merkle root.

Each non-leaf node of a merkle tree is a hash of the concatenation of it’s immediate children. The leaves of the tree are the elements of the set to which the Merkle tree proves membership.

Merkle Trees as Implemented By Bitcoin

In the case of Bitcoin, the leaves of the Merkle tree are the transaction identifiers (the hash of a raw transaction).

If there are an odd number of nodes at any level, the final node is concatenated with itself. Parent nodes are in turn concatenated in pairs and hashed, with the process repeated until a single Merkle hash remains - the Merkle root.

The Merkle root is stored in a block header, where it serves to make transactions tamper-proof - if a transaction is changed, the Merkle root would be thrown off. Because the hash of each block is included in subsequent blocks, the tamper would be immediately evident and the block with the tampered transaction would not be accepted as valid by the Bitcoin consensus rules.

To verify a transaction - check that a transaction is in a valid block - you just need the hashes of Merkle branch to compute the Merkle root, not the entire set of transactions in the block.

Flaw in Bitcoin Implementation of Merkle Trees

As can be seen in this comment in the Bitcoin Core codebase, and described comprehensively in BIP 98, there is a flaw in the Bicoin implementation of Bitcoin’s Merkle tree algorithm relating to duplicate txids.

The flaw is caused by the duplication of the last node in the case of levels that have an odd number of nodes - a practice which is non-standard for Merkle trees.

C++ Implementation

Compute a Merkle root from a set of transaction IDs (txid) expressed as hexadecimal strings. The code shown is from the file bitcoin.cpp in the accompanying repo:


/**
 * Compute the Merkle root 
 * */
void merkleRoot(std::vector<Bytes> txids, Bytes& result)
{
  if (txids.empty()) {
    throw;
  }
  // Loop until the txid container has been reduced to a single element - the Merkle root
  while (txids.size() > 1) {
    // If odd number, add the last element to end of vector.
    // Note that this is required at every level of the tree.
    if (txids.size() & 1) {
      txids.push_back(txids.back());
    }
    std::vector<Bytes> tmp;
    for (auto it = std::begin(txids); it != std::end(txids) && std::next(it) != txids.end(); it += 2) {
      Bytes concat = *it;
      Bytes result(hash_size);
      
      // Concatenate this element and the following adjacent element
      concat.insert(concat.end(), (*(it + 1)).begin(), (*(it + 1)).end());
      
      // Hash the concatenated elements, save into `result` container
      doubleSHA256(concat.data(), concat.size(), result);

      // `tmp` is a container holding the results of concatenating & hashing
      tmp.push_back(result);
      concat.clear();
    }
    // Set txids to reflect the reduction in elements from this round of concatenation/hashing
    txids = tmp;
  }

  // At this point, `txids` should be a container with a single element.
  result = txids[0];
}

In this example, the merkleRoot function receives two parameters:

std::vector<Bytes> txids - where Bytes is an alias for std::vector<uint8_t>, a collection of byte objects.
Bytes& result - A std::vector of <uint8_t> objects, passed by reference to receive the result.

Implementation

You try out the code above by cloning the accompanying GitHub repo.

Get TXIDs for a Block Using bitcoin-cli

You can use bitcoin-cli to get the transactions from a sample block and save these as a manifest file:

# Write transaction ids for block
# 00000000000000000002f5b1c49b9ddf5537d418b6c5b835172b3987a09a4b13
# to /tmp/manifest
bitcoind --daemon # bitcoind must be running

# Fetch block and process results with `jq` to obtain a file comprised of `txid` hashes for all
# transactions in the block, each on a separate line. The file `/tmp/txid.manifest` can then be
# used as input to the programme via input redirection:
block=00000000000000000002f5b1c49b9ddf5537d418b6c5b835172b3987a09a4b13
bitcoin-cli getblock ${block} | jq -r '.tx[]' > /tmp/txid.manifest
bitcoin-cli stop

This is useful for testing Merkle tree root computation - you can check the result of your implementation with the Merkle root computed by Bitcoin Core:

# set ${block} as above
bitcoin-cli getblock ${block} | jq -r '.merkleroot'

If you don’t have jq installed, you can achieve roughly the same result with standard UNIX tools like this:

bitcoin-cli getblock ${block} | grep merkleroot | awk '{print substr($2,2, length($2) - 3)}'

References

Convert integer to array of char bytes

Thu, 15 Aug 2019 00:00:00 +0100

To convert an integer to an array of char, you can bitmask against the integer in a loop to get the value for each byte.

Endianness & Bitshifting

On a little-endian system, int 256 is stored across 4 bytes in memory like this:

Multiplier	256⁰	256¹	256²	256³
Value	00	01	00	00
Address	1000	1001	1002	1003

You might imagine that iteratively bitshifting this value and masking against 0xff would provide the constituent bytes in the little-endian order, but this is not the case.

Bit shifting is always applied against a big-endian representation. This is because when the integer is loaded into the CPU’s register, it is effectively represented in big-endian format.

The CPU register is not byte-addressable, so in this context endianness does not make sense.

This means that bitwise operations are applied to a big-endian representation of the number.


int main()
{
	int num = 256;

	// When the integer is loaded into the CPU's register, it is the 
	// equivalent of conversion to big-endian format - the number in
	// the CPU register is big-endian.
	//
	// This means that bitwise operations are applied to a big-endian
	// representation of the number:
	// 00 00 01 00
	
	std::cout << "num: " << num << '\n';		// 256
	
	std::cout << "(num >> 0) & 0xff = "
		<< (num & 0xff) << '\n';		// 0

	std::cout << "(num >> 8) & 0xff = "
		<< ((num >> 8) & 0xff) << '\n';		// 1

	std::cout << "(num >> 16) & 0xff = "
		<< ((num >> 16) & 0xff) << '\n';	// 0

	std::cout << "(num >> 24) & 0xff = "
		<< ((num >> 24) & 0xff) << '\n';	// 0
	
	return 0;
}

In short, you probably don’t need to worry about endianness when bit shifting - but bear in mind even if your system is little-endian, the bitshift operation takes place on a big endian representation.

The Register

A 32-bit CPU register holding a 32-bit value is neither big-endian nor little-endian - it is a register holding a 32-bit value see ref. It is kind of big-endian though, since the rightmost bit is the least significant bit and the leftmost bit is the most significant.

System Endianness: Linux

To find endianness on a modern Linux kernel:

lscpu | grep Endian

# result:
Byte Order:            Little Endian

Conversion to an Array of Bytes

#include <iostream>
#include <iomanip>
#include <stdio.h>

#define BITS 8

const size_t BYTES_PER_INT = sizeof(int); 

int intToCharArray(char* buffer, int in)
{
	for (size_t i = 0; i < BYTES_PER_INT; i++) {
		size_t shift = 8 * (BYTES_PER_INT - 1 - i);
		buffer[i] = (in >> shift) & 0xff;
	}
	return 0;
}

int main()
{
	std::cout << "Enter an integer: ";
	unsigned int in;
	if (!(std::cin >> in)) {
		std::cout << "Not an integer. Exiting..." << '\n';
		return 1;	
	}
	char buffer[BYTES_PER_INT];
	intToCharArray(buffer, in);
	for (auto& el : buffer) {
		std::cout << std::hex << std::setw(2) << std::setfill('0')
			<< static_cast<unsigned int>(el & 0xff) << " ";
	}
	std::cout << '\n';
}

References

Writing endian-independent code in C, IBM Developer article

Information Entropy

Mon, 12 Aug 2019 00:00:00 +0100

Information Entropy is the average rate at which information is produced by a stochastic(random) source of data.

What is Information?

Information is what you get when uncertainty around an event is decreased. Before a fair coin-toss, there is an equal probabilty of the toss resulting in heads or tails - the result is uncertain. After the toss, there is no more uncertainty and information (the result of the toss) has been gained.

The amount of information in a message is defined as the minimum number of bits required to represent all possible meanings of the message, assuming that all messages are equally likely (Scshneier, 1996).

In terms of the coin toss, because there are only two outcomes you can represent all possible outcomes with a single bit.

Entropy is used to quantify uncertainty. For a coin toss, entropy is 1 since this represents the minimum number of bits to represent all possible outcomes.

Definition of Entropy

Entropy is a way of measuring the uncertainty of a stochastic variable (Schiffman).

The amount of information that could be conveyed by a message is measured by the entropy of the message.

The amount of information in a message is the minimum number of bits required to represent the message.

The entropy of a message indicating whether to proceed or not can be represented by a single bit, since by definition a bit has one of two states:

Proceed?	Minimum Representation in bits
No	0
Yes	1

The days of the week can be represented by slightly less than 3 bits:

Day of week	Minimum representation in bits
Sunday	000
Monday	001
Tuesday	010
Wednesday	011
Thursday	100
Friday	101
Saturday	110
Unused	111

There are 7 potential days of the week, so the information must be represented by a minimum of 7 binary states. How many bits are required to represent 7 distinct states?

The lowest number that can represent 7 states is 6: {0, 1, 2, 3, 4, 5, 6} - as shown above in binary format. The number 6 represented in binary format is 110, or 2^2.59, or log₂ 6.

Because a binary system represents data in discrete bits, the number of bits required to represent a number must be an integer not a fraction. In this case, 3 bits are required to represent the 7 states.

In a base 2 system, this generalises to 2ⁿ where n is the minimum number of digits required to represent all possible states.

Entropy, though measured in bits, is not information. It is rather a measurement of how much information is not available when the outcome of a random source of information is unknown.

Mnimum Representation in Bits

The minimum power of 2 that is greater than or equal to the number of states.

Where the number of states is n,

log₂ n

The binary logarithm (log₂ n) is the power to which the number 2 must be raised to obtain the value n. For any real number x:

x = log₂⁡ n ⟺ 2^x = n.

Double a number adds 1 to binary logarithm:

2³ (8)	2² (4)	2¹ (2)	2⁰ (1)	Decimal	log₂⁡ n
	1	0	0	4	2
1	0	0	0	8	3

Upper Limits

For storage of raw data, not the information content of the data:

2 bits: 4 different values
3 bits: 8 different values
4 bits: 16 different values
8 bits: 256 different values

Practical Uses

When calculating the entropy of a password or passphrase, entropy is defined as log base 2 of the number of characters/words that the password has been randomly selected from to the power of the password/passphrase length.

The Bash script below calculates the entropy of words that have been pseudo-randomly selected from a word list. In this case, character set is the number of words available to select from and the length is the number of pseudorandom words output.

#/bin/bash
...
function entropy {
	export WORDSET=$1
	export WORDS=$2
	python3 - <<- EOF
	import math; import os;
	# Entropy is log base 2 of the keyspace to the power of the length 
	r = math.log(int(os.environ['WORDSET']) ** int(os.environ['WORDS']), 2)
	print(round(r, 2))
	EOF
}

Resources

Explicit Conversion Constructor in C++

Sun, 11 Aug 2019 00:00:00 +0100

Prepending the keyword explicit to a C++ constructor declaration (i.e. adding the explicit function specifier to the constructor within the class declaration) prevents unwanted type conversions.

Without the explicit function specifier, a constructor is a converting constructor - the type can be initialised by assigning a variable of the appropriate type. Prior to C++11, the constructor needed to be called with a single non-default parameter. For example:

class A
{
	A(int) {}		// Converting constructor
	A(int, double) {}	// Converting constructor
};

class B
{
	explicit B(int) {}
	explicit B(int, double) {}
}

void main()
{
	// Converting constructors
	// -----------------------
	A x = 42;		// Copy initialisation
	A x1 = {42, 3.14};	// Copy list initialisation
	A x2(42);		// Direct initialisation
	A x3{42, 3.14};		// Direct list initialisation

	// Explicit conversion constructors
	// --------------------------------
	B y = 42;		// NOT ALLOWED: Error
	B y1 = {42, 3.14};	// NOT ALLOWED: Error
	B y2(42);		// Direct initalisation OK
	B y3{42, 3.14};		// Direct list initialisation OK
	B y4 = (B)1		// OK - exlicit cast does a static cast, direct initialisation
}

Note that prior to C++11, converting constructors needed to have a single non-default parameter.

References

Explicit conversion constructors: IBM Knowledge Centre
Converting constructors

Lambdas in C++

Fri, 09 Aug 2019 00:00:00 +0100

In C++ a lambda expression or lambda function is a closure: an unnamed (anonymous) function that is passed inline - usually to another function. Lambda expressions are a concise way of creating simple function objects.

If necessary, lambda expressions can capture variables from their context.

Lambda expressions are useful for passing an operation as an argument into an algorithm.

Using a lambda expression is like using a function pointer, but without the need to create a named object.

In C++ a Lambda expression can only be considered a closure when there is something in the capture clause ([]).

Syntax

Full syntax:

// [captures] (params) -> returnType { function body }

// Concrete example: capture a by reference; parameter is int; lambda expression returns an int 
int a = 10;
ForEach(intVec, [&a] (int x) -> int{ return a * x; });

If the return type can be deduced from the return statement in the function body, the return type can be ommitted:

// [captures] (params) { function body }
// Concrete example: No captures, parameter is int, lambda expression returns an int
ForEach(intVec, [] (int x) { return x * x; });

If the lambda expression takes no arguments:

// [captures] { function body }
// Concrete example: no captures, no parameters:
ForEach(intVec, []{ return 0; }

For other syntax options: https://en.cppreference.com/w/cpp/language/lambda

Generic Lambda Expression

Lambda expressions can take arguments that have arbitrary types.

If a parameter is marked auto, the type can be deduced:

ForEach(intsVec, [] (auto x) { return x * x });

In this case, the lambda expression becomes an instance of type:

class // unnamed
{
public:
	template<typename T>
	T operator() (T x) const { return x * x }
};

If a generic lambda has no captures, the closure type has a “public non-virtual non-explicit const conversion function template to pointer to function” see the standards document.

Passing Generic Lambda to Template Function

In the following example, ForEach() can transform elements of a vector - the type of the element being generic.

The generic lambda is straightforward enough, but note the template function signature - one way or the other, you should let the compiler deduce the type for the Lambda expression.

#include <iostream>
#include <vector>

template <typename T, typename Func>
void ForEach(std::vector<T>& container, Func f)
{
	for (auto& el : container) {
		el = f(el);	
	}
}

// Equivalent to the above - less typing with `auto`!
void ForEachAuto(std::vector<auto>& container, auto const& lambdaExpression)
{
	for (auto& el : container) {
		el = lambdaExpression(el);	
	}
}

int main()
{
	std::vector<int> intVec = { 0,1,2,3,4,5 };
	ForEach(intVec, [](auto x){ return x * x; });
	// intVec is now { 0,1,4,9,16,25 }	

	std::vector<double> doublesVec = { 1.1,2.2,3.3 };
	ForEach(doublesVec, [](auto x){ return x * x; });
	// doublesVec is now { 1.21, 4.84, 10.89 }

	return 0;
}

Note that the definition of the generic function that accepts the lambda as a parameter can either use template syntax or the auto keyword.

Captures

The captured variables can be passed by value or by reference.

Copy Values

Implicitly capture all automatic variables by value - copy values into the lambda expression:

[=]

Pass by Reference

Implicitly capture all automatic variables by reference - copy references into the lambda expression:

[&]

Override the Default Capture Method

You can set a default capture method and override this for individual variables:

// Capture by reference, except a which is captured by copy:
[&, a]{}

// Capture all by copy, except a which is captured by reference:
[=, &a]

Callbacks

A callback is a function which is invoked by another function. The calling function has the option of calling different callbacks - perhaps as a function pointer.

C++ allows a Lambda expression to be used as a callback - as exemplified above.

References

C++ Standards Committee ISOCPP: Lambda expressions
Closures
Syntax
Generic Lambdas
CopperSpice Video: Lambdas in Action

Build an Array of Strings From a File in C

Mon, 29 Jul 2019 00:00:00 +0100

Use getline() to read lines from a file, and add them to dynamically allocated memory:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv)
{
	if (argc != 2) {
		fprintf(stderr, "Please supply a file path:\n%s <file path>\n", argv[0]);
		return EXIT_FAILURE;
	}
	FILE *fp = fopen(argv[1], "r");
	if(!fp) {
		perror("ERROR");
		return EXIT_FAILURE;
	}
	
	// Read lines from file, allocate memory and build an array of lines
	// -----------------------------------------------------------------
	char *lineBuf = NULL;
	size_t n = 0;
	size_t nLines = 0;
	ssize_t lineLength = 0;
	size_t sizeIncrement = 10;
	char **lines = malloc(sizeIncrement * sizeof(char**));
	size_t i = 0;
	
	while ((lineLength = getline(&lineBuf, &n, fp)) != -1) {
		// Memory reallocation is expensive - don't reallocate on every iteration.
		if (i >= sizeIncrement) {
			sizeIncrement += sizeIncrement;

			// Don't just overwrite with realloc - the original
			// pointer may be lost if realloc fails.
			char **tmp = realloc(lines, sizeIncrement * sizeof(char**));
			if (!tmp) {
				perror("realloc");
				return EXIT_FAILURE;
			}
			lines = tmp;
		}
		// Remove \n from the line.
		lineBuf[strcspn(lineBuf, "\n")] = 0;

		// Allocate space on the heap for the line.
		*(lines + i) = malloc((lineLength + 1) * sizeof(char));

		// Copy the getline buffer into the new string.
		strcpy(*(lines + i), lineBuf);

		i++;

		// Keep track of the number of lines read for later use.
		nLines = i;
	}

	// Do something with the array of strings.
	printf("nLines: %lu\n", nLines);
	for (size_t k = 0; k < nLines; k++) {
		printf("%lu\t %s\n", k, *(lines + k));
	}
	
	// Free the buffer utilised by `getline()`.
	free(lineBuf);

	// Free the array of strings.
	for (size_t i = 0; i < nLines; i++)
		free(*(lines + i));
	free(lines);

	fclose(fp);
	return 0;
}

size_t vs ssize_t

Note that the return value of getline() is assigned to ssize_t lineLength.

The reason that the ssize_t type is used is that getline() can return a negative value (actually -1) in the event of failure or the end-of-file condition.

ssize_t is guaranteed to be able to store values in the range -1 to SSIZE_MAX and should be used for a count of bytes or an error indication.

Realloc

Note that you should check that realloc() succeeds by checking for a non-null return value - or responding to the null case as in the example above.

You can do this by assigning realloc to a temporary variable, only copying it across to your target variable when you are sure that realloc() succeeded.

Otherwise, you risk writing NULL to your target variable, thereby causing a memory leak (since you can’t access the pointer to free memory).

References

Convert an Array of Unsigned Chars to an int32_t Value

Tue, 23 Jul 2019 00:00:00 +0100

In C or C++, an array of four unsigned char values (four raw bytes) can be converted to a single int32_t value using bitwise operations.

#include <stdio.h>
#include <stdint.h>
#include <inttypes.h>

int main()
{
	// An array of raw bytes
	unsigned char raw[] = {0xde, 0xad, 0xbe, 0xef};
	for (size_t i = 0; i < 4; i++) {
		printf("%d\n", raw[i]);
	}

	// To pack into a 4 byte data type, shift each element the 
	// appropriate number of bits and perform bitwise OR on
	// all shifted elements.
	uint32_t result = (raw[0] << 24 | raw[1] << 16 | raw[2] << 8 | raw[3]);

	printf("%%u result = %u\n", result); // 3735928559
	return 0;
}

Explanation

An array of unsigned char containing the decimal values 222, 173, 190, 239 can represented in binary format:

Decimal	Binary
222	`11011110`
173	`10101101`
190	`10111110`
239	`11101111`

To get these values packed into a 4 byte data type such as uint32_t, the first (single byte) value in the array should be inserted into the most significant byte. The second should be inserted into the next most significant byte, and so on.

This can be achieved by:

Bit shifting each byte to it’s required position
Performing a bitwise OR operation on all bytes

Using the values from the above example:

# 11011110 << 24
+-----------------+-----------------+-----------------+-----------------+
| 1 1 0 1 1 1 1 0 | 0 0 0 0 0 0 0 0 | 0 0 0 0 0 0 0 0 | 0 0 0 0 0 0 0 0 |
+-----------------+-----------------+-----------------+-----------------+

# 10101101 << 16
+-----------------+-----------------+-----------------+-----------------+
| 0 0 0 0 0 0 0 0 | 1 0 1 0 1 1 0 1 | 0 0 0 0 0 0 0 0 | 0 0 0 0 0 0 0 0 |
+-----------------+-----------------+-----------------+-----------------+

# 10111110 << 8
+-----------------+-----------------+-----------------+-----------------+
| 0 0 0 0 0 0 0 0 | 0 0 0 0 0 0 0 0 | 1 0 1 1 1 1 1 0 | 0 0 0 0 0 0 0 0 |
+-----------------+-----------------+-----------------+-----------------+

# The least significant byte doesn't need shifting
+-----------------+-----------------+-----------------+-----------------+
| 0 0 0 0 0 0 0 0 | 0 0 0 0 0 0 0 0 | 0 0 0 0 0 0 0 0 | 1 1 1 0 1 1 1 1 |
+-----------------+-----------------+-----------------+-----------------+

Perform bitwise OR on all the above values:
+-----------------+-----------------+-----------------+-----------------+
| 1 1 0 1 1 1 1 0 | 1 0 1 0 1 1 0 1 | 1 0 1 1 1 1 1 0 | 1 1 1 0 1 1 1 1 |
+-----------------+-----------------+-----------------+-----------------+

References

Integers in C

Find Custom Data Types in a C++ Project

Mon, 22 Jul 2019 00:00:00 +0100

If your looking through an existing C++ project, it is likely that you will come across custom data types.

For example, if you see the following:

...
std::pair<Blob, bool> strUnHex (std::string const& strSrc){...}

…you just know that Blob is an object of some sort. The first_type in the std::pair could be:

A class-defined object
A struct defined object
A type alias

Searching a Project

Use grep:

cd myProject

grep -nr "search term" .

Search Project for Custom Type

If you want to determine information about a custom data type, finding where it is defined/aliased is a good idea:

grep for the specified options in the current directory:
grep -nr "using Blob\|class Blob\|struct Blob" .

# Output:
./src/path/Blob.h:30:using Blob = std::vector <unsigned char>;

At this point, you don’t even need to open Blob.h to know that Blob is an alias for std::vector<unsigned char>.

You could also try grepping for typedef statements:

# grep for the term "typedef<any number of any characters>Blob" recursively starting in the current directory: 
grep -nr "typedef.*Blob" .

References

Data types in C++

Arrays and Slices in Rust

Sat, 20 Jul 2019 00:00:00 +0100

Basic cheatsheet for Rust arrays and slices.

A rust array is a stack-allocated list of objects of a set type and fixed length.

Initialise:

// Arrays in rust are fixed length.
// Specify type and length in square brackets to initialize.
let arr: [i32; 5] = [1, 2, 3, 4, 5];

The correct data type and array length are now fixed and expected.

To print an array use Debug print:

println!("{:?}", arr);

Dereferencing

Dereference elements in the “universal” way:

for i in 0..arr.len() {
  println!("arr[{}] is {}", i, arr[i]);
}

Mutable Arrays

Arrays elements can be mutable, but the array length and data type is fixed:

let mut arr_to_change: [i32; 5] = [1, 2, 3, 4, 5];

arr_to_change[2] = 42;

Iteration

Arrays are not iterable. To iterate over each element, convert an array to iterator by calling .iter(), a slice method:

for el in arr.iter() {
  println!("{}", el);
}

Otherwise you could make a slice from the array:

for el in &arr {
	if Some(el) == arr.first() {print!("first! ")}
	println!("sliced: {}", el);
}

Enumerate the loop iterator to track an index:

for (i, el) in arr_to_change.iter().enumerate() {
  if i > 0 {print!(", ")};
  print!("{}", el);
}
println!();

If you try to access array elements that are out-of-bounds, Rust will “panic” - the programme terminates immediately. This prevents buffer overread bugs.

Memory Allocation

Get memory allocated:

println!("Array arr is {} bytes", std::mem::size_of_val(&arr));

Slices

Slices are a view into a block of memory represented by a pointer and a length. In other words, a slice is a view into an array.

Make a slice from the full array:

let sl: &[i32] = &arr;
println!("sl is {:?}", sl);

Make a slice of elements 0 and 1 - [0..2] is exclusive of the upper bound:

let sl2 = &arr[0..2];
// or overtly set the type:
let sl2: &[i32] = &arr[0..2];
println!("sl2 is {:?}", sl2);

Access Slice Elements

The get() method returns an element of a slice at the given index as an Option. In the example above, sl2.get(2) returns None because there is no element at that index - this prevents failure if the index is out of bounds.

get() uses Option to give type-safe bound checking array access.

Note that if you unwrap() an Option type that has a None value, the programme panics.

let i = 0;
let val = sl2.get(i);
if val != None {
  println!("sl2.get({}) returns {:?}", i, val.unwrap());
} else {
  // `val.unwrap()` at this point causes a panic.
  println!("Nothing at index {}. Move along.", i);
}

// Return an option
println!("sl2.get(1) returns {:?}", sl2.get(1));

// Unwrap the returned value
println!("sl2.get(1) returns {:?}", sl2.get(1).unwrap());
}

Resources

Passing Strings in C

Sat, 20 Jul 2019 00:00:00 +0100

If you need to operate on strings, it’s helpful to abstract this away into separate functions (which can be held in different files) to keep your code clean and to reduce repetition.

Amend Stack-Allocated String in a Function

In C, if you need to amend a string in a called function, pass a pointer to the first char in the string as an argument to the function.

It is also common to pass in the length of the string:

void function(char *myString, size_t lengthMyString);

If you don’t pass the length, you can get this value from within your function by counting up to a NUL char - which you can do easily with strlen() from the <string.h> header:

void function(char *myString)
{
  for (size_t i = 0; i < strlen(myString); i++) {
    // Access each char in the string
    printf("myString[%lu] = %c\n", i, myString[i]);
  }
}

Here’s another approach using a while loop to check for the end of the string:


// Within this function, `line` is initially a pointer to the first
// char in a string (Null terminated char array).
void f(char *line)
{
  // Continue if the value pointed at by `line` is not NUL
  while (*line) {
    // Dereference `line` and assign a new value '0'
    // then advance the pointer
    *(line++) = '0';
  }
}


int main()
{
	char line[] = "gdgegryt";
	f(line);
	printf("%s\n", line);
	return 0;
}

If you have allocated storage for the string outside the function, which you cannot exceed within the function, it’s probably a good idea to pass in the size.

Function Does Not Alter the String

If you’re not modifying the string, you can pass a const pointer and omit the length:

void printString(const char *myString);

Amend a Passed-in Buffer With Memory Allocation

If the function needs to modify a dynamically allocated (i.e. heap-allocated) string buffer from the caller, you must pass in a pointer to a pointer.

In C, function arguments are passed by value. This means that to modify a variable from within a function, you need a pointer to the variable.

This allows you to derefence the pointer, and potentially amend the value in the memory address being pointed to.

When you dynamically allocate memory to store a string (using one of the malloc() family of functions) you are assigning an area of heap memory to a char *.

If you need to change what is pointed to from with the function (e.g you might need to realloc() memory) the function must receive a pointer to the char *. In this way, you can dereference the char ** to change the value held in char * - which in this case is the memory address of the dynamically allocated storage.

char **s          char *s        char
+---+             +---+          +---+---+---+---+
| &-|------------>| &-|--------->|'a'|'b'|'c'| 0 |
+---+             +---+          +---+---+---+---+

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/**
 * Within function, amending *s changes the memory
 * address of the dynamically allocated string.
 **/
void add_xy(char **s, size_t length)
{
	*s = realloc(*s, length + 3);
	*(*s + length) = 'x';
	*(*s + length + 1) = 'y';
	// Add a null terminator
	*(*s + length + 2) = 0;
}

int main(void)
{
	char *s = NULL;
	size_t length = 0;

	// The size of s is unknown at compile time.
	getline(&s, &length, stdin);

	// Remove newline - maybe realloc memory after this.
	s[strcspn(s, "\n")] = 0;

	add_xy(&s, strlen(s));

	printf("%s\n", s);

	// getline() allocated memory, add_xy() reallocated. Free it.
	free(s);
}

References

Get a line from stdin in C

Fri, 19 Jul 2019 00:00:00 +0100

If you want to get a single line of stdin in C, getline() is the go-to function.

To get a single line from stdin:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
  printf("Please enter a line:\n");
  char *line = NULL;
  size_t len = 0;
  ssize_t lineSize = 0;
  lineSize = getline(&line, &len, stdin);
  printf("You entered %s, which has %zu chars.\n", line, lineSize - 1);
  free(line);
  return 0;
}

Why do we have to free(line)? After all, there are no malloc family functions in the above code?

The answer is that getline() allocates memory to the buffer (in the example above, line) as required.

How Does getline() Work?

getline() receives a pointer to a buffer in which to store the line string, a pointer to the size of the buffer, and a pointer to the input stream.

The function does the following:

Allocates memory for the string buffer on the heap
Updates the len variable to track the memory allocation
Reads a line into the heap-allocated string buffer
Returns the length of the string that has been read

Note that the newline character, not the null terminator, is counted in the length.

The memory allocated may be more than required for the input characters. If not enough memory is allocated initially, getline() can realloc() a longer buffer, while updating the line varible to point to the new buffer and updating len.

If getline() is called again, the original buffer is re-used unless reallocation of memory is required.

Example from `man getline`

The example in the man page loops through a file, collecting all lines into a single string. We can adapt this to read a file passed in as a command line argument:


#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
  if (argc != 2) {
    printf("please specify a file.");
    exit(EXIT_FAILURE);
  }
  FILE *stream;
  char *line = NULL;
  size_t len = 0;
  ssize_t read;

  stream = fopen(argv[1], "r");
  if (stream == NULL)
    exit(EXIT_FAILURE);

  while ((read = getline(&line, &len, stream)) != -1) {
    printf("Retrieved line of length %zu :\n", read);
    printf("%s", line);
  }

  free(line);
  fclose(stream);
  exit(EXIT_SUCCESS);
}

References

Courseera video, Duke Uni: Reading a file with getline().

Display char as Hexadecimal String in C++

Fri, 12 Jul 2019 00:00:00 +0100

Displaying a char or a collection of chars as a hexadecimal string in C++ is surprisingly tricky.

For context, let’s say you are collecting bytes from /dev/urandom in a std::vector and you need to display them to the user:


#include <iostream>
#include <fstream>
#include <vector>
#include <fstream>
#include <iterator>
#include <iomanip>

int main()
{

  int n = 0;
  std::cout << "Enter the number of random bytes to fetch from /dev/urandom:" << '\n';
  if (!(std::cin >> n)) {
  	std::cout << "Not an integer. Exiting..." << '\n';
  	return 1;	
  }
  std::ifstream file("/dev/urandom", std::ios::binary|std::ios::in);
  if (!file) {
  	std::cerr << "Couldn't open /dev/urandom. Exiting..." << '\n';
  	return 1;
  }
  std::vector<char> randomBytes(n);
  file.read(&randomBytes[0], n);

  // Displaying bytes: method 1
  // --------------------------
  for (auto& el : randomBytes)
  	std::cout << std::setfill('0') << std::setw(2) << std::hex << (0xff & (unsigned int)el);
  std::cout << '\n';

  // Displaying bytes: method 2
  // --------------------------
  for (auto& el : randomBytes)
  	printf("%02hhx", el);
  std::cout << '\n';
  return 0;
}

Method One: std::cout

Method 1 as shown above is probably the more C++ way:

Cast to an unsigned int
Use std::hex to represent the value as hexadecimal digits
Use std::setw and std::setfill from <iomanip> to format

Note that you need to mask the cast int against 0xff to display the least significant byte: (0xff & (unsigned int)el).

Otherwise, if the highest bit is set the cast will result in the three most significant bytes being set to ff.

Method 2: printf()

Method 2 uses the printf() function with the "%02hhx" format string.

This is pretty much C style code. It’s quite a bit shorter though!

User Input cin in C++

Thu, 11 Jul 2019 00:00:00 +0100

If an input variable is declared, and the value collected by std::cin does not match, std::cin returns false.

For example:

int input = 0;

std::cin >> input;

// Entering an integer at this point works.
// Entering a value which is not an integer and not numeric, std::cin returns false.
// Entering a real number (with a decimal point): fractional part is discarded and integer is saved.

It’s better to take input within a loop, with some input validation:


#include <iostream>
#include <typeinfo>
#include <limits>

int main()
{
	int input = 0;
	while (1) {
		std::cout << "Enter an int:\n";
	
		// Enter this block if taking input from cin has failed.
		if (!(std::cin >> input)) {
	
			// The error flag is set on std::cin - future attempts to
			// get input will fail unless the error is cleared.
			std::cin.clear();

			// The failed input is in the input buffer. The default for `ignore` is
			// to skip a single character. To be sure, remove the max streamsize
			// number of chars up until a newline is encountered
			std::cin.ignore(std::numeric_limits<std::streamsize>::max(), '\n');
			continue;
		}
		break;
	}
	std::cout << "You entered: " << input << '\n';
	return 0;
}

std::cin returns false if the input can’t match the expected type (in this case, int).
std::cin.ignore() extracts and discards unwanted values.
std::cin.clear() changes the internal state of the stream - unsets the error flag.

For a working example, see here

References

Entertaining SO Answer

Guard One Liner in Bash

Fri, 05 Jul 2019 00:00:00 +0100

Add a one-line guard statement in Bash:

[[ $# -eq 0 ]] && { echo "Please supply a target file: $(basename $0) <path/to/file>"; exit 1; }

Note that multiple commands in the curly braces must each be terminated with a semi-colon.

Note that there must be:

A single space character between each curly brace and the contents.
A single space between the [[ and ]] characters.

The Bash special variable $# refers to the number of command line arguments. $(basename $0) refers to the current script name.

qsort Comparison Function

Fri, 14 Jun 2019 00:00:00 +0100

In C, the qsort() function sorts an array in place using the quickersort algorithm - a variation of quick sort.

Function Signature

#include <stdlib.h>

void qsort(void *base, size_t nmemb, size_t size,
	int (*compar)(const void *, const void *));

Function arguments for qsort():

base: pointer to the start of the array
nmemb: Number of members in the array
size: Size of an individual element in the array
compar: Pointer to a comparison function that accepts 2 x const void * arguments

To use qsort(), you need to include the stdlib.h header - i.e. #include <stdlib.h>.

Comparison Function

The comparison function must return an integer that is:

Less than zero if the first argument is considered to be less than the second
Zero if the arguments are equal
Greater than zero if the first argument is considered to be greater than the second

Be careful with return values - especially if the values being compared are not integers.

Use > or ‘<’ rather than subtraction which may provide unexpected results if comparing non integer values.

Note that the comparison function receives const void * as arguments - and you may need to cast these within the comparison function.

Example

typedef struct triangle
{
	int a;
	int b;
	int c;
} Triangle;


int compareTriangles(const void *x, const void *y)
{
	const Triangle *a = x, *b = y;
	int res = area(a) != area(b) ? area(a) > area(b) ? 1 : -1 : 0;
	return res;
}

/**
* Sort an array of Triangles
*/
void sort_by_area(Triangle* tr, int n) {
	qsort(tr, n, sizeof(Triangle), compareTriangles);	
}

The above example sorts an array of Triangle type defined objects according to area (area() has been omitted for brevity).

Note the nested conditional in compareTriangles():

If area of a and b are not equal, return 1 if the area of a is greater than the area of b, -1 if not.
If the area of a and b are equal, return 0.

Also note the line const Triangle *a = x, *b = y; which copies the input pointers into the object required for the comparison.

Pointer Conversion in Comparator Function

The comparator function receives const void * type pointers - these should be cast in the comparator to the original type of the data passed in.

Multiple Indirection: Double Pointers

If the comparator function for qsort compares values in an array of pointers (pointer-to-pointer), cast to this within the comparator.

For example, if the array to be sorted is an array of pointers, the comparator should cast the input void pointers to pointer to the underlying type pointer.

The code below shows how an array of pointers to custom card_t types is sorted:

/**
* Comparator function to sort an array of type card_t *
* card_t is a typedef defined struct, with value and suit (numeric) fields.
*
*/
int card_ptr_comp(const void *inPtr1, const void *inPtr2)i
{
	const card_t *c1 = *(const card_t **)inPtr1;
	const card_t *c2 = *(const card_t **)inPtr2;
	unsigned v1 = c1->value;
	unsigned v2 = c2->value;
	
	// Alternative way of handling the cast:
	// const card_t * const *c1 = inPtr1;
	// const card_t * const *c2 = inPtr2;
	// unsigned v1 = (*c1)->value;
	// unsigned v2 = (*c2)->value;
	
	if (v1 != v2)
		return (v1 > v2) ? -1 : (v1 == v2) ? 0 : 1;
	else
		return (c1->suit > c2->suit) ? -1 : 1;
}

int main()
{
	card_t **cards = malloc(n * sizeof(card_t*));
	// ... add cards

	qsort(cards, n_cards, sizeof(cards[0]), card_ptr_comp);

	// ... free dynbamically allocated resources
	return 0;
}

More complete example:

#include <stdio.h>
#include <stdlib.h>

typedef struct person {
	int id;
	const char *name;
} person_t;

// Compare by id
int comparePeople(const void *inPtr1, const void *inPtr2)
{
	const person_t *p1 = *(const person_t**)inPtr1;
	const person_t *p2 = *(const person_t**)inPtr2;

	if (p1->id != p2->id)
		return p1->id > p2->id ? 1 : -1; 
	return 0;
}

int main()
{
	person_t a = {99, "Alice"};
	person_t b = {42, "Bob"};
	person_t c = {33, "Chris"};
	person_t *people[] = {&a, &b, &c};
	qsort(people, sizeof(people) / sizeof(people[0]), sizeof(people[0]), comparePeople);
	
	for (size_t i = 0; i < sizeof(people) / sizeof(people[0]); i++) {
		printf("id: %d\tname: %s\n", people[i]->id, people[i]->name);			
	}
	
	return 0;
}

Integer Input in C

Fri, 31 May 2019 00:00:00 +0100

Objective: get an integer input from stdin.

At first glance, scanf() looks like a reasonable way to collect integer input. The function allows formatted input to be collected (it’s name comes from the words “scan formatted”).

To scan integer input, first declare an integer and pass a pointer to this to scanf:

int input;
scanf("%d", &input);

However, because stdin is inherently unconstrained (users can type anything), scanf() may not be the best choice. You need to handle whitespace characters appropriately as well as clearing the input stream for subsequent input because scanf() doesn’t read newlines.

Note that fscanf() which scans a FILE stream and sscanf() which scans a string are rational and useful choices because they operate on structured data.

fgets: Read the Entire Line

It may be better to read the whole line using fgets(), and then process the input as required:

Use fgets() to read an entire line of stdin within a while loop.
Check the length of the input buffer - even though fgets() discards extra input, the user should be informed if input has exceeded available space and allowed to re-enter the data.
Convert input to an integer using strtol().
If input meets the length constraint and can be interpreted as an integer, the loop ends.

Example: Read Integer

// file: integer-input.h
#ifndef INTEGER_INPUT_H
#define INTEGER_INPUT_H

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>

#define MAX_DIGITS 12

/**
 * This function removes surplus characters from the input buffer.
 * Otherwise, if more than the permitted number of characters have been entered during the
 * call to fgets(), the surplus characters (after MAX_DIGITS chars) remain in the input buffer
 * and will be wrongly accepted as input on the next iteration of the loop.
 * */
static inline void ClearInputBuffer() 
{
	char c = 0;
	// Loop over input buffer and consume chars until buffer is empty
	while ((c = getchar()) != '\n' && c != EOF);
}

void getIntegerFromStdin(int *inputInteger)
{
	char *inputBuffer = malloc(sizeof(char) * MAX_DIGITS);
	memset(inputBuffer, 0, MAX_DIGITS);
	char *input = NULL;
	while (input == NULL) {
		// Note that fgets returns inputBuffer on success.
		// This becomes important when freeing - free either `input` or
		// `inputBuffer` to avoid an attempted double-free error.
		input = fgets(inputBuffer, MAX_DIGITS, stdin);
		
		// If fgets() receives less than MAX_DIGITS, the last char in the array is '\n'.
		// Therefore if the last char is not '\n', too many characters were entered.
		if (inputBuffer[strlen(inputBuffer) - 1] != '\n') {
			fprintf(stderr, "[ERROR]: Too many characters: max input is %d chars.\n", MAX_DIGITS);
			ClearInputBuffer();
			input = NULL;
			continue;
		}

		// Check that the input can be intepreted as an integer
		// Convert to integer using `strtol()`
		errno = 0;
		char *endptr = NULL;
		*inputInteger = strtol(input, &endptr, 10);
		
		// If an integer was not found, endptr remains set to input
		if (input == endptr) {
			// Remove trailing newline by adding NUL at the index of the
			// terminating '\n' character. See man strcspn - this function
			// gets the length of a prefix substring.
			input[strcspn(input, "\n")] = 0;
			printf("Invalid input: no integer found in %s.\n", input);
			input = NULL;
		}
		if (errno != 0) {
			fprintf(stderr, "[ERROR]: That doesn't look like an integer.\n");
			input = NULL;
		}
	}
	free(inputBuffer);
}

#endif

Example of usage:

// file: main.c
#include <stdio.h>
#include "integer-input.h"

int main()
{
	puts("Enter integer:");
	int x;
	getIntegerFromStdin(&x);
	printf("You entered: %d\n", x);
	return 0;
}

References

scanf() leaves newline in buffer
Concise description of fgets() vs scanf() (TLDR; use fgets())
comp.lang.c list - scanf() problems re: stdin
scanf() and whitespace
Clean & compact description of fgets() and strtol()

Conditional Printing in C and C++ Using a Ternary Statement

Wed, 22 May 2019 00:00:00 +0100

This article contrasts how output to stdout differs between C++ and C (std::cout and printf()). In particular, we’re looking at conditional output that includes variables.

C++ Stream Out Operator

The << operator in C++ is the stream insertion operator and it is used to push data into the std::out output stream.

This allows you to print to stdout like this:

#include <iostream>

int main(void)
{
	std::cout << "Hello World!\n";
	return 0;
}

std::cout figures out how to output variables - you can just feed them in and it works automagically:

float price = 10.99;
char *name = "Cheddar";
std::cout << "The price of " << name << " is " << price << '\n';
// Outputs: The price of Cheddar is 10.99

The printf() Function

The printf() function uses placeholders to insert variable values into a string for output to stdout:

#include <stdio.h>

int main(void)
{
	float price = 10.99;
	char *name = "Cheddar";
	printf("The price of %s is %f.\n", name, price);
	return 0;
}

You need to be careful to use the correct conversion specifiers in the placeholder. These are determined by the type of data to be inserted. In the example above, %s denotes a string placeholder and %f denotes a float placeholder.

Python 3 gives you the best of both worlds with the string method format(), which allows you to insert unspecified data types into placeholders within the output string.

Conditional Output

If you want to output content that is conditional, you can embed ternary statements within the C++ std::cout output stream:

int cheese = 0;

// Example of a simple ternary with output stream operators.
std::cout << (cheese ? "We have cheese." : "+++Out of Cheese Error+++") << std::endl;

If you want to embed variables in your conditonal output stream, this method doesn’t work. As far as I can tell, you need to employ an if/else block.

By way of contrast, it’s pretty easy to embed variables within a ternary statement using printf().

In the following example the objective is to let the user know the level of cheese if the level of cheese is greater than zero, but to display a warning if the level of cheese is zero.

The example is C++, but the printf() statements are valid in C.

#include <iostream>
#include <stdio.h>

int main(void)
{
	int cheese = 0;

	// This achieves what we need, but is pretty verbose:
	if (cheese)
		std::cout << "Cheese level: " << cheese << '\n';
	else
		std::cout << "+++Out of Cheese Error+++\n";

	// Format string the same for both conditions - not a good solution
	// since '0' is printed in the empty case.
	printf("%s%d\n", (cheese ? "Cheese level: " : "+++Out of Cheese Error+++"), cheese);
	
	// Better solution - choose the format string in a ternary statement
	printf(cheese ? "Cheese level: %d\n" : "+++Out of Cheese Error+++\n", cheese);
	
	return 0;
}

Copy to Clipboard in Vim under Ubuntu

Thu, 02 May 2019 00:00:00 +0100

To copy to clipboard in Vim under Ubuntu:

Select a block to copy using Visual mode.
Use "+y to yank to clipboard (instead of Y)

C++ error ambiguating new declaration

Thu, 02 May 2019 00:00:00 +0100

If you receive a C++ compiler error message that includes the words “ambiguating new declaration”, check that you have the correct return type in your function declaration and definition.

Error Message

g++ -W -Wall -std=c++17 -o bin/main IntArray.cpp main.cpp
IntArray.cpp: In function ‘void mySwap(IntArray&, IntArray&)’:
IntArray.cpp:77:37: error: ambiguating new declaration of ‘void mySwap(IntArray&, IntArray&)’
 void mySwap(IntArray& a, IntArray& b)
                                     ^
In file included from IntArray.cpp:1:0:
IntArray.h:28:19: note: old declaration ‘IntArray& mySwap(IntArray&, IntArray&)’
  friend IntArray& mySwap(IntArray& a, IntArray& b);
                   ^

Cause

// IntArray.h
...
// wrong return type!
friend IntArray& mySwap(IntArray& a, IntArray& b);

// IntArray.cpp
...
void swap(IntArray& a, IntArray& b)
{
	std::swap(a.m_ptr, b.m_ptr);
	std::swap(a.m_size, b.m_size);
}

If you’re relatively new to C++ (or maybe just tired) you might miss this. It may not seem like the most obvious error message, but the caret in the second error really pinpoints the issue.

In this case, the compiler has read the function declaration/definition in the file IntArray.cpp and spotted a problem: “error: ambiguating new declaration…”.

The declaration is incompatible with the declaration in the header file because the return type differs. The first error uses the word “new” because it can see the declaration has already been made (in the header file). This is not necessarily a problem - as function overloading involves re-declaring a function with different parameter types. However in this case, the return type differs - this is not part of function overloading, so the declarations differ and the compiler can’t make sense of what you need.

Cognition, Attention & Debugging

It’s interesting that obvious bugs like this can trip you up.

I think viewing the function declaration and definition at the same time helps. If the two files are in different tabs, you’re relying on your memory to reliably take a snapshot of the definition/declaration as you switch between files. I suspect that this is less reliable in terms of spotting bugs than viewing the header and implementation files side by side.

Operator Overloading Custom Output Stream in C++

Thu, 25 Apr 2019 00:00:00 +0100

It can be useful to display data from a user-defined type (e.g. an object instantiated from a class) using the std::cout object. This can be achieved by overloading the << operator.

This would allow you to print selected data members in a meaningful way. For example, if you have a class Point which holds data members x and y to represent horizontal and vertical displacement, the following might be a useful way to display a Point object:

Point a(10, 2);

std::cout << a << std::endl;
// Prints "(10, 2)"

You need to set up the insertion operator such that it accepts an std::ostream object on the left and a user-defined type on the right. You achieve this by declaring the overloaded operator as a friend of the class. This allows the function to accept the required parameters, and provides access to private data within the class.

The function is then defined as a free operator -i.e. not a member of a class.

Example

// Point.h
#include <iostream>

class Point {
friend std::ostream& operator<<(std::ostream& os, Point p);
private:
	float m_x;
	float m_y;
public:
	Point(float x, float y);
};

// Point.cpp
#include "Point.h"

// Member function declarations
Point::Point(float x, float y) : m_x(x), m_y(y)
{}
...
std::ostream& operator<<(std::ostream& os, Point p)
{
	// Because this is a friend function, we have access to the object's private data
	os << "(" << p.m_x << ", " << p.m_y << ")";
	return os;
}

Implementation in main.cpp:

...
Point a(99.9, 2.3);
std::cout << a << std::endl;
// Output: (99.9, 2.3)

Don’t Declare the Operator Overload as a Member of the Class

If you declare the operator<< as a non-static data member - i.e. if it is declared in the class, accepting a std::iostream& and Point as parameters, it will cause a compiler error:

g++ -W -Wall -std=c++17 -o bin/main Point.cpp main.cpp
In file included from Point.cpp:1:0:
Point.h:14:53: error: ‘std::ostream& Point::operator<<(std::ostream&, Point)’ must take exactly one argument
  std::ostream& operator<<(std::ostream& os, Point p);

This occurs because:

A non-static member method takes an implicit this parameter that is passed as a hidden argument
The << operator takes exactly two arguments e.g. for a << b, the arguments are a and b
Declared as a member function, there are too many arguments!

It doesn’t make logical sense for the overloaded << operator to be a member function, since in it’s implementation it needs to get data from a passed in object and pass this along to the output stream.

TLDR;

Declare the operator<< as a friend of the class.
DO NOT define the function as a member of the class - define it as a free function outside the class.
The friend declaration ignores the private/public designators

References

Import Private Key from Bitcoin Paper Wallet to Bitcoin Core

Sun, 03 Mar 2019 00:00:00 +0000

Import a private key from a BIP38 encrypted Bitcoin paper wallet to a Bitcoin Core (Bitcoin-QT) client wallet.

This article outlines how to import a BIP38 encrypted private key (for example, a Bitcoin paper wallet) to a Bitcoin Core wallet. This process allows the Bitcoin Core client to access and control the funds associated with the paper wallet Bitcoin address.

The process is quite involved - and much of the online documentation is either incomplete or outdated.

Note: this article describes importing rather than sweeping the paper wallet.

Importing vs Sweeping

Importing a private key hands control of the Bitcoins linked to the paper wallet to the Bitcoin Core client - the funds remain attached to the original Bitcoin address, as the private key is only imported. As such, if others have access to the paper wallet private key, they would also be able to gain access to the funds.

In contrast sweeping the private key involves making a transaction whereby all Bitcoins held on the wallet are sent to a new address managed by the receiving client - leaving the original (paper) wallet empty, and the funds in the full and exclusive control of the Bitcoin client.

The Bitcoin Core client does not have a built in function for ‘sweeping’ funds. Other wallets, such as the Electrum desktop client, Mycelium mobile wallet do have this function. I haven’t checked these.

Decrypting the Private Key

Access to the private key equates to full control of the funds attached to the Bitcoin address. Be careful not to expose the private key during the decryption & import process.

Decryption can be done using bitaddress.org. Download bitaddress.org and run locally after verifying the downloaded copy. For extra security (or added paranoia, depending on your viewpoint), run a verified copy of bitaddress.org in an offline Tails session. You could use Tails along with a persistent encrypted data partition on a USB drive with any offline computer to generate/decrypt Bitcoin private keys.

To decrypt the private key:

Download and verify a copy of bitaddress.org - e.g. bitaddress.org-vX.X.X-SHA256-XXX.html

Optional - for Extra Security

Copy bitaddress.org-vX.X.X-SHA256-XXX.html to a newly-formatted USB key
Disconnect computer from the internet
Boot into a Tails session
Add encrypted persistent storage the the Tails USB drive
Store the bitaddress.org-vX.X.X-SHA256-XXX.html file in ~/Persistent/Tor Browser/ (Note that Tor restricts local file directories - location matters)

If you’re certain that your system is secure (without malware or keyloggers etc), you could skip the previous steps and just disconnect from the internet. Personally, I take the time to run a verified bitaddress.org copy in an offline Tails session.

Common Actions

Navigate to a local copy of bitaddress.org-vX.X.X-SHA256-XXX.html in a browser
Click “Wallet Details”
Enter the encrypted private key
Select “BIP38 Encrypt?”
Enter the encryption passphrase that was used to generate the encrypted key
Click “Decrypt BIP38”

The decryption output includes the Bitcoin address, as well as public and private keys in a wide range of formats, including:

Uncompressed Bitcoin Address
Compressed Bitcoin Address
Public Key Uncompressed (130 characters [0-9A-F])
Public Key Compressed (66 characters [0-9A-F])
Private Key Hexadecimal Format (64 characters [0-9A-F])
Private Key Base64 (44 characters)
Private Key BIP38 Format (58 characters base58, starts with ‘6P’) - the original encrypted key
Private Key, Wallet Import Format
Private Key, Wallet Import Format Compressed - this is probably what you need.

Compressed vs Uncompressed Address/Private Key

Both compressed and uncompressed Bitcoin addresses are generated from the public key. The compressed version of the public key is almost half the size, and generates a Bitcoin address that is distinct from the uncompressed public key.

Because compressing keys significantly reduces blockchain space without losing any data, compressed keys are the recommended default.

To access funds, the compression state of the keys must match. If you’ve sent funds to the Bitcoin Address generated from the compressed public key, you’ll need to import the compressed Private Key (i.e Wallet Import Format compressed).

If you generated your paper wallet using bitaddress.org, the Bitcoin addresses on the paper wallet are compressed.

This means that you need to import the Private Key WIF Compressed. This address has 52 base58 characters and starts with a ‘K’ or ‘L’.

If you accidentally import the uncompressed private key, you’ll probably see a balance of zero. This doesn’t mean you’ve lost funds - you would have sent funds to the compressed address, and you therefore need to import the compressed private key.

Importing The Private Key

This is accomplished with the importprivkey command in the Bitcoin Core CLI. This command adds a private key to your wallet - it would also serve to import a private key as created by the dumpprivkey command.

The importprivkey command has the format: importprivkey “Kthebitcoinprivkey” ( “label” ) ( rescan )

Arguments:

“bitcoinprivkey” (string, required) The private key (see dumpprivkey)
“label” (string, optional, default=””) An optional label
rescan (boolean, optional, default=true) Rescan the wallet for transactions

Note: This call can take minutes to complete if rescan is true.

Examples:

# Dump a private key
bitcoin-cli dumpprivkey "myaddress"

# Import the private key with rescan
bitcoin-cli importprivkey "privatekey"

# Import using a label and without rescan
bitcoin-cli importprivkey "privatekey" "testing" false

# Import using default blank label and without rescan
bitcoin-cli importprivkey "privatekey" "" false

# As a JSON-RPC call
curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "importprivkey", "params": ["mykey", "testing", false] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/

Resources

Strings in C

Wed, 29 Aug 2018 14:49:00 +0100

In C, strings are arrays of characters terminated by an ASCII NUL character '\0'.

A string is a contiguous sequence of characters terminated by and including the first null character. The term multibyte string is sometimes used instead to emphasize special processing given to multibyte characters contained in the string or to avoid confusion with a wide string. A pointer to a string is a pointer to its initial (lowest addressed) character. The length of a string is the number of bytes preceding the null character and the value of a string is the sequence of the values of the contained characters, in order.

–C11 ISO/IEC 9899:2011 Definition of string (p 180)

String Initialisation

Strings can be declared as an array of characters or as a pointer to a character. In either case, a character string literal can be used to initialise the value.

String literals are a collection of characters surrounded by double quotes. The terminating ‘\0’ character is implicit.

String Declared as Character Array

For example:

// Declare an empty string, big enough for 5 characters
char name[6];

An array declared to have 6 characters in this way can only store 5 actual characters - one element must be reserved for the termination character.

A string can also be declared by initialising with a string literal:

char name[] = "Alice";

This defines an array of char objects, whose elements are initialised with a character string literal (with an implicit null terminating character).

In a similar way, you can also initialise with a comma separated list - note that you need to explicitly add the null character in this case:

char name[6] = {'A', 'l', 'i', 'c', 'e', '\0'};

Char Array Objects are Mutable

String literals defined as arrays (char name[] = "Mary") are stored in the stack can be modified. If a string literal was declared as a char array, you can dereference and amend elements.

Change the value stored in name from “Alice” to “Alina”:

// Amending a character array defined string is OK
// -----------------------------------------------
char name[] = "Alice";
name[3] = 'n';
name[4] = 'a';

// Modifying a string defined as a character pointer is NOT OK
// The following will compile but results in undefined behaviour.
// The result will probably be a segmentation fault
// ------------------------------------------------
char *name2 = "David";
name2[3] = 'e'; // DON'T DO THIS IF NAME IS DEFINED AS A CHARACTER POINTER
name2[4] = '\0';

String Declared as Pointer to Char

You can also define a string as a pointer to a char, initialised by a string literal. In this case, string literals are stored in a read only section of memory and are effectively constant. For example:

char *name = "Bob"

In this case, the value is stored in a read-only section in the binary file and cannot be modified. If you compile to an assembly file (use the -S compiler option in gcc), you can see the string literals in the .rodata section. In this context, rodata means “read-only data”.

/* main.s */
.file	"main.c"
.section	.rodata
.LC0:
.string	"Bob"

Though the string literal is read-only, you can modify your char * string by pointing it to a different string literal. Following on from the above example:

// First Definition
char *name = "Bob";

// Redefine the char pointer
name = "Bub";

// This re-assignment is OK - you will see both string literals in your assembly file

You can declare a variable as a pointer to a character without specifying the string length & location. For example: char *name;.

You can then dynamically allocate memory to the string later.

String Declaration: ISO/IEC 9899:201x

The declaration char s[] = "abc", t[3] = "abc"; defines “plain” char array objects s and t whose elements are initialized with character string literals. This declaration is identical to char s[] = { 'a', 'b', 'c', '\0' }, t[] = { 'a', 'b', 'c' }; The contents of the arrays are modifiable.

On the other hand, the declaration char *p = "abc"; defines p with type “pointer to char” and initializes it to point to an object with type “array of char” with length 4 whose elements are initialized with a character string literal.

If an attempt is made to use p to modify the contents of the array, the behavior is undefined.

– C11 ISO/IEC 9899:2011 Section 6.7.9

References

C11 ISO/IEC 9899:2011

C++ Namespace Template Functions

Wed, 22 Aug 2018 19:10:00 +0100

If you define a C++ template function in a namespace, the function definition needs to go in the header - not the .cpp file.

Example

// utilities.h
#include <vector>
#include <iostream>
#include "Employee.hpp"

namespace utilities
{
    // Function declaration - this function can be defined in utilities.cpp
    void employeesReport(std::vector<Employee> staff);

    // Template function must be defined in utilities.h
    template <typename T>
    void outputVec(const char * label, std::vector<T> collection)
    {
        std::cout << label << ": " << std::endl;;
        for(typename std::vector<T>::iterator it = begin(collection); it != end(collection); it++)
            std::cout << *it << (std::next(it) != collection.end() ? ", " : "");
        std::cout << std::endl;
    }
}

You could also put the implementation in .cpp and include this file from within the header file.

Why?

Template functions are not real functions - they are turned into a real function by the compiler when it encounters a use of the function.

This means that the template definition needs to be in the same scope as the call to the function. If the function is called in main.cpp and utilities.h is included in this file, then the template must be defined in utilities.h.

References

SO answer with good explanation

Radix 64 Encoding: Implementation in C

Tue, 21 Aug 2018 20:09:00 +0100

Radix 64 or base 64 is a binary-to-text encoding system that is designed to allow binary data to be represented in ASCII string format. This is an educational project accepts a string from a user and Radix 64 encodes it.

A GitHub repo accompanies this post, with a working example in C.

What is Radix 64 Encoding?

Radix 64 encoding allows binary data stored in octets (i.e. bytes) to be expressed as printable characters.

Radix-64 characters require the binary input to be split into blocks of 6. These numbers (which all range from 0 - 63) are then mapped onto a character set of printable characters.

The Radix 64 characterset includes:

A-Z
a-z
0-9
The additional characters ‘+’ and ‘/’

This amounts to a total of 64 symbols. To encode binary data into Radix 64, data is parsed in 6-bit blocks (i.e. such that each block has a maximum value of 64), and the number represented by each 6-bit block is used to look up a Radix 64 character.

Number of Output Bytes

Radix 64 encoding results in 33% more bytes - every 3 input bytes is converted to 4 output.

Given n input bytes, the output will be:

4⌈n/3⌉

The ceiling brackets ⌈x⌉ mean round the number to the upper integer. In words, this expression would be:

“The number of output characters is n divided by 3, rounded up to the next whole number and then multiplied by 4”.

Example: PGP Messages

A good example is the ascii-armor output from PGP:

jA0ECgMCl9ViayBOZkZg0kkBDqU+4ofR+bJDXd+cpfAQCk30pFcK4QmtFXYivhqy
N8WrBUN8ala9bJ8ON2+COaB1ls+Pr9ohpiWSQLlC6t6/fQLSsHFLCJq5
=GH0r

The above example is the ciphertext (excluding cleartext message headers) that results from ASCII armored symmetric encryption of the string “Hello World!” (additional metadata is added by PGP - including a salt and the cipher algorithm designator).

Note that the encrypted message consists of Radix 64 characters exclusively, with the ‘=’ sign used as padding - so in the message above, data after the ‘=’ is not part of the message. You wouldn’t see this padding if the number of the plaintext bytes was an exact multiple of three (and therefore did not require padding).

By definition, each byte of the original ciphertext was a number from 0-255. Some of these numbers do not map to printable ASCII characters. Encoding the message as Radix 64 allows the message to be printed and/or sent as an email or text message - modes of communication which do not allow the transfer of binary data.

The Radix 64 Alphabet

Index R64 value  Index R64 value  Index R64 value  Index R64 value
0     A          17    R          34    i          51   z
1     B          18    S          35    j          52   0
2     C          19    T          36    k          53   1
3     D          20    U          37    l          54   2
4     E          21    V          38    m          55   3
5     F          22    W          39    n          56   4
6     G          23    X          40    o          57   5
7     H          24    Y          41    p          58   6
8     I          25    Z          42    q          59   7
9     J          26    a          43    r          60   8
10    K          27    b          44    s          61   9
11    L          28    c          45    t          62   +
12    M          29    d          46    u          63   /
13    N          30    e          47    v
14    O          31    f          48    w         (pad) =
15    P          32    g          49    x
16    Q          33    h          50    y

Example Radix 64 Encoding

Radix 64 encoding can be difficult to get your head around. Before building a programme to do the encoding, it can help to step through the process manually.

This example will encode the string ‘david’ into Radix 64.

The string is represented by 5 chars.
The total size in bits (assuming that one char == 1 byte) is given by 8 x 5 = 40 bits.
Dividing the total number of bits by 6 leaves remainder 4 - therefore 7 sextuplets will be needed to represent the string in Radix 64
The last sextuplet will have 2 bits of padding. When this is the case, ‘=’ padding character is added.

                +-----------+-----------+-----------+-----------+-----------+
char:           | 'd'       | 'a'       | 'v'       | 'i'       | 'd'       |
                +-----------+-----------+-----------+-----------+-----------+
binary:         | 011001 00 | 0110 0001 | 01 110110 | 011010 01 | 0110 0100 |
                |                                                           |
6 bit groups:   | 011001 | 000110 | 000101 | 110110 | 011010 | 010110 | 0100  | 00  |
Decimal output: | 25     | 6      | 5      | 54     | 26     | 22     | 16    |     |
                +--------+--------+--------+--------+--------+--------+-------+-----+
Radix-64 chars  | Z      | G      | F      | 2      | a      | W      | Q     | =   |
                +--------+--------+--------+--------+--------+--------+-------+-----+

Radix 64 Encoding in C

The encoding takes place in the function base64Encode(), defined in base64.c.

The function receives:

The input buffer inputBuffer - which is a pointer to a character array (a c-style string) that contains the user input
The output buffer b64Buffer - a pointer to a dynamically allocated section of memory, sufficient to contain the Radix 64 encoded output

For the first character, the function:

Takes the first 6 bits, looks up the value against the Radix 64 encoding and assigns the correct ASCII character to b64Buffer[0]
A temporary byte is then created wich stores the data from the unused bits (i.e. the least significant 2 bits from the first char is stored as the most significant 2 bits in tmpByte)
A size_t divider variable is set - this will determine the number of bits to take from the next char - these will be combined with the bits in tmpByte to form the next Radix 64 6-bit group

Bitwise Operations

Bitwise operators allow for bit-by-bit manipulation of data stored in individual bytes.

In our case, we make use of bit-masking to collect bits from specific positions within a byte:

// base64.c
// Handling the first char
// ----------------------
// tmpByte will carry unused bits forward
unsigned char tmpByte = 0;

// We need last 2 bits, so mask against 00000011
tmpByte = inputBuffer[0] & (0xFF >> 6);

// Mask is 11111100 - note usefulness of the complement operator!
unsigned char mask = ~(0xFF >> 6);

// After getting the first 6 bits, shift them
unsigned char lookupVal = (inputBuffer[0] & mask) >> 2;

// Get the b64 character
b64Buffer[0] = encodingTable[lookupVal];

// First run: set the divider for next char
size_t divider = 4;

Originally I gathered the first 6 bits like this:

size_t lookupVal = (inputBuffer[0] >>= 2) & (0xFF >> 2);

…which bit shifts the original char (i.e. inputBuffer[0]) two places to the right before masking against 00111111.

This does collect the correct data for the Radix 64 lookup index - but it has the unwanted side-effect of mutating the original char - which meant that the function signature could not declare inputBuffer as const.

Next, the function loops through the characters. During each iteration:

The correct bits are used to determine the next radix 64 character
Excess bits are stored in tmpByte to be carried forward if necessary
The divider is set such that the correct number of bits are collected when advancing

// Loop to the penultimate character (assuming last char is a newline)
// The last 2 bits of tmpByte is the last 2 bits of the first char
// -------------------------------------------------------------------------
size_t i = 1;
size_t bufIndex = 1;
while (i < inputLength) {
    // Shift bits from previous iteration to the correct position
    tmpByte <<= divider;

    // Mask representing the first `divider` bits of a byte
    unsigned char mask = ~(0xFF >> divider);

    // Collect the first `divider` bits of the current char
    unsigned char mostSigBits = inputBuffer[i] & mask;

    // Shift to correct position
    mostSigBits >>= (8 - divider);

    // Add combined bytes to the return buffer
    b64Buffer[bufIndex] = encodingTable[tmpByte ^ mostSigBits];
    bufIndex++;

    // If the divider == 2, the rest of this byte shoudl form the next radix 64 char.
    // Set the last 6 bits of this char as a new radix 64 character
    // ---------------------------------------------------------------------
    if (divider == 2) {
        b64Buffer[bufIndex] = encodingTable[(inputBuffer[i] & (0xFF >> divider))];
        bufIndex++;
        // In this case, there is no data to carry forward
        tmpByte = 0;
    } else {
        // Carry forward the unused bits from the current character
        // Apply a logical & against a mask of the last bits
        tmpByte = inputBuffer[i] & (0xFF >> divider);
    }

    // Set divider for the next char
    // ---------------------------------------------------------------------
    divider = (divider - 2) ? divider - 2 : 6;
    i++;
}
// Add the remaining data from the tmpByte
b64Buffer[bufIndex] = encodingTable[tmpByte <<= divider];

The divider is set divider = (divider - 2) ? divider - 2 : 6;:

If the current divider is 2, set divider to 6 (because no data is carried forward from the current character)
If the divider is not 2, set the divider to divider - 2:
- Current divider is 4, 4 bits carried forward, new divider is 2

Notes on Decoding

To decode radix 64 encoded text, typically four characters are converted to three bytes. If the string contains a single padding character (i.e. ‘=’) the last four characters (including the padding character) will decode to only two bytes, while ‘==’ indicates that the four characters will decode to only a single byte. From RFC 1521:

Since all base64 input is an integral number of octets, only the following cases can arise:

the final quantum of encoding input is an integral multiple of 24 bits; here, the final unit of encoded output will be an integral multiple of 4 characters with no ‘=’ padding

the final quantum of encoding input is exactly 8 bits; here, the final unit of encoded output will be two characters followed by two ‘=’ padding characters

the final quantum of encoding input is exactly 16 bits; here, the final unit of encoded output will be three characters followed by one ‘=’ padding character.”

Manual example:

radix 64 chars  | T | Q | = | =  |
TQ== goes to a single byte
radix 64 int:   | 19  | 16  |
binary:         | 010011 | 010000 |
First 8 bits:   | 01001101 | last 4 bits are "padding" zeroes
Decimal:        | 77 |
ASCII:          | M  |

Resources

GitHub repo with sample code
Wiki article: Base 64 encoding
RFC 1421: Privacy Enhancement for Internet Electronic Mail: Part I: Message Encryption and Authentication Procedures
ASCII armor in PGP

Double Pointers and Linked List in C

Mon, 23 Jul 2018 14:53:00 +0100

This article will (hopefully) demonstrate the usefulness of double pointers (multiple indirection) in manipulating linked list elements efficiently.

Intro to Linked Lists

A linked list is an abstract data structure that is made up of a collection of nodes (or elements). Lists nodes are accessed by means of sequential access - they are accessed in an ordered/predetermined sequence.

List objects can be stored anywhere in memory - they do not need to be next to one another. This makes it easy to insert new elements, but has the disadvantage of requiring sequential access when accessing values.

Linked List Using Double Pointers

For the current example, nodes are dynamically allocated (using malloc()) and each node consists of a data field and a reference (a pointer) to the next node in the list. In the case of the last node in the list, the next field contains NULL - it is set as a null pointer.

Our nodes are defined with a C struct - for the purposes of this exercise, nodes are defined as NODE using typedef. The data field of the node consists of a single char *name member variable:

// list.h
typedef struct node
{
    char *name;
    struct node *next;
} NODE;

After this typedef declaration, NODE *head defines a variable head which is a pointer to a node struct pointer.

For the sake of simplicity this example will consider two actions:

prepend(): add a node to the start (head) of the list
append(): add a node at the end of the list

Pointers

A pointer is a variable that stores the memory address of another variable. In C, you need to define the type of the object that is being pointed at.

For example: int a = 42; int *pNum = &a;.

The variable pNum now contains the address of a, which is the memory location of an object having the data type integer. Dereferencing the pointer variable *pNum instructs the programme to return the integer value that is held at the memory address returned by pNum.

NOTE: you shouldn’t try to dereference a null pointer in C - this produces undefined behaviour.

Initialise the Linked List

The list is initialised by creating a NODE *head which is set to NULL. The variable head is now a pointer to NULL, but as NODEs are added to the list head will be set to point to the first NODE. In this way, head becomes the access point for sequential access to the list.

Adding Nodes

As the first node is added, the head pointer is amended to point to the new node, and the next pointer of the new node points to NULL. In the case of the first node, append() and prepend() have exactly the same effect.

In each case, a node needs to be created. To achieve this:

// Function to create a node and return a pointer to the node.
NODE *createNode(char *name)
{
    NODE *p = malloc(sizeof(NODE));
    p->name = malloc(strlen(name) + 1);
    strcpy(p->name, name);
    p->next = NULL; // This will be set on insertion
    return p;
}

Why Double Pointers?

Arguments are always passed to functions by value in C. In other words, when C passes control to a function, a copy of each argument is made and this copy is passed to the function - leaving the original variable unchanged.

In order to amend a variable in a calling function from within a called function, you need to pass a pointer to the variable as a function argument. This provides the called function with the memory address of the variable to be amended. Dereferencing this (pointer) variable within the called function provides access to the original value.

Simplified example:

// Function argument is the memory address of input, NOT the actual variable
void doubleInput(int *input)
{
    // Dereference the pointer to provide access:
    // *input refers the the value held by the memory address input.
    // By dereferencing the address, we can directly amend what is stored there
    // from within the function.
    *input = *input + *input;
}

in main()
{
    int myNum = 5;
    printf("%d", myNum); // Outputs 5
    void doubleInput(&myNum); // Pass the address of myNum to doubleInput()
    printf("%d", myNum); // Outputs 10
    return 0;
}

Prepend

In the function definition for prepend() shown below, the first parameter is NODE **head. This specifies that the function receives a variable with the pointer to pointer variable type NODE **. In other words, the function receives an address of a variable which is in turn a pointer to a NODE- in this case, the function argument is the address of the variable NODE *head which is a pointer that points to either the first node in the list or to NULL.

If a new node is added to the head of the list (by means of a prepend() action), the head pointer is amended to point to the new node and the next field of the new node refers to the address of the second (previously first) node.

This is the easier function to understand:

// The caller
NODE *head = malloc(sizeof(NODE));
head = NULL;

// Prepend node with the name "Goldfish"
prependNode(&head, createNode("Goldfish"));

void prependNode(NODE **head, NODE *newNode)
{
    newNode->next = *head;
    *head = newNode;
}

In the case of the new node being the first node in the list, head == NULL so the assignment newNode->next = *head correctly sets the next field of the last node (in this case, also the first node) to NULL.

If the list already contains nodes, the head variable holds a pointer to a pointer to the existing first node. Note that the address of head is passed into the prepend() function from the caller.

Because we want to insert the new node before the existing first node, the next field of the new node should point to the address of the existing first node. The declaration newNode->next = *head correctly makes this assignment.

Setting the new head reference is where double pointers become useful. Because we pass in the address of the head variable rather than the variable itself, we can access (and amend) it’s value from within the prepend() function. When the *head is set to the address of the new node from within prepend() we are saying:

“Dereference **head once to get the memory address that is held by head in the calling function, and set this address to the address of the new node.”

Append Node

If a new node is added to the end of the list (by means of append(), the next field of the previously last node is amended to point to the memory address of the new node. If the appended node is also the first node in the list, then the head pointer must be amended to point to the new node.

In this case, the challenge is to efficiently find the last node, whilst accounting for the possibility that this may be the first node in the list. Note that in any case the next field of the new node points to null.

void append(NODE **headRef, NODE *newNode)
{
    NODE **tracer = headRef;
    while (*tracer) {
        tracer = &(*tracer)->next;
    }
    newNode->next = *tracer;
    *tracer = newNode;
}

Create a tracer and set this initially to hold the memory address of head. Set tracer to the address of the pointer to the next NODE. In the case of an empty list or the last NODE, this value will be NULL.

NOTE: *tracer dereferences tracer once, so it refers to a pointer to a NODE.

The arrow operator can then be used to indirectly access the member variable. The following are equivalent:

tracer = &(*tracer)->next;
tracer = &(*(*tracer)).next;
tracer = &(**tracer).next;

Note that tracer must be a pointer to a pointer to a NODE - it must contain the memory address of a pointer to a NODE (i.e. NODE *).

After the loop, *tracer will be a pointer to NULL even if the list is empty. *tracer now refers to the pointer to the next node of the last node. Currently NULL, the next line refers it to the new NODE.

References

First & Last Element in C++ Iterator Loop

Thu, 19 Jul 2018 14:00:00 +0100

Iterator-based iteration in C++ works for all containers, including lists and associative containers.

When looping with an iterator, the index of the current iteration isn’t immediately obvious, unless you resort to incrementing a counter.

First Element

It’s pretty straightforward to get the first element - you just compare the iterator at the current iteration through container with container.begin()

Last Element

To get the last element in an iterator loop you can use std::next() (from C++11). The loop is generally terminated by iterator != container.end(), where end() returns an iterator that points to the past-the-end element. If container.next(iterator) == container.end() returns true, you’re on the last element.

Example

#include <iostream>
#include <string>
#include <map>

int main()
{
    std::map<std::string, int> myMap{{"Rocky", 1}, {"Ronnie", 2}};
    std::map<std::string, int>::const_iterator it;

    for (it = myMap.begin(); it != myMap.end(); it++) {
        // Test for first element
        std::cout << (it == myMap.begin() ? "| " : "")
        << it->first << " = " << it->second

        // Test for last element
        << (std::next(it) != myMap.end() ? ", " : " |");
    }
    std::cout << std::endl;
    return 0;
}

// Output:
// | Rocky = 1, Ronnie = 2 |

References

Default Arguments in C++ Functions

Wed, 18 Jul 2018 20:25:00 +0100

Function parameters in C++ can have a default argument. If the function is declared, the default argument is specified in the declaration - NOT the function definition.

If a function declaration contains a parameter with a default argument, all subsequent parameters must also have a default argument supplied.

If your function is not declared, the default argument can be specified in the function definition.

Example

// The function declaration contains the default parameter
int sum(int a, int b = 10);

// The function definition does not contain the default parameter
int sum(int a, int b)
{
    return a + b;
}

// If there is no function declaration, the default argument is specified in the function definition:
double multiply(double a, double b = 10)
{
    return a * b;
}

int main()
{
    std::cout << sum(2) << std::endl;
    // Outputs 2 + 10 = 12

    std::cout << sum(2,2) << std::endl;
    // Outputs 2 + 2 = 4

    std::cout << multiply(2) << std::endl;
    // Outputs 2 * 10 = 20

    std::cout << multiply(2,2) << std::endl;
    // Outputs 2 * 2 = 4
}

References

Default Arguments: CPP Language reference

Arrow Operator in C++

Wed, 18 Jul 2018 13:46:00 +0100

In C++ . is the standard member access operator. It has higher precedence than the * dereference operator.

Accessing the member of an object through a pointer requires dereferencing to happen first, so the dereferencing operation must be wrapped in parentheses.

Example 1: Access Member Variable

int main()
{
    struct Resource {
        int id;
        const char * name;
    };
    struct Resource mainPerson{1, "Cornelius"};
    Resource *p = &mainPerson;

    // Dereference first, then access member variable:
    std::cout << "ID:\t" << (*p).id << std::endl;
    std::cout << "ID:\t" << (*p).name << std::endl;

    // Equivalent:
    std::cout << "ID:\t" << p->id << std::endl;
    std::cout << "Name:\t" << p->name << std::endl;
    return 0;
}

Example 2: Iterate over a Map

When you use iterators to loop through a container, you need to dereference the iterator to access each object. The arrow operator can provide a convenient shortcut for this:

int main()
{

    std::map<std::string, double> myMap{{"David", 99.9}, {"Elaine", 99.99}, {"Bill", 45}};
    std::map<std::string, double>::iterator myMapIt;

    for (myMapIt = myMap.begin(); myMapIt != myMap.end(); myMapIt++) {

        // Accessing the member requires dereferencing to happen first,
        // hence the placement of parentheses:
        std::cout << (*myMapIt).first << ": " << (*myMapIt).second << std::endl;

        // The arrow operator is shorthand for this, providing member access by an
        // indirect selector. This is equivalent to the line above:
        std::cout << myMapIt->first << ": " << myMapIt->second << std::endl;

    }
    return 0;
}

HTTP Server in C

Tue, 12 Jun 2018 21:26:00 +0100

This article describes a simple http server socket in Linux.

Server Socket

The basic procedure:

Create socket with socket() call
bind() this to an IP and port where it can
listen() for connections, then
accept() connection and send() or receive() data to/from connected sockets

Note that if struct sockaddr_in serverAddress.sin_addr.s_addr is set to INADDR_ANY the socket is bound to all local interfaces. INADDR_ANY is a constant set to zero, defined in netinet/in.h. This will correspond to an IP address of 0.0.0.0 in the standard IPv4 notation. Note that htonl(INADDR_LOOPBACK) and inet_addr("127.0.0.1") are functionally equivalent.

Simple Example

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <netdb.h> // for getnameinfo()

// Usual socket headers
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#include <arpa/inet.h>

#define SIZE 1024
#define BACKLOG 10  // Passed to listen()

void report(struct sockaddr_in *serverAddress);

void setHttpHeader(char httpHeader[])
{
    // File object to return
    FILE *htmlData = fopen("index.html", "r");

    char line[100];
    char responseData[8000];
    while (fgets(line, 100, htmlData) != 0) {
        strcat(responseData, line);
    }
    // char httpHeader[8000] = "HTTP/1.1 200 OK\r\n\n";
    strcat(httpHeader, responseData);
}

int main(void)
{
    char httpHeader[8000] = "HTTP/1.1 200 OK\r\n\n";

    // Socket setup: creates an endpoint for communication, returns a descriptor
    // -----------------------------------------------------------------------------------------------------------------
    int serverSocket = socket(
        AF_INET,      // Domain: specifies protocol family
        SOCK_STREAM,  // Type: specifies communication semantics
        0             // Protocol: 0 because there is a single protocol for the specified family
    );

    // Construct local address structure
    // -----------------------------------------------------------------------------------------------------------------
    struct sockaddr_in serverAddress;
    serverAddress.sin_family = AF_INET;
    serverAddress.sin_port = htons(8001);
    serverAddress.sin_addr.s_addr = htonl(INADDR_LOOPBACK);//inet_addr("127.0.0.1");

    // Bind socket to local address
    // -----------------------------------------------------------------------------------------------------------------
    // bind() assigns the address specified by serverAddress to the socket
    // referred to by the file descriptor serverSocket.
    bind(
        serverSocket,                         // file descriptor referring to a socket
        (struct sockaddr *) &serverAddress,   // Address to be assigned to the socket
        sizeof(serverAddress)                 // Size (bytes) of the address structure
    );

    // Mark socket to listen for incoming connections
    // -----------------------------------------------------------------------------------------------------------------
    int listening = listen(serverSocket, BACKLOG);
    if (listening < 0) {
        printf("Error: The server is not listening.\n");
        return 1;
    }
    report(&serverAddress);     // Custom report function
    setHttpHeader(httpHeader);  // Custom function to set header
    int clientSocket;

    // Wait for a connection, create a connected socket if a connection is pending
    // -----------------------------------------------------------------------------------------------------------------
    while(1) {
        clientSocket = accept(serverSocket, NULL, NULL);
        send(clientSocket, httpHeader, sizeof(httpHeader), 0);
        close(clientSocket);
    }
    return 0;
}

void report(struct sockaddr_in *serverAddress)
{
    char hostBuffer[INET6_ADDRSTRLEN];
    char serviceBuffer[NI_MAXSERV]; // defined in `<netdb.h>`
    socklen_t addr_len = sizeof(*serverAddress);
    int err = getnameinfo(
        (struct sockaddr *) serverAddress,
        addr_len,
        hostBuffer,
        sizeof(hostBuffer),
        serviceBuffer,
        sizeof(serviceBuffer),
        NI_NUMERICHOST
    );
    if (err != 0) {
        printf("It's not working!!\n");
    }
    printf("\n\n\tServer listening on http://%s:%s\n", hostBuffer, serviceBuffer);
}

Protocol Family

When setting up a socket using socket() the following protocols are available:

Name	Purpose	Man page
`AF_UNIX`, `AF_LOCAL`	Local communication	`man unix`
`AF_INET`	IPv4 Internet protocols	`man ip`
`AF_INET6`	IPv6 Internet protocols	`man ipv6`
`AF_IPX`	IPX - Novell protocols
`AF_NETLINK`	Kernel user interface device	`man netlink`
`AF_X25`	ITU-T X.25 / ISO-8208 protocol	`man x25`
`AF_AX25`	Amateur radio AX.25 protocol
`AF_ATMPVC`	Access to raw ATM PVCs
`AF_APPLETALK`	AppleTalk	`man ddp`
`AF_PACKET`	Low level packet interface	`man packet`
`AF_ALG`	Interface to kernel crypto API

You can acces this list by running man socket.

Socket Type

The socket has the indicated type, which specifies the communication semantics. Currently defined types are:

Type	Description
`SOCK_STREAM`	Provides sequenced, reliable, two-way, connection-based byte streams. An out-of-band data transmission mechanism may be supported. Used for TCP protocol.
`SOCK_DGRAM`	Supports datagrams - connectionless, unreliable messages of a fixed maximum length. Used for UDP protocol.
`SOCK_SEQPACKET`	Provides a sequenced, reliable, two-way connection-based data transmission path for datagrams of fixed maximum length; a consumer is required to read an entire packet with each input system call.
`SOCK_RAW`	Provides raw network protocol access.
`SOCK_RDM`	Provides a reliable datagram layer that does not guarantee ordering.
`SOCK_PACKET`	Obsolete and should not be used in new programs; see packet(7).

Constructing a Local Address Structure

If serverAddress.sin_addr.s_addr is set to INADDR_ANY the socket is bound to all local interfaces. INADDR_ANY is a constant set to zero, defined in netinet/in.h. This will correspond to an IP address of 0.0.0.0 in the standard IPv4 notation. Note that htonl(INADDR_LOOPBACK) and inet_addr("127.0.0.1") are functionally equivalent.

Bind socket to local address

When a socket is created with socket(), it exists in a name space (address family) but has no address assigned to it. bind() assigns the address specified by serverAddress to the socket referred to by the file descriptor serverSocket. serverAddressLength specifies the size, in bytes, of the address structure pointed to by serverAddress.

This operation is known as “assigning a name to a socket”.

Connecting

The programme runs an infinite loop in which we wait and create a connected socket if a connection is pending.

The accept() function gets the first connection request on the queue of pending connections for the listening socket (in this case denoted by serverSocket). It then creates a new connected socket and returns a file descriptor referring to this socket. The newly created socket is NOT in a listening state. The original socket (serverSocket) is unaffected by this call.

Using `getnameinfo`

Used in the report() function

int getnameinfo(
    const struct sockaddr *sa,  // pointer to a generic socket address structure (of type sockaddr_in or sockaddr_in6)
    socklen_t salen,            // size of the above
    char *host,                 // Caller allocated buffer, will receive host
    socklen_t hostlen,          // Size of above
    char *serv,                 // Caller allocate buffer, will receive service name
    socklen_t servlen,          // Size of above
    int flags);

References

getaddrinfo/getnameinfo Wikipedia

List & Dict Comprehensions in Python

Sun, 27 May 2018 15:04:00 +0100

Python comprehensions are syntactic sugar constructs that provide a way to build a list, dictionary or set from a starting list, dictionary or set whilst altering or filtering elements.

Comprehensions follow mathematical set builder notation rather than map and filter functions.

List Comprehension

Make a List that contains the doubled values of a starting list:

values = [2, 4, 6, 8, 10]
doubled_values = [x*2 for x in values]
print(doubled_values) # Outputs [4, 8, 12, 16, 20]

You could achieve the same result like this:

values = [2, 4, 6, 8, 10]
doubled_values = []
for x in values:
    doubled_values.append(x*2)
print(doubled_values)

…which is considerably more verbose.

List comprehensions have the same effect as the map method in other languages. For example, in JavaScript you could achieve the above like this:

const values = [2, 4, 6, 8, 10]
const doubled_values = values.map((x) => x*2)
console.log(doubled_values) // Outputs [ 4, 8, 12, 16, 20 ]

Filtering

You can filter a list by adding a conditional statement. A filtered list comprehension takes this form:

[ element-expression for element in sequence if boolean-expression ]

For example:

values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
even_values = [x for x in values if x %2 ==0]
odd_values = [x for x in values if x %2 !=0]
print(even_values)
print(odd_values)

# Make a list of the cubes of all even numbers
cubed_even_values = [x**3 for x in values if x %2 ==0]

Flatten a Multi-Dimensional List

To convert a two-dimensional list to a single dimension list, containing all the original elements of the sublists as a single list:

two_d_list = [[1,2,3],[4,5,6],[7,8,9]]
flattened = [item for sublist in two_d_list for item in sublist]
print(flattened) # prints [1, 2, 3, 4, 5, 6, 7, 8, 9]

This is known as “flattening” a list.

Explanation:

flattened = [
    item			# Item to be appended to flattened
    for sublist in two_d_list	# Loop through the list to get the sublists
    for item in sublist		# Loop through each sublist to get the item
]
print(flattened) # prints [1, 2, 3, 4, 5, 6, 7, 8, 9]

Note the order:

The item to append to the collection is referenced first
Sublists are then generated from the initial 2d list
Items are then generated from the sublist

References

JavaScript Functions

Sat, 19 May 2018 20:39:00 +0100

Method Invocation Pattern

Function defined as an object property - an object method:

const myObject = {
  value: 0,
  increment: function(val) {
    this.value += typeof val === 'number' ? val : 1
  }
}

JavaScript binds this at execution (late binding), so this refers to the myObject object.

Function Invocation Pattern

Function invocation uses the format functionName(). In this case, this within the function refers to the global object. This can be incredibly confusing. The problem is evident when using an inner function within a method:

const myObject = {
  value: 1,
  double: function() {
    // In the inner function, `this` refers to the global object (e.g. `Window`).
    // Intuitively, you'd expect it to refer to the containing object - it does not.
    // A workaround is to assign a variable to `this` in the outer function.
    // By convention, this variable is named `that`.
    let that = this
    let helper = function() {
      that.value = that.value * 2
    }
    helper()
  }
}

ES6 arrow functions provide a much less hacky solution, since the arrow function this is inherited from the containing scope. The above example, rewritten with an arrow function:

const myObject = {
  value: 1,
  double: function() {
    let helper = () => {
      this.value = this.value * 2
    }
    helper()
  }
}

Set Up An Automatic LetsEncrypt Renewal Cronjob

Tue, 08 May 2018 15:00:00 +0100

This short article outlines how to setup and test a LetsEncrypt auto-renewal cronjob, tested with certbot 0.24.0 on Ubuntu 14.04.

Depending on your version of Certbot/Letsencrypt, auto-renewal may be built in

Final Command

This cronjob runs at a random second between 02:00 and 03:00 every day:

00 02 * * * /usr/bin/perl -e 'sleep int(rand(3600))' && /opt/certbot-auto renew && /etc/init.d/apache2 restart

This command specifies a pause of between 0-3599 seconds, followed by the certbot renewal. If renewal is successful, an Apache restart is triggered.

Better Command - the above command restarts Apache whether or not the certificate has renewed.

# Final: run at a random second between 02:00 and 03:00 every day
00 02 * * * /usr/bin/perl -e 'sleep int(rand(3600))' && /opt/certbot-auto renew

In this case, the Apache restart is not triggered - the restart is unecessary, since the new cert is symlinked in the Apache site config file.

Quiet Mode

Once you’ve verified that the cammand as working as expected, it’s a good idea to have it run in quiet mode. This suppresses all output apart from error messages, which will help clean up your email inbox.

Test

Check certificate expiry time to verify that renewal has worked:

# Check expiry of cert: replace example.com
sudo openssl x509 -noout -dates -in /etc/letsencrypt/live/example.com/cert.pem
notBefore=May  8 07:34:22 2018 GMT
notAfter=Aug  6 07:34:22 2018 GMT

Set up a test cronjob - this will be the same as the actual script except:

It will trigger at a random second within the specified time
It will force a certificate renewal

# Open crontab for editing
sudo crontab -e

# Test: force certificate renewal at 10:26 on a random second
26 10 * * * /usr/bin/perl -e 'sleep int(rand(60))' && /opt/certbot-auto renew --force-renew && /etc/init.d/apache2 restart

Check that the certifcate expiry time has updated by re-running sudo openssl x509 -noout -dates -in /etc/letsencrypt/live/example.com/cert.pem.

Check that Apache has reloaded:

cat /var/log/apache2/error.log | grep SIGTERM

Check the LetsEncrypt renewal log:

su
cat /var/log/letsencrypt/letsencrypt.log

You should also receive a status report email from cron.

Don’t forget to replace the test command with the actual command.

Passing Data Between Vue Components

Wed, 02 May 2018 11:48:00 +0100

Components are re-usable Vue instances that better allow you to keep code separated and modular. They are very flexible - child components can be defined and used within parents. This can be a really useful way of preventing repetitive markup.

Single-file components take things to the next level. These keep all component data within a single .vue file - comprised of a HTML like structure with <template> (markup), <script> (JavaScript) and <style> (CSS) components. This can be really useful for larger projects or projects in which the entire frontend is built in JavaScript.

If you use vue-cli to spin up a Vue/Webpack project, the project comes with single-file components pre-installed.

This article outlines how to pass data between components in the context of single file components.

Passing Data From Parent Component to Child Component

Import Child in Parent Component

Import the child component so that it can be used in the parent, and declare the component in the <script> section:

<script>
// @ is an alias to /src (in a vue-cli generated project)
import Child from '@/components/Child.vue'

// You also need to declare that the imported child component will be used:
export default {
  components: {
    Child
  }
}
</script>

Invoke the Child Component in the Parent

Within the <template> section of the parent component, call the child component. Prop data is passed as an attribute.

You can pass data inline, or define it within the data() method of the component.

<template>
  <div>
    <!-- simplest prop - pass a string -->
    <Child title="This is my title"></Child>

    <!-- Pass an object, defined inline -->
    <Child :parentData="{msg: 'xxx'}"></Child>

    <!-- Pass an object, defined in the `data()` method -->
    <Child :parentData="myData"></Child>

    <!-- Pass a string variable, defined in `data()`. Note colon. -->
    <Child :stringProp="stringMessage"></Child>
  </div>
</template>

Note that prepending a colon to the prop attribute indicates that a variable is passed, not a string.

Receiving the Prop Data in the Child

You must declare the prop in the child component <script> section - you can then use it as a normal Vue-defined variable:

<script>
export default {
  name: 'Child',
  // To use props, they must be declared
  props: {
    parentData: Object,
    stringProp: String,
    title: String
  }
}
</script>

Pass Data from Child to Parent

You can send data from a child to a parent component by means of Vue’s built-in $emit() method. The first parameter of $emit is the event that should be listened for in the parent component. The second (optional) parameter is the data value to pass.

Parent:

<!-- Parent.vue -->
<template>
  <div>
    <!--
    Listen for `childToParent`: the first parameter of the `$emit` method
    in the child component. We're also listening and reacting to an
    `increment` event - in this case, we increment a counter inline.
    -->
    <Child :parentData="myData" v-on:childToParent="onChildClick" v-on:increment="counter++"></PassProps>
  </div>
</template>
<script>
import Child from '@/components/Child.vue'
export default {
  data () {
    return {
      counter: 0,
      fromChild: '', // This value is set to the value emitted by the child
    }
  },
  name: 'about',
  components: {
    Child
  },
  methods: {
    // Triggered when `childToParent` event is emitted by the child.
    onChildClick (value) {
      this.fromChild = value
    }
  }
}
</script>

In this example, the parent listens for the childToParent event (defined by the $emit method) and triggers the onChildClick() method when the event is received. This method in turn sets the fromChild variable. It also emits an increment event in a simpler way when the button is clicked, by placing $emit inline in the template.

In the child component:

<!-- Child.vue -->
<template>
  <div class="child">
    <!-- Simplest - call `$emit()` inline-->
    <button type="button" name="button" v-on:click="$emit('increment')">Click me to increment!</button>

    <!-- set a variable then trigger a method which calls `$emit()` -->
    <label for="child-input">Child input: </label>
    <input id="child-input" type="text" name="msg" v-model="childMessage" v-on:keyup="emitToParent">
  </div>
</template>
<script>
export default {
  data() {
    return {
      childMessage: ''
    }
  },
  name: 'Child',
  methods: {
    // Define the method that emits data to the parent as the first parameter to `$emit()`.
    // This is referenced in the <template> call in the parent. The second parameter is the payload.
    emitToParent (event) {
      this.$emit('childToParent', this.childMessage)
    }
  }
}
</script>

This example emits a user-entered string back to the parent component.

Persistent Banning of IP Addresses with Fail2Ban

Thu, 26 Apr 2018 09:05:00 +0100

If you’re using Fail2Ban you can easily set up a list of banned IP addresses that Fail2Ban will use to set up DROP rules in iptables whenever Fail2Ban starts. This is very useful since it is easy to persist IP bans across reboots.

You need to modify the relevant action config file, and reference a “blocklist” file. When you add IP addresses to the blocklist and reload Fail2Ban, the relevant drop rules will be added.

Action files specify which commands are executed to ban and unban an IP address. Like with jail.conf files, if you desire local changes create an [actionname].local file in the /etc/fail2ban/action.d directory and override the required settings.

Action files have two sections, [Definition] and [Init]. The [Init] section enables action-specific settings. These can be overridden for a particular jail (in jail.local) as options of the action’s specification in that jail.

IP Blocklist and Associated Action

For our purposes, we will amend the actionstart command in the [Definition] section. This command (or commands) executes when the jail starts. To override the default action, create a corresponding .local file and add the amended actionstart command:

sudo nano /etc/fail2ban/action.d/iptables-multiport.local

# Enter the following to override the default actionstart command:

# Add drop rules for specified IPs.
# Enter IPs to permanently ban, one per line in /etc/fail2ban/ip.blocklist:
actionstart = <iptables> -N f2b-<name>
              <iptables> -A f2b-<name> -j <returntype>
              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
              cat /etc/fail2ban/ip.blocklist | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done

After creating and editing the file, save and exit (ctrl-o followed by ctrl-x).

Create a file /etc/fail2ban/ip.blocklist and enter IP addresses to ban - one per line.

Restart Fail2Ban for the changes to be applied. If you run sudo iptables -S now, you should see rules like -A f2b-ssh -s 11.22.333.444/32 -j DROP associated with your different jails.

Auto Add IP Addresses to the Blocklist

If you want to automatically add IPs to your list as they are banned, you need to amend the actionban command such that the IP is appended to your list when the IP is banned:

actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
            echo <ip> >> /etc/fail2ban/ip.blocklist

Commands specified in the [Definition] section of are executed through a system shell so shell redirection and process control is allowed. Note that commands should return 0, or an error will be logged (ref: man jail.conf).

References

Make Fail2Ban bans persistent

Amend Bitcoin-Qt Persistent Settings for Bitcoin Core

Tue, 10 Apr 2018 19:58:00 +0100

When you first run the Bitcoin core wallet, you will be given an opportunity to customise the location of the default data directory (which in Linux is $HOME/.bitcoin). If you need to change the data directory after the first launch, there is no option to do so from within the GUI. Many Stack Overflow answers suggest launching bitcoin-qt with the -datadir option - e.g. bitcoin-qt -datadir=/path/to/custom/datadir - this works, however you can easily amend the QT configuration so that the proper data directory is referenced by default.

QSettings for Bitcoin

The Bitcoin Core GUI wallet is based on QT, and it uses QSettings to store settings.

Users normally expect an application to remember its settings (window sizes and positions, options, etc.) across sessions. This information is often stored in the system registry on Windows, and in XML preferences files on Mac OS X. On Unix systems, in the absence of a standard, many applications (including the KDE applications) use INI text files. QSettings documentation

Location of Config File: Linux

On Linux/Unix systems, the following files are used by default:

$HOME/.config/Bitcoin/Bitcoin-Qt.conf
$HOME/.config/Bitcoin-Qt.conf

For myself, running Bitcoin Core under Ubuntu 16.04 the config file is $HOME/.config/Bitcoin/Bitcoin-Qt.conf

Location of Config File: Mac OS

On Mac OS X check the following locations:

$HOME/Library/Preferences/com.Bitcoin
$HOME/Library/Preferences/com.Bitcoin
/Library/Preferences/com.Bitcoin
/Library/Preferences/com.Bitcoin

Location of Config File: Windows

On Windows, check the following registry paths:

HKEY_CURRENT_USER\Software\Bitcoin
HKEY_CURRENT_USER\Software\Bitcoin
HKEY_LOCAL_MACHINE\Software\Bitcoin
HKEY_LOCAL_MACHINE\Software\Bitcoin

I don’t run MacOS or Windows so I haven’t checked these locations - this info is based on that provided in the QT QSettings documentation.

Edit the Config

Once you’ve located the config file, you can edit it like any other text file, and you’ll be able to amend the data directory:

# Make a backup just in case
cp $HOME/.config/Bitcoin/Bitcoin-Qt.conf $HOME/.config/Bitcoin/Bitcoin-Qt.conf.bak

# Edit the file
vim $HOME/.config/Bitcoin/Bitcoin-Qt.conf

# Save and exit

The data directory key is stdDataDir and looks like this:

[General]
...
strDataDir=/media/secure-hdd-2/bitcoin-data
...

Just amend this value to reference your data directory.

You can now launch bitcoin-qt as normal without specifying the custom data directory.

A Note on Moving Data Directories

Don’t open Bitcoin Core in a different data directory without first copying across all data from the original data directory (e.g. using rsync). Otherwise, blocks will need to be re-indexed - which is a lengthy process, and unecessary if you already have an up-to-date installation.

References

QSetttings documentation

Combine Vectors in C++

Wed, 04 Apr 2018 14:30:00 +0100

Build a combined vector from two input vectors.

There are a number of ways to achieve this - the function below outlines some options:

using std::string;
using std::vector;

/**
 * Accept two vectors of strings as input parameters.
 * Pass these in by reference for efficiency, and as `const` to protect the originals.
 */
vector<string> vcat(const vector<string>& top, const vector<string>& bottom)
{
    vector<string> ret = top;

    // Option 1: insert()
    ret.insert(ret.end(), bottom.begin(), bottom.end());

    // Option 2: C++ 11 range-based for-loop
    for(const string& line : bottom)
        ret.push_back(line);

    // Option 3: Loop through the extra collection using an iterator
    for(vector<string>::const_iterator it = bottom.begin(); it != bottom.end(); ++it) {
        ret.push_back(*it);
    }
    return ret;
}

Javascript Resources

Sun, 01 Apr 2018 10:26:00 +0100

Collection of resources for learning Javascript.

Node

nodeschool.io:
- Learn javascripting basics npm install -g javascripting
- Learn Node basics: npm install -g learnyounode
- Learn git: npm install -g git-it
Get started with Node

Online Javascript Courses and Cheatsheets

Useful cheat sheet from WebsiteSetup:

JavaScript Cheat Sheet: article with downloadable PDF.

Udacity has loads of free JavaScript courses. I did the Design Patterns course, which was very useful:

Udacity: JavaScript Design Patterns
Code Academy: Javascript online interactive course.
Introduction to Computing Principles
Lynda: Learning Node.js

General Javascript Books & eBooks

Bip32 Generator git clone https://github.com/bip32/bip32.github.io.git ; cd bip32.github.io ; open index.html - Bip32 Deterministic Heirarchical Keys (prefix xpub* and xprv*) based on a simple brainwallet (aka arbitrary mnemonic) passphrase.
Bip39 Mnemonic Code Converter npm install bip39 ; npm run compile lets you create Bip39 mnemonics (typically 12 words) used for deterministic keys, typically for Bip32.

<!– * Bitcoin 101 - Getting Started With Bitcore - A Full JavaScript Implementation of Bitcoin

https://github.com/wobine/blackboard101/blob/master/BitcoreDayOne.html
Bitcoin Quick Tools - Simple browser-based app (loaded locally) that is Bitcore.js. Has key and address generator and simple transaction tool. Useful working example of browserfy bitcore.js code. –>

Function Pointers in C++

Thu, 29 Mar 2018 09:10:00 +0100

In C++ like many other languages, functions can be assigned to variables. Functions can be passed into other functions as parameters.

Full Example

#include <iostream>
#include <vector>

typedef std::vector<int> intVec; // Define std::vector type for brevity

// Declare functions
// ---------------------------------------------------------------------------------------------------------------------
// This function receives the function pointer as third parameter
intVec combineVecs(intVec vec1, intVec vec2, int(*func)(int a, int b));

// These functions will be assigned to function pointers:
int multiply(int a, int b);
int xorInts(int a, int b);

int main(int argc, char const *argv[])
{
    intVec v1 = { 1, 1, 1, 0 };
    intVec v2 = { 0, 1, 0, 1, 10 };

    // Set up function pointers
    // -----------------------------------------------------------------------------------------------------------------
    // Declare a function pointer variable.
    // Defines a symbol xorEls, pointer to a function that takes two ints as arguments and returns an int
    int (*xorEls)(int a, int b);

    // Set it with the address of a function (the use of '&' is optional).
    xorEls = &xorInts;

    // Pass the function pointer for use in another function
    // -----------------------------------------------------------------------------------------------------------------
    // Function pointer is the third function argument.
    intVec xored = combineVecs(v1, v2, xorEls);

    for(int& val : xored) {
        std::cout << val << std::endl; // Output result
    }

    // Repeat with a different function to demonstrate the point of function pointers
    // -----------------------------------------------------------------------------------------------------------------
    // Output a new vector which holds the product of each element in two input vectors.
    int (*multiplyEls)(int a, int b);
    multiplyEls = &multiply;

    intVec multiplied = combineVecs(v1, v2, multiplyEls);

    for(int& val : multiplied) {
        std::cout << val << std::endl;
    }

    return 0;
}

// Function definitions
// ---------------------------------------------------------------------------------------------------------------------
/**
 * Combine two integer vectors - operate on elements of input vectors in place
 * by means of a provided function, creating a new vector. Performs the operation
 * up to the length of the shortest provided vector.
 * @param  vec1 std::vector<int>
 * @param  vec2 std::vector<int>
 * @param  func A function pointer
 * @return zipped A std::vector<int> object with each element operated on in-place.
 */
intVec combineVecs(intVec vec1, intVec vec2, int(*func)(int a, int b))
{
    intVec combined;
    typedef typename std::vector<int>::size_type const containerSize;
    containerSize vec1Size = vec1.size();
    containerSize vec2Size = vec2.size();
    containerSize zipLen = (vec1Size >= vec2Size)
        ? vec2Size
        : vec1Size;
    for (size_t i = 0; i < zipLen; i++) {
        // Append the result of the passed-in function to `combined`
        combined.push_back(func(vec1.at(i), vec2.at(i)));
    }
    return combined;
}

int multiply(int a, int b)
{
    return a * b;
}

int xorInts(int a, int b)
{
    return a ^ b;
}

References

Function Pointers Tutorial

Parse Command Line Arguments in C and C++

Tue, 27 Mar 2018 00:00:00 +0100

Command line arguments are available by adding two parameters.

Run Programme: ./prog -a -b -cde

This passes four arguments to the programme:

./prog
-a
-b
-cde

This data is available within main() as the second input parameter (by convention, argv[]). The first input parameter to main holds the number of passed arguments (in this case, four) and by convention is called argc.

The argv[] parameter is a C-style array of primitive strings (char * objects)- and may be represented as char * argv[] or char ** argv.

Note that main(argc, char ** argv) and main(argc, char * argv[]) are equivalent:

char ** argv: declare argv as pointer to pointer to char (A pointer to char *)
char * argv[]: declare argv as array of pointer to char (A primitive array of char * objects)

They are equivalent because a pointer to char * is the same as an array of char *.

In the current example, argc is 4 and the four elements of argv are pointers to the arrays of characters passed in as arguments.

          +---+         +---+---+---+---+---+---+---+
argv ---->| 0 |-------->| . | / | p | r | o | g | \0|
          +---+         +---+---+---+---+---+---+---+
          | 1 |-------->| - | a | \0|
          +---+         +---+---+---+
          | 2 |-------->| - | b | \0|
          +---+         +---+---+---+---+---+
          | 3 |-------->| - | c | d | e | \0|
          +---+         +---+---+---+---+---+
          | 4 |--------> NULL
          +---+

Parse Command Line Arguments: Demo

#include <iostream>

/**
 * Demo of parsing command-line argument array.
 */
int main(int argc, char **argv)
{
    char c;
    const char * prog = argv[0];

    // Extract arguments from argument array.
    // -------------------------------------------------------------------------
    // Because of the parentheses, argv (a pointer) is incremented first, then
    // dereferenced. The loop conditional is also dependent on the first element
    // of the new pointed-to element being the char '-', which denotes the start
    // of an option. Note that argv is incremented first, which excludes the first
    // element (the command).
    while((++argv)[0] && argv[0][0] == '-')
    {
        // Set appropriate actions
        // ---------------------------------------------------------------------
        // Walk down the string (char array) pointed to by the current argv[0]
        // pointer. The expression `*++(argv[0])` increments argv[0] and provides
        // a pointer to the first non '-' character. Dereferences the pointer to
        // provide the value. Loops to the end of the end of the char array, so
        // multiple options can be read (i.e. `command -abc`).
        // `while(c = *++argv[0])` is the same as `while((c = *++argv[0]) != 0)`
        while(c = *++argv[0])
        {
            switch(c) {
                case 'a':
                std::cout << "Set flag a" << std::endl; break;
                case 'b':
                std::cout << "Set flag b" << std::endl; break;
                case 'c':
                std::cout << "Set flag c" << std::endl; break;
                case 'd':
                std::cout << "Set flag d" << std::endl; break;
                case 'e':
                std::cout << "Set flag e" << std::endl; break;
                case 'h':
                std::cout << "Provide some help!" << std::endl; break;
                default:
                std::cout << "Option " << c << " is unknown." << std::endl;
                std::cout << "usage: " << prog << " <-abcde -h>" << std::endl;
                return 1;
            }
        }
    }
    return 0;
}

C++ Pointers

Thu, 22 Mar 2018 21:24:00 +0000

In C++, a pointer is an object whose value is the memory address of another object. In other words, a pointer is a variable which stored the memory address of another variable.

A pointer references a location in memory. When declaring a pointer, the type of the pointed-to variable must be specified:

int x = 5;
int * xPtr = &x; // Declare xPtr as a pointer to an int x, referenced by the address of x.

double y = 5.55;
double * yPtr = &y; // Declare yPtr as a pointer to a double y, referenced by the address of y.
...

Specifying type when declaring a pointer is necessary because when the pointer is dereferenced (i.e. the pointed-to value is accessed) the type needs to be known.

Dereferencing

To access the value held in the memory address held by a pointer, the pointer is dereferenced using the * operator - the same operator used to declare a pointer variable:

int main(int argc, char const *argv[])
{
    int x = 5; // assign a value to integer variable x
    int * ptr = &x; // Assign the address of x to integer pointer ptr
    std::cout << *ptr << std::endl; // Output the pointed-at value - in this case, 5;
    ...
}

Pointer Access

For:

int x = 5;
int * ptr = &x;

…the following holds:

Variable	Output
`ptr`	Returns the address that `ptr` points to: 0x7fff829afe04
`&ptr`	Returns the pointer’s own address, the address of `ptr`: 0x7fff829afe08
`*ptr`	Dereference the pointer `ptr` - return the value held at the address held (pointed-to) by `ptr`: 5
`&x`	Return the address of `x`: 0x7fff829afe04
`*&ptr`	Dereference the address of ptr - return the value held in the address of `ptr`: 0x7fff829afe04
`*&x`	Dereference the address of `x`, returns the value held in the address of `x`: 5

Pointer to Pointer

Pointers generally contain the memory address of a variable - but they can also contain the address of another pointer.

int main () {
  int num = 10;
  int *numPtr = &num; // Pointer to num
  numPtrPtr = &numPtr; // Pointer to pointer

  std::cout << "Value of num : " << num << std::endl; // Outputs 10
  std::cout << "Dereference numPtr: " << *numPtr << std::endl; // Outputs 10
  std::cout << "Dereference numPtrPtr: " << **numPtrPtr << std::endl; // Outputs 10

  return 0;
}

Using a pointer-to-pointer as a function parameter makes it possible to modify values within a function with those values persisting outside the function.

It is not likely that you need to use pointer-to-pointer type structures in modern C++ code. References are available for passing to a function by reference rather than value, and rather than pointing to an array of strings, it’s probably better to use std::vector<std::string>. However, the char ** notation does pop up - I recently saw this while researching the C getopt() function - in which an example showed arguments to the main() function are presented as char **argv, rather than the more common char char *argv[].

Pointer to Pointer/Pointer to char * array Equivalence

A sequence of strings can be passed to main() as an argument, if the main function is able to receive them.

You give main() the capacity to receive arguments by adding two parameters to the function - an int and a pointer to char.

The following are equivalent:

/**
 * Declare argv as array of pointer to char
 */
int main(int argc, char * argv[])
{
  // Loop through parameters
  for (size_t i = 0; i < argc; i++) {
        std::cout << "argv[" << i << "]: " << argv[i] << std::endl;
  }
}

/**
 * Declare argv as pointer to pointer to char
 * This declaration is valid - as far as I can tell, it is less common in C++.
 */
int main(int argc, char **argv)
{
  // Access arguments in the same way as above.
}

Because char *argv[] is read ‘declare argv as array of pointer to char’, and the array argument is demoted to a pointer to the first element in that array, char *argv[] is equivalent to char **argv - which is read as: ‘declare argv as pointer to pointer to char’

References

Introduction to Vectors in C++

Tue, 20 Feb 2018 09:26:00 +0000

C-style arrays in C and C++ are not dynamic - the compiler needs to know the size of the array (the type and number of elements) in order to properly allocate memory. This is quite different to interpreted languages like Python, PHP and JavaScript, where arrays are inherently dynamic - you don’t need to specify data type and size in advance.

Vectors in C++ are standard containers which provide powerful dynamic-array like functionality, with fixed-time access to elements in any order. Vectors are defined by a template class available in the standard library - you just need to add #include <vector> to your header file to have access to vector functionality.

A C++ vector can be described as a dynamic C-style array - with fast and efficient access to individual elements in any order. You don’t need to worry about memory and size allocation. Subscripting (i.e. myVector[]) access is provided as with C-style arrays.

Vectors work by setting up an initial memory allocation for your data. If more data is added and the allocation is exceeded, the vector class creates a new, bigger array in memory. The old array is copied across to the new, before being deleted. This is all abstracted away so you don’t need to worry about it, though you should probably take measures to minimise re-allocation - which will help maximise performance.

This article is an ongoing summary of my notes on functions in C++. Please don’t assume that what you read here is accurate. For a more complete description of vectors in C++, check the references presented below.

Declaration

Include the appropiate header to give access to the vector class. Then declare the vector - which shoudl specify the data type that will be contained by the vector:

#include <vector>

// Define a vector that will contain integers
std::vector<int> myIntsContainer;

// Define a vector that will contain doubles
std::vector<double> myDoublesContainer;

For convenience, you can use a typedef statement to make declarations more concise:

// Set up aliases
typedef std::vector<int> intVec;
typedef std::vector<double> doubleVec;

// Declarations
intVec myIntsContainer;
doubleVec myDoublesContainer;

Initialising

To initialise, you can use an initialiser list provided your compiler supports this. This method works with g++ version 5.4.0 on Ununtu, compiling with the c++11 flag.

// Method 1
std::vector<int> myVec {1,3,5};

// Method 2 (equivalent to method 1)
std::vector<int> myAltVec = {2,4,6};

You can also specify the initial size during the declaration, and only set values later:

// Specify size in the vector constructor.
// This works with int data type, but will cause problems with aggregate data types.
// See the std::vector::reserve() method
std::vector<int> myVec(3)
myVec = {7,8,9};

// You can still dynamically add elements, even though the size was specified during initialisation
myVec.push_back(10); // myVec is now {7,8,9,10}

...

The reserve() method is a better option for specifying an original size rather than using the vector constructor, which will cause problems when the vector holds aggregate data:

#include <iostream>
#include <vector>

int main(int argc, char const *argv[])
{
    struct KeyVal {
        std::string key;
        int val;
    };
    std::vector<KeyVal> options;

    // Set the capacity for this vector
    options.reserve(10);

    // Add initial data
    options = {
        {"Option 1", 1},
        {"Option 2", 2},
        {"Option 3", 3}
    };

    // Up to 7 more elements can be added with re-allocation
    options.push_back({"Option 4", 4});

    for(KeyVal& option : options) {
        std::cout << option.key << ": " << option.val << std::endl;
    }
    return 0;
}

Initialising With Values ———————— To initialise a vector with 5 elements, each element being the int 42:

std::vector<int> v(5, 42); // v is {42, 42, 42, 42, 42}

In this case, the first parameter is the number of elements, and the second parameter is the value to which they should be set.

Be careful to distinguish between this method and curly-brace initialisers:

std::vector<int> v{5, 42}; // v is {5, 42}

Curly brace initialisation treats each element in the intialisation list as an element, and constructs the vector accordingly.

Adding Elements

The vector.assign(n, el) method overwrites the vector, such that it contains n elements, each defined by el.

The vector.push_back(el) method appends an element to the vector.

To add an element to a position specified by an iterator use vector.insert().

Examples:

// Define an empty vector `doubleContainer` that will contain the double data type
std::vector<double> doubleContainer;

// Assign
// -----------------------------------------------------------------------------
// Populate the vector with 10 elements, each with a value of 3.14159
doubleContainer.assign(10, 3.14159);

// The original vector will be over-written by assign
doubleContainer.assign(3, 9.806); // {9.806, 9.806, 9.806}

// Append
// -----------------------------------------------------------------------------
doubleContainer.push_back(3.14159); // {9.806, 9.806, 9.806, 3.14159}

// Insert - requires an iterator
// -----------------------------------------------------------------------------
// Create an iterator for the vector
std::vector<double>::iterator it;

// Set a position for the iterator - in this case, after the first element
it = doubleContainer.begin() + 1;

// Insert a value
doubleContainer.insert(it, 5.601); // {9.806, 5.601, 9.806, 9.806, 3.14159}

...

Note that:

assign() overwrites the vector
insert() requires an iterator as first parameter, not a simple index
push_back() appends to the vector

Add Elements at a Precise Position

struct User {
      int id; std::string name;
};
std::vector<User> users;

// emplace() requires an iterator
std::vector<User>::iterator uIt = users.begin();
users.emplace(uIt, User{42, "David"});

// The iterator relating to this container is invalidated by `emplace()`.
// To call `emplace()` again, the iterator must be reset.
uIt = users.begin();
users.emplace(uIt + 1, User{444, "Elaine"});

for(User& u : users) {
    std::cout << "Name: " << u.name << ", ID: " << u.id << std::endl;
}

Note that if a reallocation is triggered by emplace() (i.e. the allocated memory block grows), all iterators, pointers and references for the container are invalidated. If no reallocation occurs, only those pointing to the position and beyond are invalidated. All iterators, pointers and references to elements before position keep referring to the same elements they were referring to before the emplace() call.

More Efficient Appending

The vector::emplace_back() function inserts a new element at the end of the vector.

The new element is constructed in place using the provided arguments. This avoids a copy action, improving efficiency as compared with push_back().

The allocated storage space will only be extended if the new vector size exceeds the current vector capacity.

When the vector element is made up of aggregate data (like a struct), the benefits of emplace_back() vs push_back() may be negated. In this case, you should use an initialiser list constructor in the data structure, which will allow the new element to be constructed in-place without a copy.

// `emplace_back()` with aggregate data
// -------------------------------------------------------------------------
struct Book {
    // Initialiser list constructor.
    Book(int id, std::string title, std::string author): id(id), title(title), author(author) {}
    int id;
    std::string title;
    std::string author;
};

// Initialise a vector of Books
std::vector<Book> books;

// Reserve memory - vector will hold up to 20 elements before re-allocation is necessary
books.reserve(20);

// When using `push_back()`, a temporary object of appropriate type is created and initialized
books.push_back({ 456, "A Scanner Darkly", "Philip K Dick", 15.99 });

// ...has the same performance as `push_back()`, because a temporary `Book` is constructed:
books.emplace_back(Book{ 777, "Guards Guards!", "Terry Pratchett", 9.99 });

// However, the following works if the Struct has an initialiser list constructor:
books.emplace_back(999, "Idoru", "William Gibson", 9.99);

Clear

The clear() method removes all elements:

// Populate the vector with 10 elements, each with a value of 3.14159
doubleContainer.assign(10, 3.14159); // `doubleContainer` now has 10 elements

doubleContainer.clear(); // `doubleContainer` now has 0 elements

...

Erase

Deletes a single element from a vector.

std::vector<int> intArray;

// Add 10 elements to the vector
for(int i = 0; i <= 10; i++) {
    intArray.push_back(i);
}

// intArray vector is now {0,1,2,3,4,5,6,7,8,9}

// Delete 5th element
intArray.erase(intArray.begin() + 4);

// intAarray vector is now {0,1,2,3,5,6,7,8,9}
...

Erase Based on a Predicate

You can combine erase with std::remove_if() from the stl <algorithm> header to remove elements in the collection based on a predicate:

#include <iostream>
#include <vector>
#include <algorithm>

int main()
{
	std::vector<int> myVec{1, 2, 3, 4, 5, 6, 7, 8, 9, 10};

	// Predicate is a lambda that returns 0 if the element is even, 1 otherwise.
	// IN this case, `remove-if` removes odd elements.
	myVec.erase(std::remove_if(myVec.begin(), myVec.end(), [](auto x){return x % 2;}),
				myVec.end());
	for (auto& el : myVec) {
		std::cout << el << (el == myVec.back() ? "" : " ") << std::endl;
	}
	return 0;
}

Optimisation

When the memory allocated for your vector is exceeded as the vector grows, a re-allocation is triggered. Re-allocation is resource-expensive and should be avoided if possible.

One simple way of avoiding unecessary reallocation is to set a rational capacity for the vector with vector.reserve(). If you do this, re-allocation won’t occur until you exceed the defined reserve.

The method vector.reserve(n) sets the vector capacity such that n elements can be contained. If n is greater than the current vector capacity the container storage will be reallocated, increasing its capacity to n (or greater). vector.reserve() has no effect on vector size and does not alter the vector elements.

If n is less than the current capacity the function call does not cause a reallocation and the vector capacity is not affected.

References

Beginner Tutorial on Vectors - despite the name, this is a really comprehensive article
Great intro video to vectors by TheChernoProject
Advanced vectors video by TheChernoProject - optimising useage of the vector class
std::vector::push_back()
std::vector::emplace_back()
std::vector::reserve())
std::vector
push_back() vs emplace_back()

Generic Programming - Simple C++ Templates

Wed, 07 Feb 2018 11:05:00 +0000

C++ is a strongly typed language - all variables must have a specific type, either explicitly defined by the programmer or inferred during compilation. However, function operations are often identical regardless of the data type being operated on. For example, the body of a function that sums two numbers will be identical for both int and float types.

C++ allows generic functions to be created using templates. This means that the function can work on many different data types without being rewritten for each one. The template generates an ordinary type or function when it is compiled, based on the arguments supplied to the template.

Simple Template

Simple template definition where all input parameters and return variables are of the same type:

// Define the template
template <typename T>
T genericSumTwo (T a, T b)
{
    return a + b;
}

int intA = 2;
int intB = 2;
float floatA = 2.22;
float floatB = 2.44;

// Use the template with `int` type
std::cout << genericSumTwo(intA, intB) << std::endl;
/* Outputs 4 */

// Use the template with `float` type
std::cout << genericSumTwo(floatA, floatB) << std::endl;
/* Outputs: 4.66 */

Template with Different Data Types Input

Templates can accept different data types - with the return data type specified in the template definition:

// Template accepts multiple variable types
template <typename T, typename U>
T getBigger(T input1, U input2)
{
    if (input1 > input2) {
        return input1;
    }
    return input2;
}

// Usage
float A = 2.22;
int B = 1;
std::cout << getBigger(A, B) << std::endl;
/* Returns result as a float: 2.22 */

This definition allows different variable types as input parameters, and specifies that the return type will be the same as the type of the first supplied parameter (in this case, T).

Specify Values - Non-type Parameters

You can also specify values as part of the template specification:


template <typename T, int N>
class Array
{
private:
	T m_Array[N];
	int m_size = 0;
public:
}

In this case, the class has a member m_Array which is initialised with N elements of type T.

The template is invoked like this:

// The object will have a member m_Array, an array of ints with size 10.
Array intArray<int, 10>

References

Define User Input From an Integer List in C++

Sun, 04 Feb 2018 11:37:00 +0000

There are many ways to collect user input. In this example a user is prompted to make a selection from a numbered list. This sets the int choice variable, which can then be used to control programme flow.

The possible values for user selection, along with a related string label for each value are set up using a struct.

Input is handled by std::getline(), with the input converted to an integer by means of the std::stringstream operator >>. I’ve done it this way rather than using std::cin so that an empty return can be handled and interpreted as the default value, if this has been set in the selectAction() function call.

Example Code

Compile this using c++11 standards. for example: g++ -o test -std=c++11 test.cpp

#include <iostream>
#include <algorithm> // Needed for any_of()
#include <string>
#include <map>
#include <sstream>

int selectAction(int defaultChoice);

int main(int argc, char const *argv[])
{
    int userChoice = 0;
    std::cout << "This program prompts a user for input." << std::endl;
    // Pass in a default value 1
    userChoice = selectAction(1);
    std::cout << "Choice made: " << userChoice << std::endl;
    return 0;
}

int selectAction(int defaultChoice)
{
    // Create a data structure `optionPair` so that values may be associated with
    // labels, and initialize an array of `optionPair` elements.
    struct optionPair {
        std::string label;
        int value;
    } options[] = {
        {"Label for Option 1", 1},
        {"Label for Option 2", 2},
        {"Label for Option 3", 3}
    };
    // Create an array of valid input values
    const int nOptions = sizeof(options)/sizeof(optionPair);
    int validChoices[nOptions];
    for (int i = 0; i < nOptions; i++) {
        validChoices[i] = options[i].value;
    }
    int choice;
    bool validChoice;
    while (true) {
        int choice = 0;
        std::string input = "";
        std::cout << "Please select an option:" << std::endl;
        for (auto&& option : options)
        {
            std::cout << option.value << ") " << option.label << std::endl;
        }
        std::getline(std::cin, input);
        std::stringstream tempStream(input);
        // Is it empty? User just hit return
        if (tempStream.str().empty()) {
            choice = -1;
        } else {
            // The user entered something, so convert it to an integer. If the
            // entry is not a number, it will be transformed to 0.
            tempStream >> choice;
        }

        // `[&]` specifies that the lambda uses variable capture - needed here
        // since the lambda accesses the `choice` variable.
        validChoice = std::any_of(std::begin(validChoices), std::end(validChoices), [&](const int& i)
        {
            return i == choice;
        });

        if (defaultChoice && choice == -1) {
            // User entered return & a default exists - return the default
            return defaultChoice;
        } else if (validChoice) {
            // User selection returned if valid
            return choice;
        }

        // User feedback for invalid entry
        std::cout << "You've entered an invalid selection." << std::endl
        << "Please enter a number from the specified list. Thanks." << std::endl;
    }
}

I’m sure there are better ways to allow users to choose from a list of integer values - please leave a comment if you have a suggestion.

References

Beginner C++ Functions

Sat, 13 Jan 2018 11:10:00 +0000

This article is an ongoing summary of my notes on functions in C++. Please don’t assume that what you read here is accurate. For a more complete description of functions in C++, check the references presented below.

Functions in C++ provide a way to group statements. Functions consist of:

A name
A list of parameters (which may be zero)
A body - which consists of statements enclosed with curly braces

In contrast to scripting languages like JavaScript and PHP, functions in C++ must be declared if they are to be used before their definition.

Every C++ programme has at least one function called main().

Function Declaration in C++

A function declaration specifies the function name and the type of data that it returns and accepts. The declaration is terminated with a semi-colon.

For example:

return_type functionName(parameter1_type parameter1, parameter2_type parameter2);
// or simply:
return_type functionName(parameter1_type, parameter2_type);

Specific example of function declaration:

// Declare a function that sums two float inputs
float sumInputs(float input1, float input2);

// Parameter names are not needed for function declaration, so this is also valid:
float sumInputs(float, float);

Function Definition in C++

The function definition specifies the actual body of the function - the actual grouped statements. The general form of a C++ function definition:

return_type functionName( parameters )
{
  // Body of the function: statements
  // Function terminates either by returning or throwing an exception
}

This example doubles an input integer:

int doubleInput(int input)
{
  return input * 2;
}

Declaration Rules

If a function is defined after it is used, it must be declared in advance. If the function is defined before it is used, the function declaration is optional. However, it is considered good practice to list function declarations at the top of the header file. If the function is defined in a header file which is included in the main programme, the function declaration is not strictly necessary since the header file is included before the function is called.

Best Practices

If a variable passed as a function parameter will not be modified in the function, it is considered best practice to declare the variable as const so that it can’t be changed:

References

Securely Erase A Drive from the Linux Command Line

Tue, 12 Dec 2017 09:39:00 +0000

If a disk contains secure information, it may need to be securely erased.

This article outlines a simple disk-wipe procedure for the Linux command line. Tested on Ubuntu 16.04.

Determine the Target Drive

To find the drive, run the lsblk command. This will output drive name and mount point:

lsblk
# Typical output:
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                             8:0    0 465.8G  0 disk
├─sda1                                          8:1    0   512M  0 part  /boot/efi
├─sda2                                          8:2    0   488M  0 part  /boot
└─sda3                                          8:3    0 464.8G  0 part
  └─sda3_crypt                                252:0    0 464.8G  0 crypt
    ├─ubuntu--vg-root                         252:1    0 448.8G  0 lvm   /
    └─ubuntu--vg-swap_1                       252:2    0    16G  0 lvm
      └─cryptswap1                            252:3    0    16G  0 crypt [SWAP]
sdb                                             8:16   0   1.8T  0 disk
└─sdb1                                          8:17   0   1.8T  0 part  /media/datadrive
sdc                                             8:32   1   7.5G  0 disk
└─sdc1                                          8:33   1   7.5G  0 part
  └─luks-04235321-8ad9-4631-934c-2c09cfa700e7 252:4    0   7.5G  0 crypt /media/david/secure-data
sdd                                             8:48   1   7.5G  0 disk
└─sdd1                                          8:49   1   7.5G  0 part  /media/david/Thumbdrive
loop0                                           7:0    0  80.5M  1 loop  /snap/core/2462
loop1                                           7:1    0  80.5M  1 loop  /snap/core/2381
loop2                                           7:2    0  79.5M  1 loop  /snap/core/2312
loop3                                           7:3    0 182.6M  1 loop  /snap/atom/9
loop4                                           7:4    0 182.6M  1 loop  /snap/atom/8

Determine the relevant disk name from this list.

Double Check: Avoid Foot-Shooting

When you run the dd command in the “Wipe the Disk” section of this article, the target disk will be completely overwritten.

You should therefore double check that you’re operating on the right drive:

# Replace sdX with your target drive name
cat /sys/class/block/sdX/device/{model,vendor}

The output should correspond to the target disk you’re expecting to wipe.

Wipe the Disk

This is achieved by writing random data from /dev/urandom to the target disk.

Block size is set to 1M for the sake of increasing speed - dd will read and write up to 1M bytes at a time.

Setting the status option to “progress” prints periodic transfer stats to stderr.

# Replace sdX with your target drive name
dd if=/dev/urandom of=/dev/sdX bs=1M status=progress

The problem with this method is that dd just writes indefinitely - until eventually it times out. It works, but it is more time consuming than it needs to be.

Better Method

Use parameters with dd to wipe a partition/drive:

sudo fdisk -l /dev/sdX

# Output:
GPT PMBR size mismatch (15702015 != 15826943) will be corrected by w(rite).
Disk /dev/sdX: 7.6 GiB, 8103395328 bytes, 15826944 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C70EC09A-1A70-4728-9D0C-B4122C401FFA

Device       Start      End  Sectors  Size Type
/dev/sdX1     2048  5122047  5120000  2.5G EFI System
/dev/sdX2  5126144 15701982 10575839    5G Linux filesystem

To wipe whole drive:

Start=2048
End=15826944 # From line 2 of fdisk output
BytesInSector=512 # From line 3 of fdisk output
dd if=/dev/urandom of=/dev/sdX bs=${BytesInSector} count=${End} seek=${Start} status=progress

Resources

Manage and Safely Delete Revisions in WordPress

Tue, 14 Nov 2017 20:00:00 +0000

In my opinion, Revisions in WordPress are a waste of time. I don’t know anyone that actually uses them - and worse, they can have a serious impact on site performance, especially in the admin area.

Revision Management Deleting: There is an API function to delete revisions, but there is no UI. That can certainly change. WordPress Codex, Revisions

Personally, I find this ridiculous - it means that for the average user every time a post is updated the database gets a bit more bloated - and there is no UI to remove this bloat.

When to be Concerned

You may hit a 500 server error when trying to access an edit page for content that has a lot of associated revisions.

Editing a reasonably sized page with hundreds of revisions can cause problems, even with 256MB server memory available. This trac ticket outlines the problem nicely. I had to deal with this exact scenario today, which prompted this post.

Enabling error logging may give you a message like this:

[14-Mar-2018 10:43:37 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/example.com/web/wp/wp-includes/meta.php on line 840

…you’ll find a similar message in your Apache logs.

Increase Available Memory

This is a band-aid solution, but increasing the available memory can provide an instant fix, and buy some time. Under Ubuntu, edit /etc/php/7.0/apache2/php.ini:

; Maximum amount of memory a script may consume (was 128MB, now 256M)
; http://php.net/memory-limit
; Bump up the memory, within reasonable bounds
memory_limit = 256M

IMPORTANT: Restart Apache for the changes to take effect. You could also add a directive to a .htaccess file if you’re using one, or add a directive in an Apache virtual host configuration.

Now that the client has stopped panicking, you can fix the real problem: Revisions.

TLDR; Delete Revisions

You need MySQL access to delete revisions.

Get a MySQL command prompt, using your target database:

mysql -u root -p my_database

Unfortunately, it’s not quite as simple as just deleting posts from wp_posts table having a post_type of ‘revision’ - you should also remove associated postmeta and wp_term_relationships records.

You can achieve this with the following MySQL/MariaDB commands:

DELETE a,b,c FROM wp_posts a
LEFT JOIN wp_term_relationships b ON (a.ID = b.object_id)
LEFT JOIN wp_postmeta c ON (a.ID = c.post_id)
WHERE a.post_type = 'revision';

This command deletes records from tables a, b and c - defined as wp_posts, wp_term_relationships and wp_postmeta. The rows deleted are determined by these criteria:

Rows in wp_posts table which have a post_type of ‘revision’
Rows in wp_term_relationships which have an object_id equal to the ID of a wp_posts ‘revision’ row.
Rows in wp_postmeta which have a post_id equal to the ID of a wp_posts ‘revision’ row.

For a custom agency build with a lot of custom fields, number 3 is likely to be a big one. For each revision saved, WordPress will add a new postmeta record for each custom field.

Prevent Revisions

Limit revisions to a rational number, or prevent them altogether - add this to wp-config.php:

define('WP_POST_REVISIONS', 10); // Older revisions will be automatically deleted
define('WP_POST_REVISIONS', FALSE); // No revisions at all

Resources

Symlinking to Directories in Linux

Wed, 18 Oct 2017 20:30:00 +0100

If you need to symlink a directory in Linux, you need to make a distinction. The linked-to directory is intended to:

Be the symlink: navigating to the symlink is equivalent to navigating to the linked directory
Be accessed via a link within the linking directory

If you intend the former, you need to be careful that the directory does not already exist - if it does, the symlink will be placed inside the existing directory.

Example: The Directory is the Symlink

In this case, the ‘directory’ path/to/current will contain the files from /path/to/target. Crucially, /path/to/current does not exist.

# /path/to/current does not exist
ln -s /path/to/target path/to/current

# cd to the parent directory and list contents...
cd path/to && ls -la
# Output:
drwxrwxr-x 2 david david 4096 Oct 18 21:01 ./
drwxrwxr-x 8 david david 4096 Oct 18 21:01 ../
lrwxrwxrwx 1 david david    4 Oct 18 21:01 current -> /path/to/target/

In this case, to access files in /path/to/target via the symlink you would reference /path/to/current:

cd path/to/current && ls -la
# Output:
drwxrwxr-x 2 david david 4096 Oct 18 21:03 ./
drwxrwxr-x 7 david david 4096 Oct 18 21:09 ../
-rw-rw-r-- 1 david david    0 Oct 18 21:02 file-1
-rw-rw-r-- 1 david david    0 Oct 18 21:02 file-2
# Where file-1 and file-2 are contained in /path/to/target

Example: Directory Contains the Symlink

In this case, the /path/to/current directory does exist:

# /path/to/current does exist
ln -s /path/to/target path/to/current

cd path/to/current && ls
# Output:
drwxrwxr-x 2 david david 4096 Oct 18 21:01 ./
drwxrwxr-x 8 david david 4096 Oct 18 21:01 ../
lrwxrwxrwx 1 david david    4 Oct 18 21:01 target -> /path/to/target

In this case, to access files in /path/to/target via the symlink you would reference /path/to/current/target.

Rsync –link-dest

If you’re using the --link-dest option with rsync to make incremental backups, make sure you are referencing the correct symlinked reference directory - if not, rsync won’t create hardlinks and your backups will not be incremental. I recently made this error when mis-referencing the most recent backup by dynamically adding a symlink to an already existing directory.

Resources

https://stackoverflow.com/a/9104390/3590673

Access $USER Environment Variable in a Cron Triggered Script in Ubuntu

Wed, 18 Oct 2017 19:52:00 +0100

When triggering a BASH script in a Cronjob in Ubuntu the script will not have access to the $USER environment variable.

This is the case even if the script is trigerred by the user’s crontab. Fortunately this is easily fixed.

Within the crontab, $LOGNAME represents the current user - so you can set the $USER variable before calling your script:

# Open the crontab for your user:
crontab -e

# Sample cron job:
# m h  dom mon dow   command
05 17 * * * env USER=$LOGNAME /usr/local/bin/backup-home

…you can now use $USER in the script as usual.

Background

Several environment variables are set automatically by cron:

SHELL: is set to /bin/sh
LOGNAME: is set from the line in /etc/passwd corresponding to the crontab’s owner
HOME: is set from the line in /etc/passwd corresponding to the crontab’s owner

Note that HOME and SHELL can be overridden in the crontab but LOGNAME cannot.

References

Crontab man page

Encrypted Backup Drive in Ubuntu Xenial 16.04

Sun, 15 Oct 2017 11:37:00 +0100

Context

This article refers to creating a LUKS encrypted backup drive for Ubuntu 16.04 Xenial Xerus - but the steps are likely very similar for any modern Linux distro.

The exisiting main boot drive is encrypted. The main user’s home drive is also encrypted. The aim is to have a partition on an additional (internal) hard drive which is LUKS encrypted and automatically unlocked on boot. If the disk is removed, it should require a keyfile for decryption. The purpose of this is to locally backup up sensitive data automatically.

This will ensure that data remains secure - it will be encrypted at rest, and only accessible after booting the OS on the encrypted boot drive - data will be protected by the keyfile, which in turn is inaccessible until the main OS boots, which in turn recquires the encryption passphrase.

Encrypt the Partition

Encrypt the partition using the Gnome Disk utility:

Select and unmount the partion
Format partition - select “Encrypted, compatible with Linux systems(LUKS _ Ext4)” in the “Type” dropdown
Enter a passphrase when prompted save this passphrase: it can be used to unlock the partition in the event of disaster recovery

Whilst in the Disks utility, you can also mount the partition.

Note the device name for the partition (e.g. /dev/sdb*).

Create a Keyfile in the Root User Home Directory

Make a keyfile in the root user home directory:

sudo mkdir /root/.keyfiles
sudo dd if=/dev/urandom of=/root/.keyfiles/hdd-1.key bs=1024 count=4
# Make this read-only by owner (in this case, root):
sudo chmod 0400 /root/.keyfiles/hdd-1.key

Add Keyfile to LUKS

LUKS/dm_crypt enabled devices may hold up to 10 different keyfiles/passwords.

In addition to having the already setup password, we’re going to add this keyfile as additional authorization method:

# Replace 'sdX' with the correct device designation
sudo cryptsetup luksAddKey /dev/sdb2 /root/.keyfiles/hdd-1.key

Determine the UUID of the Partition

ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Oct 15 10:23 00d75d72-84b0-418f-7dda-2d42002fe045 -> ../../sdX
...

Create a Mapper

Create a mapper in crypttab that references the keyfile:

# /etc/crypttab
# Append this line - add the relevant UUID - first value is the identifier for this mapping
sdX_crypt UUID=00d75d72-84b0-418f-7dda-2d42002fe045 /root/.keyfiles/hdd-1.key luks,discard

Mount on Boot

To mount the partition on boot:

Create a mount point
Add a command to /etc/fstab

# Create a mount point:
sudo mkdir /media/secure-hdd

# Edit /etc/fstab:
sudo nano /etc/fstab

# /etc/fstab
# Append this line:
/dev/mapper/sdX_crypt /media/secure-hdd ext4 defaults 0 2

Reboot to check it works.

Open without Keyfile

Install cryptsetup
Decrypt the volume using the original passphrase
Mount

Install cryptsetup:

sudo apt install cryptsetup

Decrypt the volume:

sudo cryptsetup luksOpen /dev/sdX sdX_crypt
# Enter passphrase when prompted

Mount:

sudo mkdir /media/secure-hdd-recovered
sudo mount /dev/mapper/sdX_crypt /media/secure-hdd-recovered

Resources

LetEncrypt Certbot on Ubuntu Xenial Xerus

Sat, 23 Sep 2017 11:57:00 +0100

LetsEncrypt is a free, open and automated certificate authority that operates for public benefit. It is a project of the non-profit Internet Security Research Group (ISRG).

This guide refers to installing and configuring LetsEncrypt and it’s client, Certbot, on Ubuntu 16.04 Xenial Xerus.

Certbot: the LetsEncrypt Client

Certbot is a client that allows you to fetch and configure SSL/TLS certificates. It also updates virtual host directives to ensure that site resources redirect to HTTPS.

Certbot is an easy-to-use automatic client that fetches and deploys SSL/TLS certificates for your webserver. Certbot was developed by EFF and others as a client for Let’s Encrypt and was previously known as “the official Let’s Encrypt client” or “the Let’s Encrypt Python client.” Certbot will also work with any other CAs that support the ACME protocol.

– https://certbot.eff.org/about/

Install Certbot

See up-to date instructions here.

sudo apt-get update
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-apache

Set Up Certs - Specific Domains

If you have pre-existing certs, this may be a good solution:

sudo certbot --apache -d example.com -d www.example.com

This command sets up certificates and creates appropriate virtual host directives that reference the certificates.

Set Up Certs - Interactive Session

You could run sudo certbot --apache instead - this will open an interactive session that will prompt you for various options. If upgrading from a previous version of the LetsEncrypt client, this is probably the best option:

sudo certbot --apache

# This outputs:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org

Which names would you like to activate HTTPS for?
-------------------------------------------------------------------------------
1: example.com
2: www.example.com
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel):

The new vhost directives will be created in the /etc/apache2/sites-available directory in the format example.com-le-ssl.conf. The vhost configs will be automatically enabled, and a redirect will be written into the corresponding directive for port 80 (the non-HTTP original vhost directive).

Renew Certs

sudo certbot renew

Check Renew

Test that the renew process will work by performing a dry-run:

sudo certbot renew --dry-run

Automatic Renewal: Cronjob

Certificates have a three-month lifespan, so automatic renewal is recommended.

LetsEncrypt/Certbot sets up a cronjob. This is located at /etc/cron.d/certbot. It looks like this:

# /etc/cron.d/certbot: crontab entries for the certbot package
#
# Upstream recommends attempting renewal twice a day
#
# Eventually, this will be an opportunity to validate certificates
# haven't been revoked, etc.  Renewal will only occur if expiration
# is within 30 days.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew

The script first checks that /usr/bin/certbot exists and is executable, and that systemd is NOT present before running renew on a random minute of the hour.

The cronjob:

Runs every twelve hours
Tests that /usr/bin/certbot exists and is executable (i.e. certbot is installed)
ALSO (-a) tests that the directory /run/systemd/system does NOT exist (i.e. systemd is not present on the system)
If the previous two conditions are satisfied, pauses for a random number of seconds < 3600
When the sleep period has elapsed, certbot renew runs in quiet mode

Ubuntu 16.04? Your Cronjob Does Nothing!

The test command in the cronjob under /etc/cron.d/certbot stops execution if systemd is present - which in the case of Ubuntu, it is.

In this case, the timing of renewals is controlled in /lib/systemd/system/certbot.timer. Note that the execution of this is controlled in /etc/systemd/system/timers.target.wants, which contains a symlink to /lib/systemd/system/certbot.timer.

The /lib/systemd/system/certbot.timer file looks like this:

[Unit]
Description=Run certbot twice daily

[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=3600
Persistent=true

[Install]
WantedBy=timers.target

This timer runs the service /lib/systemd/system/certbot.service, which looks like this:

[Unit]
Description=Certbot
Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
Documentation=https://letsencrypt.readthedocs.io/en/latest/
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true

Note that by default, a service is activated with the same name (excluding suffix) as the timer - so certbot-timer activates certbot-service.

Support LetsEncrypt

You can support LetsEncrypt and Certbot by donating here.

References

Iterating Over Dictionary in Python

Thu, 21 Sep 2017 21:08:00 +0100

A dictionary in Python is a collection of unordered values accessed by key rather than index. Dictionary elements have no intrinsic order.

This short article shows several methods for iterating over Python dictionaries.

The examples in this article are Python 3.

Example dictionary:

transaction = {
  "amount": 10.00,
  "payee": "Joe Bloggs",
  "account": 1234
}

Iterate via keys

for key in transaction:
    print("{}: {}".format(key, transaction[key]))

# Output:
account: 1234
payee: Joe Bloggs
amount: 10.0

Iterate values only

for value in transaction.values():
    print(value)

# Output:
1234
Joe Bloggs
10.0

Iterate over key value pairs

for key, value in transaction.items():
    print("{}: {}".format(key, value))

# Output:
account: 1234
payee: Joe Bloggs
amount: 10.0

I often confuse this with the PHP foreach construct - forgetting to chain the items() method to the dictionary being iterated over.

Iterate over keys in the sorted order of the keys

for key in sorted(transaction):
    print("{}: {}".format(key, transaction[key]))

# Output:
account: 1234
amount: 10.0
payee: Joe Bloggs

Reference

Operations on Python Dictionaries

Upgrading Monero Command Line Interface

Tue, 19 Sep 2017 19:29:00 +0100

Notes on upgrading Monero command line interface on Ubuntu 16.04 From Wolfram-Warptangent to Helium-Hydra.

Download New Binaries

Download the binaries, along with the list of hashes that will allow you to verify the download:

# Make a suitable directory and move into it
mkdir ~/monero/helium-hydra && cd $_

# Download the Monero Linux 64 bit CLI client
wget https://downloads.getmonero.org/cli/monero-linux-x64-v0.11.0.0.tar.bz2

# Download the GPG-signed canonical list of hashes
wget https://getmonero.org/downloads/hashes.txt

Verify the Download

To verify the download, you need to:

Verify that you have a true copy of hashes.txt
Check that the sha256sum checksum for your downloaded tar file matches the quoted value in hashes.txt

To verify hashes.txt, you must install the appropriate GPG key, which is found here in the Monero source code repository (/utils/gpg_keys).

The hashes.txt file will state which signature has been used - in this case, we need to add the GPG signature for fluffypony to GPG.

# Download the raw public key:
wget https://raw.githubusercontent.com/monero-project/monero/master/utils/gpg_keys/fluffypony.asc

# Import this public key to GnuPG:
gpg --import fluffypony.asc

You can now use GnuPG to verify hashes.txt. From within the same directory as the file, run:

gpg --verify hashes.txt

If the hashes.txt file is genuine, you should see:

# email has been obfuscated - you should see the correct email in your output
gpg: Signature made Tue 12 Sep 2017 21:37:04 IST using RSA key ID 1CCD4FCD
gpg: Good signature from "Riccardo Spagni <***@******.net>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: BDA6 BD70 42B7 21C4 67A9  759D 7455 C5E3 C0CD CEB9
     Subkey fingerprint: 94B7 38DD 3501 32F5 ACBE  EA1D 5543 2DF3 1CCD 4FCD

You can now use hashes.txt to verify the Monero download:

sha256sum -c hashes.txt 2>&1 | grep OK

# Successful verification will look like this:
monero-linux-x64-v0.11.0.0.tar.bz2: OK

Extract

Extract the files from the downloaded tarball:

cd ~/monero/helium-hydra
tar -xjvf monero-linux-x64-v0.11.0.0.tar.bz2

Move/copy Wallet files from old to new directory:

# Replace 'MyWallet' with your wallet name - note lack of spaces after commas
cp ~/monero/wolfram-warptangent/monero-v0.10.3.1/{MyWallet,MyWallet.address.txt,MyWallet.keys} ~/monero/helium-hydra/monero-v0.11.0

Run Monero

When you move into the new Monero directory and run ./monerod, it should start syncing to the forked blockchain - it will not need to download the Monero blockchain from scratch.

In my case, I had already synced after the hard-fork with the previous CLI client, and this caused a problem. To get around this, remove some downloaded blocks by running monero-blockchain-import:

# Get rid of the most recent 1000 blocks - it may be necessary to remove more
./monero-blockchain-import --pop-blocks 1000

Connect Up and Run Monero

A convenient way to manage Monero version upgrades is to create symlinks to Monero executables within a directory that is in your $PATH.

For example, after you download and verify the ‘helium-hydra’ Monero binaries, you might symlink to relevant files from within your /usr/local/bin directory:

sudo ln -s ~/monero/helium-hydra/monero-v0.11.0.0/monerod monerod
sudo ln -s ~/monero/helium-hydra/monero-v0.11.0.0/monero-wallet-cli monero-wallet-cli

The /usr/local/bin directory now contains the following:

lrwxrwxrwx  1 root  root       56 Sep 19 10:35 monerod -> /home/david/monero/helium-hydra/monero-v0.11.0.0/monerod*
lrwxrwxrwx  1 root  root       66 Sep 19 10:36 monero-wallet-cli -> /home/david/monero/helium-hydra/monero-v0.11.0.0/monero-wallet-cli*

To run monerod from the helium-hydra package, just run monerod in your terminal. Likewise, monero-wallet-cli will run the correct wallet CLI binary.

Attach Click Event to Many Elements in Javascript

Tue, 05 Sep 2017 10:00:00 +0100

Plain JavaScript: loop through elements by class name and attach a click event using Array.forEach.

The getElementsByClassName() method returns a HTMLCollection object. This needs to be converted to an array before iterating with Array.forEach():

const view = {
  display: function() {
    const address = controller.getAddress()
    let displayAddress = `
    <table>
    <tr><td>Address:</td><td class="selector" title="Double click to select">${address.address}</td></tr>
    <tr><td>Secret: </td><td class="selector" title="Double click to select">${address.secret}</td></tr>
    <tr><td>Public Key: </td><td class="form-selector">${address.publicKey}</td></tr>
    <tr><td>Private Key: </td><td>${address.privateKey}</td></tr>
    </table>
    `
    this.displayAddress.innerHTML = displayAddress
    // The forEach method requires an array, rather than a HTML collection
    const selectors = [].slice.call(document.getElementsByClassName('selector'))
    selectors.forEach((element, index) => {
      element.addEventListener('click', (event) => {
        // Send the text content of the clicked element for processing
        controller.copyToClipboard(event.target.textContent)
      })
    })
  }
}
// Prompt user to copy text to clipboard
const controller = {
  copyToClipboard: (text) => {
    window.prompt("Copy to clipboard: Ctrl+C, Enter", text);
  }
}

Verify and Setup Litecoin Core

Fri, 01 Sep 2017 19:33:00 +0100

This guide refers to downloading and verifying Litecoin Core in Ubuntu 16.04. The guidelines should be pretty much the same for any Debian-based Linux distro.

Download the Litecoin Core Client & Signatures Document

Download the relevant Litecoin Core .tar file and the GPG signatures document.

To keep things organised, create a dedicated subdirectory and move into it:

mkdir ~/Downloads/litecoin && cd $_

# Download the provided tar - at the time of writing this is:
# litecoin-0.14.2-x86_64-linux-gnu.tar.gz
curl https://download.litecoin.org/litecoin-0.14.2/linux/litecoin-0.14.2-x86_64-linux-gnu.tar.gz

# Download the Litecoin hash signatures document to the same directory
curl https://download.litecoin.org/litecoin-0.14.2/linux/litecoin-0.14.2-linux-signatures.asc

Verify the Signatures Document

Download a copy of the Litecoin Core PGP key, as shown on the main website:

curl "https://pgp.mit.edu/pks/lookup?op=get&search=0xFE3348877809386C" | sed -n '1,/<pre>/d;/<\/pre>/q;p' > litecoin.pub.key
# Parsing HTML like this is bad...you should just curl the document and manually
# copy the key from between the <pre> tags, or hit the URL in your browser and copy the key.
# I used sed out of interest.

Now you can import the Litecoin public key:

gpg --import litecoin.pub.key

# Check that it installed correctly:
gpg --list-keys

Once the key is installed, you can verify the document that contains the hash signatures:

gpg --verify litecoin-0.14.2-linux-signatures.asc

Checksum

You can then safely use the signatures document to verify that the checksum of the download matches the expected checksum provided by the project:

sha256sum -c litecoin-0.14.2-linux-signatures.asc 2>&1 | grep OK

# Or:
grep litecoin-0.14.2-x86_64-linux-gnu.tar.gz litecoin-0.14.2-linux-signatures.asc | sha256sum --check

# In either case, a successful return will be:
litecoin-0.14.2-x86_64-linux-gnu.tar.gz: OK

# ... this means that the SHA256 hashes match.

Glitch in Signatures Document

The Signatures document contains Windows line endings, which causes the sha256sum check to fail (on Ubuntu, likely also on Mac):

sha256sum -c litecoin-0.14.2-linux-signatures.asc 2>&1 | grep OK
# returns blank

grep litecoin-0.14.2-x86_64-linux-gnu.tar.gz litecoin-0.14.2-linux-signatures.asc | sha256sum --check
# returns:
sha256sum: 'litecoin-0.14.2-x86_64-linux-gnu.tar.gz'$'\r': No such file or directory
: FAILED open or read4-linux-gnu.tar.gz
sha256sum: WARNING: 1 listed file could not be read

Fortunately this is easily remedied. Open the signatures file in Gedit, and ‘Save As’ with the same filename. You’ll be given an option for ‘Line Ending’, for which you should choose Unix/Linux.

Resources

Bitcoin Cold Wallet Setup Bash Script

Tue, 22 Aug 2017 20:07:00 +0100

Update - the script in this article will not work with Bitcoin Core v0.15.0 and higher. Check out my airgap utilities repo on GitHub instead.

If you want to examine a cold wallet in an offline Tails session, you’ll need to reference the Bitcoin binaries in your persistent volume. You’ll also need to move the required wallet into the Bitcoin data directory - which won’t exist until you run bitcoin-qt for the first time in the session.

This gets tedious, so I wrote this script to connect everything up.

#!/bin/bash
# Open a Bitcoin Core cold wallet in an offline Tails session.
# This script runs a simple zenity GUI which prompts the user to select:
# - The cold wallet file
# - The `bitcoin-qt` binary
# Both of these files will probably be on your persistent volume.
# ------------------------------------------------------------------------------
function set_cold_wallet {
  WALLET_FILE=$(zenity --file-selection --title="Select a Cold Wallet to run." --filename=${PWD}/)
  case $? in
    0)
    echo "\"$WALLET_FILE\" selected.";;
    1)
    echo "No file selected.";;
    -1)
    echo "An unexpected error has occurred.";;
  esac
}

function set_bitcoin_binary {
  BITCOIN_CORE=$(zenity --file-selection --title="Select the bitcoin-qt binary." --filename=${PWD}/)
  case $? in
    0)
    echo "\"$BITCOIN_CORE\" selected.";;
    1)
    echo "No file selected.";;
    -1)
    echo "An unexpected error has occurred.";;
  esac
}

function create_symlink {
  BITCOIN_DIR=~/.bitcoin
  if [[ ! -d "${BITCOIN_DIR}" ]]; then
    echo "Creating ${BITCOIN_DIR}"
    echo ""
    mkdir ${BITCOIN_DIR}
  fi

  if [[ -L "${BITCOIN_DIR}"/cold-wallet.dat ]]; then
    echo "Remove exisiting symlink, ${BITCOIN_DIR}/cold-wallet.dat"
    rm ${BITCOIN_DIR}/cold-wallet.dat
  fi

  echo "Make a symlink ${BITCOIN_DIR}/cold-wallet.dat pointing to ${WALLET_FILE}"
  ln -s ${WALLET_FILE} ${BITCOIN_DIR}/cold-wallet.dat
}

function run_wallet {
  echo "Starting Bitcoin Core with wallet ${WALLET_FILE}"
  ${BITCOIN_CORE} -wallet=cold-wallet.dat
}

# Run this script
# ------------------------------------------------------------------------------
echo "ONLY RUN IN A COLD ENVIRONMENT."
read -p "Do you wish to proceed? [y/N]" PROCEED
case $PROCEED in
  [Yy]* )
  set_cold_wallet
  set_bitcoin_binary
  create_symlink
  run_wallet
  ;;
  [Nn]* )
  echo "END"
  ;;
  * ) echo "Please answer yes or no.";;
esac

Update

As of v0.15.0, the Bitcoin Core client does not allow symlinked wallets. As such, the above script no longer works.

You would need to adjust the script so that it copies your specified cold wallet into the default Bitcoin data directory: (~/.bitcoin/cold-wallet.dat). Rather than rewrite the isolated script, I wrote a set of airgap utilities for Bitcoin/Litecoin core that includes loading a cold wallet into an offline session. The scripts are optimised for an offline Tails session and you can access the package here. The package allows you to:

Dump private keys into a symmetrically GPG encrypted file - for super paranoid paper backups
Load a fresh copy of Bitcoin/Litecoin core (useful for generating new cold wallets)
Load a specified cold wallet into an instance of Bitcoin/Litecoin Core
Check the passphrase for a specified cold wallet

Resources

BASH scripts to manage Bitcoin Core/Litecoin Core wallets in an offline Tails session

Bitcoin Cold Storage Using a Bitcoin Core Wallet

Fri, 18 Aug 2017 10:19:00 +0100

This article describes how to:

Create a cold-storage wallet using Bitcoin Core in a live offline Tails session
Generate and collect public Bitcoin addresses for the cold wallet
Manage the wallet and wallet passphrase

The cold wallet is created by running Bitcoin Core in an offline Tails session. Once created, keep the wallet cold - never enter the passphrase within anything other than an offline Tails (or similar) session.

I’ve written these guidelines for myself as an aid to memory - please use them with caution. If you’re not comfortable using the command line, get a Trezor, Ledger or other hardware wallet.

Don’t keep your funds on an exchange: Loads of exchanges have been hacked, so you’re better off holding funds in a cold wallet of some description.

Prerequisites

You’ll need a USB drive with a working copy of Tails: download the Tails ISO, verify it, then install Tails on a USB drive using the Tails installer. You’ll then be able to boot into a very secure live environment. Detailed instructions on Tails installation for Linux Debian, Ubuntu and Mint can be found here.

You will also need the Bitcoin Core binaries, which you can get here. Once you’ve downloaded Bitcoin Core, you should verify that the file hasn’t been tampered with.

To run Tails, power down your PC, insert the Tails USB drive and boot from the USB. Tails is surprisingly usable - it’s based on Debian and comes bundled with a load of utility and privacy focused software. On a modern computer (and decent USB drive) I find it stable, quick-booting and responsive.

Encrypted Persistence Volume

Note that Tails is amnesiac by design (The “a” in Tails stands for amnesiac) - nothing will be stored from your session unless you set up an encrypted persistent volume.

The first time you boot into Tails, create an encrypted persistent storage partition on the USB drive - this is straightforward, just follow the instructions here. Carefully store the passphrase for your encrypted persistence volume - I recommend KeePassX for generating and storing strong passwords/passphrases. KeePassX is actually bundled with Tails.

The encrypted partition (/Persistent) provides a persistent location for:

Bitcoin Core binaries
Storing the cold wallet
Storing a CSV formatted file listing public Bitcoin addresses to receive funds

Once you’ve added a persistent storage partition, power down the PC. Mount the USB drive in your regular distro - for which you’ll need the encryption passphrase - and add the Bitcoin Core binaries to the Tails USB drive’s /Persistent volume.

Run Bitcoin Core in an Offline Tails Session

Temporarily disconnect the computer’s ethernet connection and/or disable the Wifi connection. Boot into your Tails USB disk.

Open a terminal and cd to the Bitcoin Core binaries directory: cd ~/Persistent/bitcoin-x.x.x/bin
Run ./bitcoin-qt
Hide the pop-up warning about recent transactions - the instance does not need to sync to the network for our purposes
At this point, a new wallet.dat file will have been created in the root of the data directory(e.g. ~/.bitcoin/wallet.dat)
Encrypt the wallet: in the Bitcoin Core client, click “Settings > Encrypt Wallet”
Enter a strong passphrase at the prompt (use diceware, or generate a strong passphrase using KeePassX)
Store the passphrase in a secure location - KeePassX is ideal for this

View Receiving Addresses

At this point, Bitcoin Core will have created two receiving addresses that are managed by the wallet. Access these addresses via the Bitcoin Core File menu:

Click “File” > “Receiving Addresses”
Click export to export in CSV format
Save this file on the persistent drive

“Label” Refers to the “Account” Name. Two unnamed accounts are created by default during wallet setup. Rename these if necessary - right click the label field in the Receiving Addresses view and click “Edit”.

Note that “accounts” are deprecated in Bitcoin Core, but you’ll find numerous references to accounts in online tutorials and documentation. The API still has lots of references to accounts - for example, to list out addresses, the getaddressesbyaccount method is required.

Save the Wallet!

IMPORTANT: Save the generated wallet to your persistent volume.

I recommend a logical file naming protocol that involves the date and the words “cold-wallet”:

# Open terminal and copy the wallet to the persistent volume
cp ~/.bitcoin/wallet.dat ~/Persistent/cold-wallets/20-06-2017-cold-wallet.dat

If you don’t explicitly save the wallet file, when you close your Tails session, the wallet will be lost. If you were to send funds to a public address controlled by this wallet, they would be irretrievable.

Double Check

Before sending Bitcoin to addresses managed by the new wallet, Open the saved *.dat file in an offline Tails session and double check:

It is managing the correct public addresses
The encryption passphrase is correct (enter walletpassphrase "yourpassphrase" 5 in console - the correct passphrase should return “null”)

You can open a wallet by passing in the -wallet option when opening Bitcoin Core - BUT the wallet must be located in the Bitcoin core data directory. You can add a symlink to set this up:

ln -s ~/Persistent/cold-wallets/20-06-2017-cold-wallet.dat ~/.bitcoin/cold-wallet.dat

You can now launch Bitcoin core and reference the cold wallet:

# Move into the Bitcoin Core directory
cd ~/Persistent/bitcoin-x.x.x/bin

# Launch Bitcoin Core with a specified wallet:
./bitcoin-qt -wallet=cold-wallet.dat

I’ve written a utility script that connects up the cold wallet: https://github.com/DavidCWebs/bitcoin-cold-wallet-setup.

Transfer Funds

Transfer Bitcoin to one of the public Bitcoin addresses managed by the cold wallet.

Don’t unlock the cold wallet on any internet-connected machine, or on any machine that might be infected with malware.

Backup Private Keys

If required, you could dump the private keys of the cold wallet. If you do this, you’ll need to keep these extremely safe. Access to the private keys equates to full control over the Bitcoin. On the plus side, you could store these keys on paper which would ensure that you will always be able to import keys.

Personally, I have decided not to backup private keys - instead relying on multiple copies of a cold wallet encrypted with a strong passphrase. The wallet can then be safely stored online, since it is effectively useless without the encryption passphrase. Note that if someone gained control of your encrypted wallet without the passphrase, although they would not be able to transfer funds they would be able to access your transaction history. The passphrase should be stored securely (for example, in multiple copies of an encrypted KeePassX database in more than one location).

Accessing the Wallet

To access funds managed by the wallet, you just need to open it in a “hot” Bitcoin Core client. To do this in Ubuntu:

Move the cold wallet into the Bitcoin Core data directory
Run bitcoin-qt, passing the wallet filename to the “wallet” option

For example:

cd /path/to/bitcoin-core
./bitcoin-qt -wallet=cold.dat

This will load your cold wallet into your Bitcoin Core client. Once you’ve entered the passphrase, it’s good practice to consider the wallet “warmed-up” - so you could transfer any remaining funds to a new cold wallet.

If you’re managing cold wallets like this, be very careful to:

Encrypt wallets with a strong passphrase
Secure your passphrase and your wallet - there is NO password reset option if you forget!

Resources

Bitcoin Wiki: Setup secure offline savings wallet
Detailed instructions on Tails installation in Linux Debian, Ubuntu and Mint
Download Bitcoin Core
Bitcoin Wiki on Cold Storage
Bitcoin Core APIs
Bitcoin Core GitHUb issue on Accounts discusses replacing account concept with simple labels
Utility script to set up cold wallet for offline viewing
Diceware - easier if you use gedit displaying line numbers for ease of lookup - do this offline if you’re paranoid!

Open Encrypted Bitcoin Core Wallet

Mon, 31 Jul 2017 21:41:00 +0100

To decrypt a Bitcoin wallet in the bitcoin-cli, you need the walletpassphrase command. This article shows how to avoid leaving the password in your shell history.

These notes refer to Bitcoin core running on Ubuntu 16.04.

# Silently read password
read -s PASSWORD
# Enter password

# Open the wallet for 60 seconds
./bitcoin-cli walletpassphrase "${PASSWORD}" 60

# Do stuff - in this case, dump out a private key for a specified address
./bitcoin-cli dumpprivkey 12JZ2gDDRRLV9H5spbzygFok5VDd4jsWCV

# Lock Wallet when finished
./bitcoin-cli walletlock

# Empty the $PASSWORD variable
PASSWORD=

Automatically Embed GitHub Gists in WordPress

Mon, 31 Jul 2017 11:56:00 +0100

To embed GitHub Gists in WordPress by simply pasting the Gist URL on a new line in the WordPress visual editor, register a handler on the wp_embed_register_handler hook.

Register Gists for Oembed Functionality

<?php
wp_embed_register_handler( 'gist', '#https?://gist.github.com/.*#i', function($url) {
    $url = $url[0];
    if( !preg_match( '#\.js$#i', $url ) )
        $url .= '.js';
    return sprintf('<script src="%s"></script>', esc_attr($url));
});

Duplicate a Drive in Linux

Fri, 28 Jul 2017 14:25:00 +0100

The dd command allows you to duplicate or clone drives exactly - the target will have the same permissions, formatting and media name.

If the source drive is LUKS encrypted, the target drive will have exactly the same encryption state, and will be unlocked by the same passphrase as the original.

I find this useful for making cloned copies of thumbdrives containing password databases.

Determine Paths

This can be achieved with the lsblk command:

lsblk

# Typical output
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                             8:0    0 465.8G  0 disk  
├─sda1                                          8:1    0   512M  0 part  /boot/efi
├─sda2                                          8:2    0   488M  0 part  /boot
└─sda3                                          8:3    0 464.8G  0 part  
  └─sda3_crypt                                252:0    0 464.8G  0 crypt
    ├─ubuntu--vg-root                         252:1    0 448.8G  0 lvm   /
    └─ubuntu--vg-swap_1                       252:2    0    16G  0 lvm   
      └─cryptswap1                            252:3    0    16G  0 crypt [SWAP]
sdb                                             8:16   0   1.8T  0 disk  
└─sdb1                                          8:17   0   1.8T  0 part  /media/datadrive
sdc                                             8:32   1   7.5G  0 disk  
└─sdc1                                          8:33   1   7.5G  0 part  
  └─luks-04235321-8ad9-4631-934c-2c09cfa700e7 252:4    0   7.5G  0 crypt /media/david/secure-data
sdd                                             8:48   1   7.5G  0 disk  
└─sdd1                                          8:49   1   7.5G  0 part  /media/david/Thumbdrive
loop0                                           7:0    0  80.5M  1 loop  /snap/core/2462
loop1                                           7:1    0  80.5M  1 loop  /snap/core/2381
loop2                                           7:2    0  79.5M  1 loop  /snap/core/2312
loop3                                           7:3    0 182.6M  1 loop  /snap/atom/9
loop4                                           7:4    0 182.6M  1 loop  /snap/atom/8

In this case, /dev/sdc is the source (or ‘input file’ in dd parlance) and /dev/sdd is the target (or ‘output file’).

Clone Drives

Make sure you correctly identify the drives - especially the output/target, since dd will overwrite all data. In this case, we can easily identify the pair of thumbdrives based on their size.

## Clone sdc to sdd
sudo dd if=/dev/sdc of=/dev/sdd

When finished, the new drive will be an exact copy - with the same identifier. As such, it won’t be possible to mount the original and the clone at the same time.

Because of this, if you want to repeat the process you may need to temporarily change the passphrase on the target drive, or wipe it altogether. To wipe the drive:

sudo dd if=/dev/zero of=/dev/sdd

References

/dev/zero - provide null characters
Cloning an encrypted drive

Ignore IP Addresses in Fail2Ban

Thu, 20 Jul 2017 10:29:00 +0100

Fail2Ban can help protect your Linux server from attack. It’s a Python package that monitors log files and dynamically adjusts firewall rules to block malicious IP addresses.

You can set (or use pre-configured) Python regexes - in Fail2Ban parlance, ‘filters’ - to determine malicious requests. If Fail2Ban detects a particular frequency of the filter regex in a specified log file, an iptables rule is dynamically generated that bans the abusive IP address for a set period of time.

Fail2Ban Config

Fail2Ban can be configured with actions that determine the exact behaviour for a given ‘jail’. In this way, responses to particular actions can be fine-tuned. For example, you could configure Fail2Ban to trigger a ban for the originating IP address:

After 3 failed SSH login attempts over a 10 minute period
After a single attempt at user-enumeration - in the context of WordPress, this type of request is almost certainly malicious
After 6 failed submissions over a five minute period on a public-facing user login form

In Ubuntu 16.04, the main configuration file is /etc/fail2ban/jail.conf. You should not edit this, as your customisations will be over-written on upgrade. Instead, extend the configuration by either:

Adding override rules to a /etc/fail2ban/jail.local file
Adding config rules on a file-by-file basis in the /etc/fail2ban/jail.d directory, with the .conf suffix

The config files should reference filters - which determine the target regexes. There are a wide range of pre-configured regexes available here: /etc/fail2ban/filter.d.

Override IP Addresses

Fail2Ban allows you to list IP addresses which should be ignored. This can be useful for testing purposes, and can help avoid locking clients (or yourself) out unecessarily.

To achieve this, just add ignoreip = 127.0.0.1/8 x.x.x.x y.y.y.y to the relevant action.

Note that if you add a specific IP address to an action, it will override the default value. The child action overwrites the ignoreip rule - it does not merge IP addresses.

Example jail.local

The code below defines actions in a single file. You don’t need to copy across the entire /etc/fail2ban/jail.conf file - just extend the necessary sections.

# File: /etc/fail2ban/jail.local

# Fail2ban overrides
# These rules override `/etc/fail2ban/jail.conf`.
# =============================================================

[DEFAULT]
# Ban bad hosts for one hour:
bantime = 3600
# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport
# Emails
destemail = serveradmin@example.com
sendername = Fail2Ban_Archimedes
mta = mail
# Action
action = %(action_mwl)s
# List of safe IP addresses
ignoreip = 127.0.0.1/8 x.x.x.x

# ============================================================
# Jails Enabled
# ============================================================

# sshd jail is already enabled by default, so this isn't strictly necessary
[sshd]
enabled = true
bantime  = 36000
# Override the default value - overwrites, does not merge
ignoreip = 127.0.0.1/8 x.x.x.x y.y.y.y

[ssh]
enabled  = true
port     = 9999
# Use a pre-configured filter
filter   = sshd
bantime  = 36000
logpath  = /var/log/auth.log
maxretry = 6
ignoreip = 127.0.0.1/8 x.x.x.x y.y.y.y

# For WordPress jails, you need to configure WordPress to write to the specified log.
# You also need to create appropriate filters e.g. `/etc/fail2ban/filter.d/wordpress-hard.conf`.

# This jails IP addresses that are certainly malicious (e.g. trying to enumerate users)
[wordpress-hard]
enabled = true
filter = wordpress-hard
bantime  = 36000
logpath = /var/log/auth.log
maxretry = 1
port = http,https

# Softer jail - allows users to retry their logins
[wordpress-soft]
enabled = true
filter = wordpress-soft
bantime  = 36000
logpath = /var/log/auth.log
maxretry = 3
port = http,https

References

Fail2Ban

Declaring Variables in Ecmascript 6

Sat, 15 Jul 2017 20:30:00 +0100

Variables are used as symbolic names for values. Variable names are known as ‘identifiers’. In JavaScript, identifiers are case-sensitive and must begin with a letter, underscore or dollar sign.

TLDR;

Don’t use var to declare variables - use const where possible and let otherwise. Use a transpiler like babel to sort things out for pre-ES6 browsers.

Variable Declaration

Ecmascript 6 introduced the let and const statements for declaring variables. This means that variables can be declared as follows:

var: Declare a variable, optionally assigning a value
let: Declare a block-scoped local variable, optionally assigning a value
const: Declare a block-scoped read-only constant

Source: MDN JavaScript Guide

Scoping

Variables defined with var and const are function scoped, whereas variables defined with let are block-scoped.

Block scoping basically means that the variable scope is limited by the enclosing {} - so a variable is declared by means of let is limited in it’s scope to the block, statement or expression in which it is used.

This can be useful when dealing with loop iterations - when the counter variable is defined by var, all loop iterations (confusingly) share the same function-scoped counter variable, leading to unexpected results. Declaring the index with let produces a much more intuitive result:

// Declare iterator with let
// -----------------------------------------------------------------------------
'use strict'
var list = []
for (let i = 0; i < 5; ++i) {
  list.push(() => {
    return i;
  })
}
console.log(list.map((f) => { return f() }))
// returns Array [ 0, 1, 2, 3, 4 ]

// Declare iterator with var
// -----------------------------------------------------------------------------
'use strict'
var list = []
for (var i = 0; i < 5; ++i) {
  list.push(() => {
    return i;
  })
}
console.log(list.map((f) => { return f() }))
// returns Array [ 5, 5, 5, 5, 5 ]

Source: SE answer

const

This is another way to declare variables - like let, variables const has block scope. Variables declared in this way are read-only constants.

You can’t change the value of a constant by re-assignment, and it can’t be re-declared. However, the value held by a constant is not immutable. If you define an object using const, the object contents can be amended.

A constant can’t have the same name as a function or variable in the same scope. Constants can be declared with uppercase or lowercase letters - a common convention is to use all-uppercase letters.

A constant can have global scope (i.e. on the window object) if defined outside a block.

If you don’t need to change a variable, it’s probably a good idea to use const. This helps to Minimize Mutable State - Mutable state makes mistakes more likely.

References

SE answer with a good description of why let is better than var
Douglas Crockford discussing the issue
Useful YouTube video from funfunfunction - as well as being really useful, this guy is very entertaining!
MDN on const
MDN on let

Reorganise Letsencrypt Certificates

Thu, 13 Jul 2017 11:26:00 +0100

This article refers to a Letsencrypt client as installed by sudo apt-get install python-letsencrypt-apache. Up until quite recently, this was the recommended Letsencrypt installation for Ubuntu Xenial 16.04.

Insurance: Make a Backup!

Backup the entire /etc/letsencrypt directory - recursively copy the entire letsencrypt directory:

cp /etc/letsencrypt/ /etc/letsencrypt.backup -r

If you mess up during the process of certificate reorganisation, revert to the original and save the broken state for reference:

# Recover from broken state
mv /etc/letsencrypt /etc/letsencrypt.broken && mv /etc/letsencrypt.backup/ /etc/letsencrypt

Certificate Lineage

Determine certificate lineages by listing out the domains associated with each certificate - look in subdirectories under /etc/letsencrypt/live:

openssl x509 -in /etc/letsencrypt/live/example.com/cert.pem -text -noout | grep DNS
openssl x509 -in /etc/letsencrypt/live/www.example.com/cert.pem -text -noout | grep DNS

See additional information for a breakdown of this command.

Remove a Superfluous Certificate

We determine that the example.com cert is superfluous, and holds references to invalid domains. The cert is not being used, but generates ugly error messages during the renewal dry-run.

Be careful - make sure the cert is not being referenced in any Virtual Host directives.

If you are sure it’s safe, remove it and run the renewal process:

rm -rf /etc/letsencrypt/live/example.com/
rm -rf /etc/letsencrypt/archive/example.com
rm /etc/letsencrypt/renewal/example.com.conf

# Attempt a dry-run renewal first if necessary
sudo letsencrypt renew --dry-run --agree-tos

# Run the renewal
sudo letsencrypt renew

Reissue Certificate

I haven’t attempted this - but running the following should install a new certificate:

# CAUTION: Not tested
sudo letsencrypt certonly --webroot-path /var/www/html -d example.com -d www.example.com

If your virtual hosts for the specified domains are referencing certs like so:

SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem

…the new cert should work.

Additional Information

OpenSSL is an open source toolkit for the SSL and TLS network protocols and related cryptography standards, accessed with the - openssl utility. The x509 command provides utilities for displaying, converting and signing certificates. Summary of the above command:

-in filename: the input filename to read from (standard input if not specified)
-text: print the certificate in text form
-noout: prevent output of the encoded version of the request

The text form output includes full details - public key, signature algorithms, issuer & subject names, serial number, extensions present, any trust settings and the DNS records covered by the certificate. In the context of this article, we’re only interested in the certificate lineage/associated domains - so the openssl x509 command is piped to grep DNS to output the DNS data only.

References

This article is largely based on the very useful answer on Letsencrypt discourse by pfg
openssl x509 man page

Setup New Vue Webpack Project With Bulma

Tue, 11 Jul 2017 20:10:00 +0100

How to set up a new Vue project with Webpack and Bulma using the Vue CLI.

Vue CLI

Vue CLI is a simple yet powerful CLI that you can use to scaffold Vue projects.

Install Vue CLI Globally:

# Using Yarn
yarn global add vue-cli

# Using NPM
npm install -g vue-cli

You can now use the Vue CLI to scaffold projects. There are a number of templates available, but this article focuses on Webpack.

Scaffold a Project

To scaffold a Vue project with the Webpack template:

# Move into a suitable location
cd /var/www/html

# Scaffold a Vue/Webpack project into /var/www/html/new-project.dev
vue init webpack new-project.dev

Install Dependencies

Move into the new project directory and install the base dependencies:

# Using yarn:
yarn install

# Using NPM:
npm install

Add Dependencies

Add Bulma:

# Using Yarn
yarn add bulma

# Using NPM:
npm install bulma

# To add multiple packages with Yarn, separate with a space. e.g.:
yarn add axios bulma

Run Dev server

# Using Yarn
yarn dev

# Using NPM
npm run dev

Connect Up Bulma

Add a suitable SASS manifest file that you’ll use to import all your SASS modules:

# From project root:
touch src/assets/sass/main.scss

Open this file and import the main Bulma Sass file:

// src/assets/sass/main.scss
@import '~bulma/bulma'

Note that the tilde symbol refers Webpack/sass-loader to the node_modules directory.

You then need to reference this in Webpack’s main entry point, src/main.js:

// src/main.js
import Vue from 'vue';
import App from './App';
import router from './router';

// Require the main Sass manifest file
require('./assets/sass/main.scss');

Vue.config.productionTip = false;

/* eslint-disable no-new */
new Vue({
  el: '#app',
  router,
  template: '<App/>',
  components: { App },
});

Bulma Sass - Additional Dependencies

Note that to compile Sass, Webpack needs the node-sass node module and the Webpack sass-loader module.

Add these to your project dependencies:

yarn add node-sass sass-loader

You can now run yarn dev to build your development server.

Problem With Ubuntu Snap-Installed Atom (code editor) and Webpack Server

You may experience problems because xdg-open does not run properly when launched within Atom (i.e. in platformio-ide-terminal) when Atom has been installed by means of a Ubuntu snap. This is because there is a GTK+ 2.x and GTK+ 3 conflict.

To resolve this, you can manually specify the browser to be used when the Webpack server is triggered. Amend build/dev-server.js by passing in a preferred browser as a second parameter into the call to node-opn:

devMiddleware.waitUntilValid(() => {
  console.log('> Listening at ' + uri + '\n')
  // when env is testing, don't need open it
  if (autoOpenBrowser && process.env.NODE_ENV !== 'testing') {
    // Set the browser here!
    opn(uri, {app: 'firefox-developer'})
  }
  _resolve()
})

Install and Run Geth (golang implemenation of Ethereum) on Ubuntu

Mon, 03 Jul 2017 10:48:00 +0100

Ethereum is a decentralized blockchain-based platform that runs smart contracts. Because of the decentralized nature of Ethereum, smart contracts run exactly as programmed - short of an internet apocalypse, there is no possibility of downtime, censorship, fraud or third-party interference. The network is cryptographically secure, decentralized and tamper-proof.

The ethereum network is “fuelled” by Ether. Clients of the platform pay Ether to the machines carrying out the requested computations. In this way, people are compensated for contributing resources and developers are incentivized to write efficient applications.

It’s a bit tricky to get started with Ethereum - there are numerous confusing articles floating about the internet. This article is intended to be a very basic guide to getting started with the golang implementation of Ethereum - geth, a command line tool. The article covers installing the package, setting up accounts and syncing the Ethereum blockchain. For a curated list of useful resources, see the References section.

Go Ethereum: Geth

Go Ethereum (geth) is the official golang implementation of the Ethereum protocol. It is one of the three original implementations (the others being C++ and Python) of the Ethereum protocol. The package is fully open source and licensed under the GNU LGPL v3.

Geth is basically a command line interface (CLI) for running a full ethereum node. As well as being a CLI, geth can be associated with a JavaScript console

Installation

Install go-ethereum on Ubuntu:

sudo apt-get install software-properties-common
sudo add-apt-repository -y ppa:ethereum/ethereum
sudo apt-get update
sudo apt-get install ethereum

Note that this installs a suite of Ethereum utilities as well as geth:

bootnode
evm
disasm
rlpdump
ethtest

If you just want to install geth, run sudo apt-get install geth rather than sudo apt-get install ethereum.

Initial Setup: Create Account

Show geth options:

geth --help

Set up an account:

# Set up account - enter a strong passphrase
geth account new

# Check that accoun
geth account list

A Note on Passwords and Security

If you forget your account password/passphrase, you lose all access to that account - and any funds that it holds. There is no password reset! Notwithstanding this, you should set a strong passphrase for the sake of good security.

KeePassX is available for Ubuntu - it is a password storage tool that constitutes an encrypted database. I strongly recommend using such a tool - you just need to remember one strong passphrase to unlock your KeePassX database. This will allow you to save extremely strong passwords for your Ethereum accounts, since you’ll never have to remember them.

Initial Setup: Sync the Ethereum Blockchain

The best option is to run geth with the fast sync option - this spares bandwidth usage, though it requires extra processing power. Fast sync downloads transaction receipts rather than the entire blockchain for historical records, and pulls an entire recent state database. See this PR comment for more details.

To fast sync with a 2GB memory cache:

geth --fast --cache=2048

This builds the ethereum blockchain in ~/.ethereum/geth/chaindata.

If you have synced the blockchain on one trusted computer, you can export this and import to a second machine. I tried this - it was taking so long that I aborted and opted for the fast-sync method.

As of 3 July 2017 the chaindata directory is 29 GB. If you have a bandwidth cap, you could run the sync during unrestricted time-periods by means of a crontab script.

Running Geth to Stay in Sync

Once you have fast-synced, you need to run geth periodically to keep up-to-date with the ethereum blockchain.

JavaScript Console

You can launch geth with the console option to provide an interactive JavaScript console. Alternatively, launch geth and run geth attach in another terminal. The latter method is convenient as you’ll be less distracted by logging output.

This example shows how you might monitor progress in an attached console:

# Assuming geth is running in a separate terminal...this will open a Javcascript console
geth attach

The JavaScript console:

// Check if syncing
eth.syncing

// Get the current block number
eth.blockNumber

// Return the number of peers connected
net.peerCount

// Exit this console
exit

Clean Exit of Geth

You can shut down geth with CTRL-C.

If you have attached a JavaScript console to a running geth instance, this won’t work. You need to first exit the JavaScript console - you can do this cleanly by entering exit or CTRL-C in the JS console first, before closing the original geth instance by entering CTRL-C in the relevant terminal.

References

Verification of Bitcoin Core Download in Ubuntu

Sat, 01 Jul 2017 10:09:00 +0100

There are numerous ways to install Bitcoin Core in Ubuntu:

via PPA
Cloning from Github
Downloading the binaries from https://bitcoin.org/en/download

Install Via Downloaded Package from bitcoin.org

Download Bitcoin Core: https://bitcoin.org/en/download

Select Linux (tgz), which will trigger a download - at the time of writing: bitcoin-0.14.2-x86_64-linux-gnu.tar.gz.

Verification

Download the release signature document. At the time of writing, this is here: https://bitcoin.org/bin/bitcoin-core-0.14.2/SHA256SUMS.asc. This downloads SHA256SUMS.asc.

This file is a signed PGP message that contains the SHA256 sums for the various Bitcoin core downloads. By comparing the relevant value with the SHA256 sum of the bitcoin-0.14.2-x86_64-linux-gnu.tar.gz that you have downloaded, you can verify the authenticity of what you have downloaded.

You should first verify that the signature document itself is authentic. This is done using Gnu Privacy Guard (GPG)

Verification of the Signatures Document

Download the relevant GPG signing key - these are presented on the download page under the title “Bitcoin Core Release Signing Keys”. Click the correct version to download the relevant key. At the time of writing, this is laanwj-releases.asc.

Your ~/Downloads directory should now contain:

david@desktop:~/Downloads$ ls -la
total 24164
drwxr-xr-x  2 david david     4096 Jun 29 19:27 .
drwx------ 47 david david    12288 Jun 29 19:14 ..
-rw-rw-r--  1 david david 24607581 Jun 29 18:21 bitcoin-0.14.2-x86_64-linux-gnu.tar.gz
-rw-rw-r--  1 david david    17940 Jun 29 18:57 laanwj-releases.asc
-rw-rw-r--  1 david david     1957 Jun 29 18:55 SHA256SUMS.asc

Import the Public Key

Move into your ~/Downloads directory and run:

gpg --import laanwj-releases.asc
gpg: /home/david/.gnupg/trustdb.gpg: trustdb created
gpg: key 36C2E964: public key "Sender Name (Bitcoin Core binary release signing key) <sender@example.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg: no ultimately trusted keys found

# Verify that key was imported:
gpg --list-keys
/home/david/.gnupg/pubring.gpg
------------------------------
pub   4096R/36C2E964 2015-06-24 [expires: 2019-02-14]
uid                  Sender Name (Bitcoin Core binary release signing key) <sender@example.com>

Check Signatures Document

You can now establish the authenticity of the signatures document by running:

gpg --verify SHA256SUMS.asc
gpg: Signature made Sat 17 Jun 2017 11:33:26 IST using RSA key ID 36C2E964
gpg: Good signature from "Sender Name (Bitcoin Core binary release signing key) <sender@example.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: ...

Check the Authenticity of the Download: SHA256 Checksum

You can now use the signatures document to establish the authenticity of the Bitcoin download.

cd ~/Downloads

sha256sum -c SHA256SUMS.asc 2>&1 | grep OK

# If all is OK, this will be the result:
bitcoin-0.14.2-x86_64-linux-gnu.tar.gz: OK

Note: the tar.gz file provided for download doesn’t appear to contain source files (as suggested here), so you can’t use this to compile. It does include binaries.

Install Via Github

You can clone Bitcoin source code from Github, and compile from this.

cd ~
git clone https://github.com/bitcoin/bitcoin.git

There are quite a few dependencies - these are outlined in the doc/build-unix.md document: https://github.com/bitcoin/bitcoin/blob/master/doc/build-unix.md.

Installing from GitHub is probably a pretty secure method as you’ll be pulling the files over an encrypted connection - so long as you trust that the Github repo has not been tampered with (which makes Github probably as trusted an installation route as the downloadable binaries).

Install Via PPA

To install via PPA, add ppa:bitcoin/bitcoin to your system’s Software Sources:

sudo add-apt-repository ppa:bitcoin/bitcoin
sudo apt-get update

This is the stable Channel of bitcoin-qt (GUI) and bitcoind (CLI) for Ubuntu. The Launchpad description actually recommends use of the official binaries, where possible, to limit trust in Launchpad/the PPA owner.

Running Bitcoin

Once you have verified your download, move the downloaded file into a suitable location where the binaries can be run. The downloaded file is a zipped directory so it should be OK to unzip in situ:

tar -zvxf bitcoin-0.16.0-x86_64-linux-gnu.tar.gz

The binaries are located in the bin directory. To run bitcoin-qt, move into bin and run ./bin/bitcoin-qt. The first time you run this, the programme will build the default data directory for you.

It is more convenient to create a symlink to the executables (e.g. bin/bitcoin-qt, bin/bitcoind) in /usr/local/bin. You can then easily start Bitcoin executables from the command line.

TLDR

Download core:

# NB: check version
wget https://bitcoincore.org/bin/bitcoin-core-0.16.0/bitcoin-0.16.0-x86_64-linux-gnu.tar.gz

Download signatures:

wget https://bitcoincore.org/bin/bitcoin-core-0.16.0/SHA256SUMS.asc

Check signatures doc:

gpg --verify SHA256SUMS.asc

Verify the signing key e.g.: https://github.com/bitcoin/bitcoin/tree/master/contrib/verifybinaries. Verify the shasum for the download:

sha256sum -c SHA256SUMS.asc 2>&1 | grep OK

If all is OK, this will be the result:

bitcoin-0.16.0-x86_64-linux-gnu.tar.gz: OK

Move the download and unzip:

tar -zvxf bitcoin-0.16.0-x86_64-linux-gnu.tar.gz

Resources

Enable all Apache Config Files

Thu, 22 Jun 2017 16:44:00 +0100

Use this BASH script to quickly enable all Apache virtual host configurations.

THis can be handy if for example you are setting up a new development environment.

Background: Enabling Virtual Hosts in Apache 2.4

To set up Apache 2.4 virtual host configurations under Debian/Ubuntu you add a configuration file with a <VirtualHost> block in the /etc/apache2/sites-available directory.

Such config files must have the .conf file extension.

To enable the site, you need to run the a2ensite script, passing the config filename as an argument. This creates a symlink to the config file in the /etc/apache2/sites-enabled directory. You then need to reload Apache.

Script

The following script loops through all config files in /etc/apache2/sites-available and runs a2ensite. There’s no problem if any site is already enabled, since this does not result in an error.

#!/bin/bash
#
# Enable all config files in /etc/apache2/sites-available directory.
#
# Usage: Make this script executable and run as a user with sudo privileges.
# ==============================================================================

BASE_DIR="/etc/apache2/sites-available"

for FILEPATH in $BASE_DIR/*; do
  if [[ -f "$FILEPATH" ]]; then
    CONF=$(basename "$FILEPATH")
    a2ensite ${CONF}
    if [ $? != 0 ]
    then
      echo "a2ensite NOT successful for ${CONF}"
    else
      echo "a2ensite successful for ${CONF}"
    fi
  fi
done

# Reload apache
read -p "Do you want to reload Apache (y/n)?" CONT
if [ "$CONT" = "y" ]; then
    service apache2 reload
    if [ $? -eq 0 ]
    then
      echo "Apache reloaded - check sites in browser."
    else
      echo "There was a problem reloading Apache."
    fi
else
  echo "Changes won't take effect until Apache is reloaded or restarted.";
fi

TL;DR

Copy across the contents of /etc/apache2/sites-available from the old environment to the new (e.g. with rsync).

Download and run the above script:

# Make a directory if necessary
mkdir -p ~/sysadmin/apache
cd ~/sysadmin/apache

# Get the script
curl https://gist.githubusercontent.com/DavidCWebs/509e6e8829e24fb7cfa1e6c82c2b24e2/raw/982693b8de7082acb44082274628de0a63593929/a2enmod-all.sh -o a2enmod-all.sh

# Make executable
sudo chmod +x a2enmod-all.sh

# Run script
sudo ./a2enmod-all.sh

Move across /etc/hosts from the old environment to the new.

References

a2ensite, a2dissite Man page

Vue Components

Thu, 01 Jun 2017 09:01:00 +0100

Vue Components

Vue provides a way for you to abstract elements into re-usable pieces (or in other words, components!).

Vue components are custom elements that have behaviour attached to them by the Vue compiler. For example, you might define an <article> component that encapsulates all the code relating to display of content associated with an article.

I’m new to Vue so please treat this content with suspicion! If you have any Vue tips, or if you’d like to point out where I’m going wrong, please leave a comment.

Data can be passed into components by means of props and slots.

Vue provides several methods for registering and using components. In general, global components are registered by means of Vue.component(id, definition), where:

id is a string that will be set to the component name
definition is a function or an options object (that might define markup, props etc)

Registering a Component: Basic Method

// Register component before instantiating the Vue instance
// Reference the template inline
Vue.component('news-article', {
  template: `
  <div class="article"><h2>This is an Article Title</h2></div>
  `
});

// New Vue root instance
new Vue({
  el: '#app'
})

In this example, the HTML for the template is passed to the constructor inline. For a project of any size, this is likely to be too messy.

Another option is to define the HTML for the template as a script tag of type “text/x-template” (which will be ignored by browsers):

<script type="text/x-template" id="news-article-template">
  <h2>This is an Article Title</h2>
</script>

<script type="text/javascript">
Vue.component('news-article', {
  template: "#news-article-template"
});
</script>

The constructor now references the id of the “text/template” template.

Note that the order in which you define the template is important. You must define the text/x-template tag before you reference it in the Vue.component constructor.

Registering a Component: Within a JavaScript File

The example below registers a <custom-note></custom-note> component and defines a prop (title) that can be passed in to the component. It also sets up validation on the prop - which returns a helpful error if the script runs in development mode.

Vue.component('custom-note', {
  props: {
    title: {
      type: String,
      required: true
    }
  },
  template: `
    <div class="card">
      <h2 class="title">{{title}}</h2>
      <slot></slot>
    </div>
  `
})

new Vue({
  el: '#root',
})

A simpler way to define the prop:

Vue.component('custom-note', {
  props: ['title'],
  template: `
    <div class="card">
      <h2 class="title">{{title}}</h2>
      <slot></slot>
    </div>
  `
})

Laravel 5.4 and MariaDB Errors

Sun, 28 May 2017 19:59:00 +0100

When running migrations on Laravel 5.4 and MariaDB you may encounter this error:

[Illuminate\Database\QueryException] SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes (SQL: alter table `users` add unique `users_email_unique`(`email`))

[PDOException] SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes

Solution

Amend the AppServiceProvider - which you’ll find here: app/Providers/AppServiceProvider.php. Use Illuminate\Support\Facades\Schema to set a default string length which can be used in migrations. The ameded class might look like this:

<?php

namespace App\Providers;

use Illuminate\Support\Facades\Schema;
use Illuminate\Support\ServiceProvider;

class AppServiceProvider extends ServiceProvider
{
    /**
     * Bootstrap any application services.
     *
     * @return void
     */
    public function boot()
    {
        Schema::defaultStringLength(191);
    }

    /**
     * Register any application services.
     *
     * @return void
     */
    public function register()
    {
        //
    }
}

Migrate the Database

You may end up with a part-migration error, whereby tables have been created during a broken migration, but the migrations table is missing. Here’s an example showing the database after receiving the above-quoted error:

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 82
Server version: 10.1.23-MariaDB-1~xenial mariadb.org binary distribution

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [my_db_dev]> show tables;
+---------------------+
| Tables_in_my_db_dev |
+---------------------+
| migrations          |
| users               |
+---------------------+

2 rows in set (0.00 sec)

With the database in this state, you won’t be able to roll back migrations.

To fix, delete the tables manually at the MariaDB prompt:

# Open the MariaDB command line focused on your database:
mysql -u root -p my_db_dev

# Show tables to determine what tables you need to drop:
SHOW TABLES;

# Drop tables:
DROP TABLE users, migrations;

After manually amending the database, close the MariaDB CLI by typing \q.

At this point you should be able to run the migration successfully:

php artisan migrate

Nano Highlighting in Config Files

Tue, 23 May 2017 09:18:00 +0100

It’s handy to have syntax highlighting in config files (like ~/.bashrc and Apache virtual host configs).

Nano includes syntax definitions from config files saved in the format /usr/share/nano/*.nanorc.

To add syntax highlighting for config files, add the following to /usr/share/nano/conf.nanorc:

syntax "\conf$" "conf" "bash_profile" "bash_history" "\rc$" "(\.|/|)control$" "(\.(repo|conf|config|cfg|cnf|rc|lst|list|defs|ini|desktop|mime|types|preset|cache|seat|service|htaccess)$|(^|/)(\w*crontab|mirrorlist|group|hosts|passwd|rpc|netconfig|shadow|fstab|inittab|inputrc|protocols|sudoers)$|conf.d/|.config/)"


# default text
color white "^.*$"

# special values
icolor brightblue "(^|\s|=)(default|true|false|on|off|yes|no)(\s|$)"

# keys
icolor cyan "^\s*(set\s+)?[A-Z0-9_\/\.\%\@+-]+\s*([:]|\>)"

# commands
color blue "^\s*set\s+\<"

# punctuation
color blue "[.]"

# numbers
color red "(^|\s|[[/:|<>(){}=,]|\])[-+]?[0-9](\.?[0-9])*%?($|\>)"

# keys
icolor cyan "^\s*(\$if )?([A-Z0-9_\/\.\%\@+-]|\s)+="

# punctuation
color blue "/"
color brightwhite "(\]|[()<>[{},;:=])"
color brightwhite "(^|\[|\{|\:)\s*-(\s|$)"

# section headings
icolor brightyellow "^\s*(\[([A-Z0-9_\.-]|\s)+\])+\s*$"
color brightcyan "^\s*((Sub)?Section\s*(=|\>)|End(Sub)?Section\s*$)"
color brightcyan "^\s*\$(end)?if(\s|$)"

# URLs
icolor green "\b(([A-Z]+://|www[.])[A-Z0-9/:#?&$=_\.\-]+)(\b|$| )"

# XML-like tags
icolor brightcyan "</?\w+((\s*\w+\s*=)?\s*("[^"]*"|'[^']*'|!?[A-Z0-9_:/]))*(\s*/)?>"

color green "\<(break|case|continue|do|done|elif|else|esac|exit|fi|for|function|if|in|read|return|select|shift|then|time|until|while)\>"
color green "\<(declare|eval|exec|export|let|local)\>"
color green "[{}():;|`$<>!=&\\]" "(\]|\[)"
color green "-[Ldefgruwx]\>"
color green "-(eq|ne|gt|lt|ge|le|s|n|z)\>"
color brightblue "\<(awk|cat|cd|ch(grp|mod|own)|cp|echo|env|grep|install|ln|make|mkdir|mv|popd|printf|pushd|rm|rmdir|sed|set|tar|touch|umask|unset)\>"

# Basic variable names (no braces).
color brightred "\$[-0-9@*#?$!]" "\$[[:alpha:]_][[:alnum:]_]*"
# More complicated variable names; handles braces and replacements and arrays.
color brightred "\$\{[#!]?([-@*#?$!]|[0-9]+|[[:alpha:]_][[:alnum:]_]*)(\[([[:space:]]*[[:alnum:]_]+[[:space:]]*|@)\])?(([#%/]|:?[-=?+])[^}]*\}|\[|\})"

# Comments.
color brightblack "(^|[[:space:]])#.*$"

# Strings.
color brightyellow ""(\\.|[^"])*"" "'(\\.|[^'])*'"

# Trailing whitespace.
color ,green "[[:space:]]+$"

Apache Server Security on Ubuntu 16.04 Xenial

Mon, 22 May 2017 08:47:00 +0100

This article lists some sensible configuration defaults for Apache 2.4 on a Ubuntu 16.04 server. It explains how to apply these globally at the server level.

Objectives:

Prevent the Apache server signature that is printed as part of a web request - this is not needed and gives would-be hackers info about your server
Prevent direct access to certain files (e.g. .git, .json, .sql)
Prevent directory browsing - don’t allow random strangers to sniff out your directory structure
Disallow Apache overrides at the directory level (e.g. disallow config .htaccess files)

Note that if you disallow .htaccess files you’ll probably need to include suitable rewrite rules in each site’s Apache virtual host configuration. This is not only slightly more secure, it should be faster.

Global Apache Config

Open the Apache config file:

sudo nano /etc/apache2/conf-enabled/security.conf

Make the following changes:

#...
ServerTokens Prod

#...
Set ServerSignature Off

#...
# Add the following block:
# ==============================================================================
# Prevent access to any .git directory
# ==============================================================================
<DirectoryMatch "/\.git">
    Require all denied
</DirectoryMatch>
# ==============================================================================
# Protect a specified range of files from direct access
# ==============================================================================
<FilesMatch "^(wp-config\.php|php\.ini|php5\.ini|install\.php|php\.info|readme\.md|README\.md|readme\.html|bb-config\.php|\.htaccess|\.htpasswd|readme\.txt|timthumb\.php|error_log|error\.log|PHP_errors\.log|\.svn)">
    Require all denied
</FilesMatch>

# ==============================================================================
# Prevent access to JSON config files
# ==============================================================================
<FilesMatch "\.(json)$">
    Require all denied
</FilesMatch>

# ==============================================================================
# Prevent access to sql dumps: these should NOT be in your document root, but just in case...
# ==============================================================================
<FilesMatch "\.(sql)$">
    Require all denied
</FilesMatch>

# ==============================================================================
# Prevent xmlrpc attacks - if you're not using xmlrpc, block this file to avoid hack attempts
# ==============================================================================
<FilesMatch "^(xmlrpc\.php)">
  Require all denied
</FilesMatch>

Prevent Embedding and Sniffing

Set headers for all files served:

NOSNIFF header prevents IE mime-sniffing files
SAMEORIGIN header prevents embedding of content in different site

Note this modification will cause Apache to fail unless the headers module is enabled:

sudo a2enmod headers
# ...and restart Apache

Add the following to /etc/apache2/conf-enabled/security.conf:

# Setting this header will prevent MSIE from interpreting files as something
# else than declared by the content type in the HTTP headers.
# Requires mod_headers to be enabled.
Header set X-Content-Type-Options: NOSNIFF

# Setting this header will prevent other sites from embedding pages from this
# site as frames. This defends against clickjacking attacks.
# Requires mod_headers to be enabled.
Header set X-Frame-Options: SAMEORIGIN

Prevent Directory Browsing

Open the relevant config file for editing:

# Consider making a backup of the original first:
cp /etc/apache2/apache2.conf /etc/apache2/apache2.conf.bak

# Open the file for editing:
sudo nano /etc/apache2/apache2.conf

Remove/Comment out:

<Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride None
        Require all granted
</Directory>

Add this block:

# ------------------------------------------------------------------------------
# Disable directory browsing
# ------------------------------------------------------------------------------
<Directory /var/www/>
        Options -Indexes
        Options FollowSymLinks
        AllowOverride None
        Require all granted
</Directory>

…restart Apache:

sudo /etc/init.d/apache2 restart

Your Apache server should now be a bit more secure. The configuration settings have been applied globally - rather than in numerous .htaccess files at the directory level.

Laravel Database Management

Sat, 20 May 2017 16:28:00 +0100

This article focuses on Laravel 5.4 development in a Ubuntu 16.04 desktop environment. It is assumed that a suitable database has already been created.

Create a Database Table

The Laravel way involves creating and running a suitable “migration”. Migrations allow the app to create a new table or modify existing tables. Migrations are very powerful - amongst other things, they enable:

Version control of the database
Database modifications to be easily rolled-back

Create Migration

To create a migration:

php artisan make:migration create_users_table

This creates a new file in the database/migrations directory. The filename is based on the option passed to php artisan make:migration and a timestamp - for example: database/migrations/2017_05_29_094813_create_users_table.php.

If the migration is to create a new table, you can specify the table name by passing the create option to make:migration command:

php artisan make:migration create_users_table --create=users

This will stub out the migration file with the specified table. It’s probably a good idea to name the migration descriptively in the imperative case - for example “create_xx_table” or “adding_excerpt_to_posts_table”.

The following command sets up a migration that will create a database table called “stages”:

php artisan make:migration create_stages_table --table=stages

Once the migration has been created, it needs to be run to actually create the database table:

php artisan migrate

In this case, the migration class will be in database/migrations/2017_05_20_112319_create_stages_table.php.

Before running the migration and creating the new table, you’ll probably want to add some columns. You can do this by editing the up() method on the newly created migration. For example, to add a simple title column:

<?php
public function up()
    {
        Schema::create('stages', function (Blueprint $table) {
            $table->increments('id');
            $table->string('title');
            $table->timestamps();
        });
    }

NOTE: you need to ensure that the additional table methods are added BEFORE the timestamps() method or they won’t be applied.

Run or Rollback Migration

To apply the changes to the database, you need to run the migration:

php artisan migrate

To rollback a migration:

php artisan migrate:rollback

See: https://laravel.com/docs/5.4/migrations#generating-migrations

Rollback Problems

When attempting a rollback for the first time you may experience this error:

PHP Fatal error:  Uncaught UnexpectedValueException:
The stream or file "/var/www/html/test-laravel/storage/logs/laravel.log"
could not be opened: failed to open stream:
Permission denied in /var/www/html/test-laravel/vendor
/monolog/monolog/src/Monolog/Handler/StreamHandler.php:107

This occurs because you’re running the rollback command as your current user. The storage directory will be owned by the server user (www-data in the case of Ubuntu), and will probably belong to your main user’s group (depending upon how you assigned permissions when you first installed the app).

The fix is simple - chmod the log file to give your user write permissions. The file will probably have -rw--r--r-- permissions, so change these to 664:

sudo chmod 664 /var/www/html/test-laravel/storage/logs/laravel.log

Seeding a Table

Laravel 5.4 ships with the Faker Factory class included, which can be used to “seed” the database.

Create a seeder stub for the ‘stages’ table:

php artisan make:seeder StagesTableSeeder

This stubs out a StagesTableSeeder class in database/seeds/StagesTableSeeder.php.

Example Seeder:

<?php

use Illuminate\Database\Seeder;

class UsersTableSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        factory(App\User::class, 50)->create();
    }
}

Defining Model Factories

The method above uses the factory() helper function. This references a Model Factory that you can create in database/factories. You could add all factory logic in database/factories/ModelFactory.php - but it’s easier to keep code organised if you duplicate this file and add factory logic for your models within appropriately named files. For example, database/factories/PostFactory.php:

<?php

/*
|--------------------------------------------------------------------------
| Model Factory for Posts
|--------------------------------------------------------------------------
|
| Adds a random user_id to the dummy posts.
|
*/

/** @var \Illuminate\Database\Eloquent\Factory $factory */
$factory->define(App\Post::class, function (Faker\Generator $faker) {

    $userID = App\User::inRandomOrder()->first()->id;

    return [
        'title' => $faker->sentence(),
        'body' => $faker->text(60),
        'excerpt' => $faker->text(30),
        'user_id' => $userID,
    ];
});

You can call the new seeder from the run() method in seeds/DatabaseSeeder.php:

<?php

use Illuminate\Database\Seeder;

class DatabaseSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        $this->call(UsersTableSeeder::class);
        $this->call(PostsTableSeeder::class);
    }
}

This helps to keep seeder classes nice and neat.

Run the seeders:

php artisan db:seed

Rollback all database migrations and re-seed:

php artisan migrate:refresh --seed

WordPress Reserved Terms and Post Requests

Wed, 17 May 2017 11:21:00 +0100

When submitting a form to a WordPress site, there are a number of reserved keywords that should not be used as name attributes - including “name”, “author” and “type”. Using such keywords may result in odd errors - either unexplained 404 errors or the template reverting to the index.php template.

The latter error looks particularly odd, because the permalink is unchanged. Debugging $_POST and $_REQUEST names in this context is not simple.

Most recently, when refactoring a project I managed to completely break form submission. The natural assumption is that there is a problem with your code. Eight hours of git resets and diffs later, I realized that I had set a form field with the name attribute “name” - coincidence can be a bitch.

Since I’ve cumulatively wasted a couple of days of my life on this issue, I’m documenting it here to prevent future nervous meltdowns.

Prefix all Form Names

A good general rule for WordPress forms is:

PREFIX FORM NAMES

Reserved Keywords

I can personally attest that you need to avoid the string “name” as a name attribute (use “my_prefix_name” instead). The rest of the list is taken from the WordPress Codex:

attachment
attachment_id
author
author_name
calendar
cat
category
category__and
category__in
category__not_in
category_name
comments_per_page
comments_popup
custom
customize_messenger_channel
customized
cpage
day
debug
embed
error
exact
feed
hour
link_category
m
minute
monthnum
more
name
nav_menu
nonce
nopaging
offset
order
orderby
p
page
page_id
paged
pagename
pb
perm
post
post__in
post__not_in
post_format
post_mime_type
post_status
post_tag
post_type
posts
posts_per_archive_page
posts_per_page
preview
robots
s
search
second
sentence
showposts
static
subpost
subpost_id
tag
tag__and
tag__in
tag__not_in
tag_id
tag_slug__and
tag_slug__in
taxonomy
tb
term
terms
theme
title
type
w
withcomments
withoutcomments
year

There is a more extensive list here.

References

Reset Project to the Last Git Commit

Fri, 05 May 2017 13:56:00 +0100

When you’ve made a mess of your code and you need to reset your project to the last git commit.

Reset changes and remove untracked files from the working tree:

cd /path/to/project
git reset HEAD --hard
git clean -fd

References

Using Bootstrap 4 & Laravel Mix - Autoload Tether

Sun, 30 Apr 2017 18:02:00 +0100

If you are using Bootstrap 4 with Laravel 5.4, you may encounter a problem related to Tether.

Bootstrap 4 requires Tether. Running the default Laravel mix webpack config throws an error because the Tether module is not available as a variable in the Bootstrap module. You might get a Console error like this:

Uncaught Error: Bootstrap tooltips require Tether (http://tether.io/)
    at app.f3e519c….js:14883
    at app.f3e519c….js:15484
    at Object.<anonymous> (app.f3e519c….js:15669)
    at Object.VERSION (app.f3e519c….js:15671)
    at __webpack_require__ (app.f3e519c….js:20)
    at Object.<anonymous> (app.f3e519c….js:12095)
    at Object.<anonymous> (app.f3e519c….js:12129)
    at __webpack_require__ (app.f3e519c….js:20)
    at Object.<anonymous> (app.f3e519c….js:11192)
    at __webpack_require__ (app.f3e519c….js:20)

Fortunately there’s a pretty easy fix.

Solution

Laravel mix has an autoload() method that requires modules wherever they are needed.

To fix the Tether issue, add this to webpack.mix.js:

mix.autoload({
    tether: ['Tether', 'window.Tether']
});

This requires tether wherever either the variable Tether or window.Tether is referenced (in this case, by Bootstrap). I think Bootstrap 4 uses the latter.

References

Laravel Mix Autoloading

Managing Dependencies in a Laravel Package

Sun, 26 Mar 2017 15:22:00 +0100

When developing a custom Laravel package, you may need to require a separate package to provide functionality.

Composer makes it easy to include packages as a dependency. This is a good thing as it makes code more maintainable and means to don’t have to re-invent the wheel.

You also need to bind the third-party package to the Laravel app - you shouldn’t do this manually, as it would make using your package unecessarily complicated. Fortunately, Laravel makes it easy for you to register packages from within your custom package’s service provider class.

Use Composer to Bundle Dependencies

In this example, we’ll add the “illuminate/support” and “laravelcollective/html” packages. In your custom package composer.json:

{
    "require": {
        "illuminate/support": "5.4.*",
        "laravelcollective/html":"^5.4.0"
    }
}

When you run composer update at the app level, Composer is smart enough to connect the dots and add the required packages to the app vendor directory.

Register Services

Within the boot() method of the custom packages service provider, you can register (bind) the “laravelcollective/html” service provider. You can also load up an alias for the package:

<?php

namespace Carawebs\Blog;

use Illuminate\Foundation\AliasLoader;
use Illuminate\Support\ServiceProvider;
use Collective\Html\HtmlServiceProvider;

/**
 * Bind to the Laravel service container.
 * This class should be included in the Laravel app's
 * `config/app.php` 'providers' array.
 */
class BlogProvider extends ServiceProvider
{
    /**
    * Bootstrap the application services.
    *
    * @return void
    */
    public function boot()
    {
        // Register the HtmlServiceProvider class - like you'd generally do in the
        // `config/app.php` 'providers' array.
        $this->app->register(HtmlServiceProvider::class);
        $loader = AliasLoader::getInstance();
        $loader->alias('Form', '\Collective\Html\FormFacade');

        // Unrelated
        $this->loadRoutesFrom(__DIR__.'/routes.php');
        $this->loadMigrationsFrom(__DIR__.'/migrations');
        $this->loadViewsFrom(__DIR__.'/views', 'blog');
    }
}

You could simplify the alias loading by passing in an array of alias => classname values to the getInstance() method:

<?php
// ...
AliasLoader::getInstance(['Form'=>'\Collective\Html\FormFacade']);

This method acceps an array and merges this with the registered aliases.

References

See $this->app->register() method on GitHub
See AliasLoader::alias() method on GitHub

Manage WordPress Plugins Using Composer

Fri, 24 Mar 2017 11:06:00 +0000

This article shows how to use Composer to manage project plugin dependencies.

The example given is for a Bedrock project. Bedrock is an improved WordPress project structure from the Roots team that manages dependencies with Composer.

Manage Plugins from The WordPress Plugin Directory

You can easily access any plugin that’s held in the WordPress Plugin Directory by means of the WordPress Packagist Composer repository.

WordPress Packagist mirrors the WordPress plugin and theme directories. If you add the wpackagist repository to Composer’s “repositories”, you can then require the ACF free version.

In Bedrock’s composer.json add the wp-packagist repo:

{

    "repositories": [
        {
            "type": "composer",
            "url": "https://wpackagist.org"
        }
    ]

}

Then in the same file, require the plugin:

{
    "require": {
        "php": ">=5.6",
        "composer/installers": "~1.0.12",
        "vlucas/phpdotenv": "^2.0.1",
        "johnpbloch/wordpress": "4.7.2",
        "oscarotero/env": "^1.0",
        "roots/wp-password-bcrypt": "1.0.0",
        "roots/soil": "3.7.1",
        "wpackagist-plugin/advanced-custom-fields" : "4.1.*"
    }
}

Run composer update and it should fetch the plugin.

Custom Plugins: Not in the WordPress Directory

We use a lot of custom plugins - the easiest way to manage these is to:

Push your code to a public VCS (e.g. git) repo (GitHub, BitBucket)
Set them up as a Composer package on Packagist
In the plugin’s composer.json, make sure you set "type": "wordpress-plugin"
Require the plugin in the project composer.json

Here’s an example from a Bedrock project composer.json - this requires the carawebs/wp-custom-content Composer package (which is a WordPress plugin that sets up custom post types from a config array):

{
    "require": {
        "php": ">=5.6",
        "composer/installers": "~1.0.12",
        "vlucas/phpdotenv": "^2.0.1",
        "johnpbloch/wordpress": "4.7.3",
        "oscarotero/env": "^1.0",
        "roots/wp-password-bcrypt": "1.0.0",
        "carawebs/wp-custom-content": "dev-master"
    }
}

Running composer update then instructs Composer to download the plugin to the appropriate directory - in the case of Bedrock, /web/app/plugins. You can also take advantage of Composer autoloading - in the plugin composer.json set the appropriate directory:

{
    "autoload": {
        "psr-4": {
            "Carawebs\\CustomContent\\": "src"
        }
    }
}

Assuming that you’re using Composer autoloading, you’ll now be able to access your plugin’s namespaced classes using the namespace you’ve defined (in the example above, Carawebs\CustomContent\ClassName).

Access Specific Commits

You can request a specific tagged commit - so in the example above, you could create a git tag 1.0.0 and specifically require this in the project composer.json:

{
    "require": {
        "carawebs/wp-custom-content": "1.0.0"
    }
}

You can also access branches - using the branch name as a suffix on ‘dev-‘ like so:

{
    "require": {
        "carawebs/wp-custom-content": "dev-experimental-branch"
    }
}

This can be useful if you want to build functionality as a pure Composer library rather than a plugin - you can keep this code in a ‘library’ branch on your git repo. This might be useful if you needed to include your functionality in another plugin.

Filtering the WordPress Menu

Fri, 24 Mar 2017 10:15:00 +0000

You can filter both the <li> tag and the contained anchor tag in a WordPress menu using the ‘nav_menu_css_class’ and ‘nav_menu_link_attributes’ filters.

The ‘nav_menu_link_attributes’ and ‘nav_menu_css_class’ filters are not well documented - but very useful nonetheless. You can see the filters in wp-includes/class-walker-nav-menu.php on line 136 and 179.

This example shows how to add the required classes for a Bootstrap 4 menu markup - in which the <li> requires the class nav-item and the anchor tag requires the class nav-link:

<?php
add_filter( 'nav_menu_css_class', function($classes) {
    $classes[] = 'nav-item';
    return $classes;
}, 10, 1 );

add_filter( 'nav_menu_link_attributes', function($atts) {
        $atts['class'] = "nav-link";
        return $atts;
}, 100, 1 );

Initial Workflow for Composer Package Development

Thu, 16 Mar 2017 19:40:00 +0000

During the initial stages of developing a Composer package, you probably don’t want to go to the trouble of setting up an empty Packagist package. You probably also shouldn’t be developing in the vendor directory.

I haven’t done much research on the best workflow for this solution - but the method described in this article, which utilises a Composer path repository - seems to work pretty well. I heard Taylor Otwell (creator of Laravel) talk at Laracon Online last week and he uses this approach.

Initial Setup

In your project root directory, create a ‘packages’ directory. This in turn should contain a ‘vendor-name’ directory which shouild contain a ‘package-name’ directory:

cd /path/to/project/root
mkdir -p packages/vendor-name/project-name

I think it’s best to name directories according to Packagist naming conventions.

Composer Init in the New Project

Move into your project and run composer init:

cd packages/vendor-name/project-name
composer init

Composer will prompt you to set important metadata like the package name. This will be needed later when Composer (at the parent project level) attempts to auto-load your new project files.

Set Up Path Repository

In the main project composer.json, you need to create a path reference for your package within the “repositories” section. The following fragment from the parent project composer.json defines two Composer path repositories, sage-blade-data and wp-metadata-accessor. In this particular case, the parent project is cloned from the Sage WordPress starter theme - which comes already set-up for Composer.

{
    "repositories": [
        {
            "type": "path",
            "url": "./packages/carawebs/sage-blade-data"
        },
        {
            "type": "path",
            "url": "./packages/carawebs/wp-metadata-accessor"
        }
    ]
}

Require Local Packages in composer.json

Once the repositories have been set up with the correct path, you can add them to the project dependencies, as shown in this extract:

{
    "require": {
        "php": ">=5.6.4",
        "composer/installers": "~1.0",
        "illuminate/view": "~5.4.0",
        "illuminate/config": "~5.4.0",
        "carawebs/sage-blade-data": "dev-master",
        "carawebs/wp-metadata-accessor": "dev-master"
    }
}

Important: if you have a local repo without releases (maybe you haven’t even run git init yet), you need to specify "dev-master" for the required local packages. "*" won’t work.

Autoloading

Within the new project composer.json, set up autoloading paths:

{
    "name": "carawebs/sage-blade-data",
    "description": "Data controllers for Sage 9 WordPress theme - insert data into Blade templates.",
    "type": "library",
    "license": "GPL",
    "authors": [
        {
            "name": "David Egan",
            "email": "david@carawebs.com"
        }
    ],
    "autoload": {
        "psr-4": {
            "Carawebs\\SageBladeData\\": ""
        }
    },
    "require": {}
}

This sets the new project’s namespace. Once you’ve run composer update, your project classes will be autoloaded.

Composer Update Connects (Symlinks) Things Up

Run:

composer update

In the parent project root directory. Composer will then create a new vendor-name directory with the project vendor directory. The package files will be symlinked here. For example:

cd /path/parent-project/vendor/carawebs

david@david-desktop:/path/parent-project/vendor/carawebs$ ls -la
total 8
drwxrwxr-x  2 david david 4096 Mar 16 18:27 .
drwxrwxr-x 11 david david 4096 Mar 16 18:27 ..
lrwxrwxrwx  1 david david   39 Mar 16 15:21 sage-blade-data -> ../../packages/carawebs/sage-blade-data
lrwxrwxrwx  1 david david   44 Mar 16 18:27 wp-metadata-accessor -> ../../packages/carawebs/wp-metadata-accessor

This allows you to work on your new package from within your packages directory, but still have your project included within Composer (with dependencies, autoloading, etc).

Composer Update Problems

I recently hit a snag when setting up a package for development in a Laravel project:

Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package carawebs/repositories dev-master exists as carawebs/repositories[dev-repo-pattern] but these are rejected by your constraint.

This happens because if the path-repository project that you’re requiring is not a VCS repository, you need to explicitly define the version in the package composer.json file:

If the package is a local VCS repository, the version may be inferred by the branch or tag that is currently checked out. Otherwise, the version should be explicitly defined in the package’s composer.json file. If the version cannot be resolved by these means, it is assumed to be dev-master.

— Composer documentation on path repositories.

In my case, I solved the issue simply by running git init in the package directory.

References

Composer docs on path repository
SO Answer outlines the “dev-master” issue

Arrow Functions in ECMAScript 6

Tue, 14 Mar 2017 21:47:00 +0000

ES6 introduces a new syntax for writing JavaScript functions: arrow functions. Arrow function expressions have more concise syntax than a function expression.

Arrow functions also allow you to access this from a surrounding method.

Arrow functions can have either a “concise body” or a “block body” form:

var func = x => x * x;                  // concise syntax, implied "return"
var func = (x, y) => { return x + y; }; // with block body, explicit "return" needed

Example from MDN Article on Arrow Functions by Mozilla Contributors, licensed under CC-BY-SA 2.5.

These are equivalent:

// ES5
function calc(one, two) {
    return one + two
}

// ES5
var calc2 = function(one, two) {
    return one + two
}

// ES6 Arrow syntax
var arrowCalc = (one, two) => {
    return one + two
}

// ES6 Short syntax Arrow - note no 'return' keyword, braces
var shortArrow = (one, two) => one + two

These are also equivalent:

var words = [
  'cat',
  'dog',
  'elephant',
  'crow'
];

var wordsLength1 = words.map(function(word) {
  return word.length;
});

var wordsLength2 = words.map((word) => {
  return word.length;
});

// If a single parameter is being passed in as an identifier, you can omit parentheses
// If an expression follows the arrow it will be returned - no return keyword necessary/allowed
var wordsLength3 = words.map(word => word.length);

Example adapted from MDN Article on Arrow Functions by Mozilla Contributors, licensed under CC-BY-SA 2.5.

Implicit Return

Implicit return only happens for single statement arrow functions. If an arrow function is declared with {}, even if it’s a single statement, implicit return does not occur.

Returning Object Literals

Wrap the object literal in parentheses - code inside {} is parsed as a sequence of statements.

Function that returns an object:

var myObject = () => ({
    firstName: "John",
    lastName: 'McLane'
});

Line Breaks

Line breaks between parameters and the arrow are not allowed.

References

MDN Article on Arrow Functions - good reference article
Good description of Arrow Funcions - outlines lexical variables in arrow functions

Webpack Configuration for JavaScript, CSS and Babel

Sun, 12 Mar 2017 19:55:00 +0000

Notes on configuring and using Webpack 2.x for JavaScript and CSS. This article includes a description of the use of Babel, so that next generation JavaScript can be used whilst maintaining browser compatibility.

These are personal notes that I’m making these as I learn Webpack. I favour Yarn over NPM, and the notes reflect this. You could always use NPM to manage dependencies.

Why Webpack?

Webpack is a tool that bundles modules (files) for your application. It can use loaders to transform non JS resources (like CSS) into JavaScript. Webpack has a rich plugin system. Webpack facilitates code-splitting, which aids in project organisation. In short, it is a very versatile build tool for all your front-end assets.

There are four core concepts that underpin webpack:

Entry: Once an entry point is defined, webpack can work out dependencies and what to bundle.
Output: Define the name and location of the bundle to be emitted.
Loaders: In webpack, files are treated as modules - but webpack only understands JavaScript. Loaders transform resource files so that webpack can use them.
Plugins: Extensible functionality by means of a powerful & customizable plugin system.

Install Webpack in Project

This command installs webpack as a project dev dependency.

cd /path/my-project
# If necessary, initialise a package
yarn init

# Install Webpack
yarn add webpack --dev

You should now see webpack listed in the devDependencies section of your package.json.

Run From CLI

To see Webpack in basic operation, create a file src/main.js in your project root. Run:

webpack src/main.js dist/bundle.js

Webpack will compile src/main.js down to dist/bundle.js, which then can be referenced in your HTML.

Define Script in package.json

For convenience, you could define a webpack command in the scripts section of package.json:

{
  "scripts": {
    "build": "webpack src/main.js dist/bundle.js"
  }
}

Running yarn build will now trigger the webpack task.

Reference Extra Files

In Node.js, there are two module systems that you can choose to import/export functionality between files:

Import modules using require and export using module.exports
Import using ES6 import and export using ES6 export

If you add additional JavaScript files, you can reference them in the entry-point JS file like this (CommonJS):

// src/main.js
alert(require("./additional.js"));

// src/additional.js
module.exports = "This is additional.js"

Or like this (ES6):

// src/main.js
import extra from './additional.js';
alert(extra);

// src/additional.js
export default "This is additional.js"

Loaders

To add CSS to the application, webpack needs to load the stylesheet in the entry JavaScript file. Because webpack can only handle JavaScript, a “Loader” is needed to process CSS. In this case, the required loader is css-loader.

To apply the stylesheet, processing the CSS isn’t enough - the resultant style rules need to be injected inline. This is achieved with the style-loader Loader.

To install both loaders in the project using Yarn:

yarn add css-loader style-loader --dev

Using Loaders

You can call loaders “inline” when requiring modules in the entry point JavaScript file:

require("!style-loader!css-loader!./main.css");

This shows the syntax for applying loaders within the require statement. As we’ll see, it’s easier to define loaders in a config file.

Be careful that you’re not looking at the docs for webpack 1.x: the old syntax does not require the -loader suffix.

Note that the loaders are applied in a right-to-left order - and the order is important. In this case, the processed CSS must be piped to the style loader.

Apparently you can use loaders through the webpack CLI - but I wasn’t able to get this working on Ubuntu 16.04.

Webpack Config

Webpack configuration can be defined in a configuration file. This drastically simplifies the entry-point JavaScript and makes code more concise and maintainable.

The webpack config file is a JS file that exports an object. The object is processed by webpack based on the defined properties. The config file is a standard Node.js CommonJS module.

The module.rulesproperty of webpack.config.js allows you to specify loaders for particular file types:

// webpack.config.js
// Configure CSS processing & injection
var webpack = require('webpack');
var path = require('path');

module.exports = {
    entry: './src/main.js',
    output: {
        // Webpack prefers an absolute path:
        path: path.resolve(__dirname, './dist'),
        filename: 'bundle.js'
    },
    module: {
        rules: [
            {
                // Uses regex to test for a file type - in this case, ends with `.css`
                test: /\.css$/,
                // Apply these loaders if test returns true
                use: ['style-loader', 'css-loader']
            }
        ]
    }
}

Note that the order in which loaders is applied is important - the right-most loader is applied first.

Write ES2015/ES6 - For All Browsers

Use next gen JavaScript today - using Babel

First, pull in the Babel node module:

yarn add babel-loader babel-core --dev

You also need to configure Babel - you do this by means of a preset plugin and a .babelrc config. Install the ES2015 plugin:

yarn add babel-preset-es2015 --dev

…then create .babelrc in the project root, with the following content:

{
  "presets": ["es2015"]
}

Finally, amend the webpack.config.js - tell webpack to apply the configured Babel loader to all JavaScript files, excluding those in node_modules:

module: {
        rules: [
            {
                test: /\.css$/,
                use: ['style-loader', 'css-loader']
            },
            {
                test: /\.js$/,
                exclude: /node_modules/,
                loader: 'babel-loader'
            }
        ]
    }

References

Laracasts - Webpack Video Tutorials Highly recommended intro to Webpack
Webpack 2.x Get Started Guide Good intro to Webpack by Drew Powers
webpack 2.x Loaders: Official Docs
‘export’ statement
ECMAScript 6 Modules

Reduce a Numerically Indexed Array to a Simple Array in PHP

Sun, 12 Mar 2017 11:45:00 +0000

Reducing a numerically indexed multi-dimensional array to a simple array can be achieved easily by means of combining array_merge() with the ‘splat’ (...) operator.

This uses the concept of argument unpacking. It can be really useful if you are starting with a numerically indexed array of arrays and you need to combine them into a single simple array of key-value pairs.

Starting Point

<?php
// The Array
$array = [
    ['key1' => 'val1', 'key2'=>'val2'],
    ['key3' => 'val3'],
    ['key4' => 'val4'],
    ['key5' => [
        'A' => 'foo',
        'B' => 'bar'
        ]
    ]
];

The output of var_dump($array) looks like this:

array (size=4)
  0 =>
    array (size=2)
      'key1' => string 'val1' (length=4)
      'key2' => string 'val2' (length=4)
  1 =>
    array (size=1)
      'key3' => string 'val3' (length=4)
  2 =>
    array (size=1)
      'key4' => string 'val4' (length=4)
  3 =>
    array (size=1)
      'key5' =>
        array (size=2)
          'A' => string 'foo' (length=3)
          'B' => string 'bar' (length=3)

Operation

Assuming that you’re using PHP version 5.6 and upwwards (and you should be), you can run the array through array_merge() using the ‘splat’ operator:

<?php
$newArray = array_merge(...$array));

As I understand it (please correct me via a comment if I have this wrong):

array_merge() accepts an initial array along with a variable list of arrays to merge.
Using the ‘splat’ operator tells PHP to unpack the input array into variables.
In our case, these variables are arrays, because the staring point was an array of arrays - so we’re passing multiple arrays into array_merge() for processing.
Because each variable is a singular array, they’re no longer numerically indexed.
array_merge() will merge the multiple arrays, and the output array is as associative array, indexed by the keys.

Result

The output of array_merge(...$array)); looks like this:

array (size=5)
  'key1' => string 'val1' (length=4)
  'key2' => string 'val2' (length=4)
  'key3' => string 'val3' (length=4)
  'key4' => string 'val4' (length=4)
  'key5' =>
    array (size=2)
      'A' => string 'foo' (length=3)
      'B' => string 'bar' (length=3)

Aside: Why Splat?

I don’t know if unpacking arguments is technically an “operation”, so ... may not really be an operator, and it doesn’t look like something that’s gone splat. I’ve read that the symbol for this is in Ruby “*”, and this is where the term originated.

References

PHP 5.6 and the ‘splat’ operator

Using Composer to Manage Specific Version of a Package in a Private Repo

Tue, 28 Feb 2017 20:15:00 +0000

According to their website, Composer is a dependency manager for PHP. While true, I think this is a bit of an understatement. I’m fairly new to Composer, but it’s revolutionising my workflow.

Package in Private Repo

You may need to keep your work in a private VCS repo while it’s still in the early stages of development. It’s easy to set up git repos as public packages on Packagist, the PHP package repository - but if you need to keep work private you can set Composer up to use a private repo. Here’s a simple example composer.json for such a use case:

{
    "name": "project-vendor-name/project-name",
    "description": "Your description",
    "type": "library",
    "license": "MIT",
    "authors": [
        {
            "name": "Joe Bloggs",
            "email": "joe@example.com"
        }
    ],
    "repositories": [
        {
            "type": "vcs",
            "url": "git@bitbucket.org:vendor-name/private-project.git"
        }
    ],
    "require": {
        "vendor-name/private-project": "dev-master"
    }
}

This assumes that you’re accessing your private repo by means of SSH keys.

Use a Specific Tag or Commit

To lock your project to a specific commit, add a hash to dev-master followed by a git tag or commit reference:

"vendor-name/private-project": "dev-master#1.0.5"

Now when you run composer update, your project remains locked to a specific version (in this case, tag 1.0.5) of the required package.

Caveat

In the long run, you’re better off creating public Composer packages from your work:

Note: This feature has severe technical limitations, as the composer.json metadata will still be read from the branch name you specify before the hash. You should therefore only use this as a temporary solution during development to remediate transient issues, until you can switch to tagged releases. The Composer team does not actively support this feature and will not accept bug reports related to it. https://getcomposer.org/doc/04-schema.md#package-links

References

Composer Package Links, Docs

Let's Encrypt & Apache on Ubuntu Xenial

Sat, 25 Feb 2017 11:37:00 +0000

Let’s Encrypt is a free, automated, and open Certificate Authority. The service is provided by the Internet Security Research Group.

If you’re managing a Virtual Private Server and you want to enable encrypted connection to websites by means of HTTPS(SSL/TLS), Let’s Encrypt is a good choice.

Let’s Encrypt Client/Certbot

Certbot is the name of the client that deploys and manages Let’s Encrypt certificates on your server. Confusingly, depending on your system and the version you install, the client may have a different name.

Certbot on Ubuntu 16.04 Xenial

The package is called python-letsencrypt-apache. To install:

sudo apt-get install python-letsencrypt-apache

This installs Certbot - but you’ll be using the command letsencrypt.

Set Up a New Cert: Apache

Certbot/Letsencrypt has an Apache plugin that sets up the required Apache configuration. To run:

sudo letsencrypt --apache

This triggers a ncurses dialog that prompts you to select the domains for which you’re enabling SSL. You’ll also be prompted to choose whether or not all resources should be served over HTTPS, or if the site should serve mixed content.

As part of the cert setup, existing config files in /etc/apache2/sites-available are used to set up new configurations for port 443 - and a rewrite rule is added to the exisiting config for port 80 (HTTP traffic). Depending on the selections you make during install, the rewrite rule redirects HTTP traffic to HTTPS.

I have noticed one glitch in this process. If the existing config file already contains a rewrite rule, the amended config does not include the necessary RewriteEngine on statement to actually activate the rewrite. You can see an example of this here:

<VirtualHost *:80>
        ServerName example.com
        ServerAlias www.example.com
        ServerAdmin info@example.com
        DocumentRoot /var/www/html/example.com/web
        <Directory /var/www/html/example.com/web>
                Options -Indexes +FollowSymLinks
                AllowOverride None
                Require all granted
        </Directory>
        <Files "wp-login.php">
                Require ip X.X.X.X
        </Files>
        <Directory /var/www/html/example.com/path/to/admin-dir>
                AllowOverride None
                Require ip X.X.X.X
        </Directory>
        # WordPress Config
        <Directory "/var/www/html/example.com/web">
                <IfModule mod_rewrite.c>
                RewriteEngine On
                RewriteBase /
                RewriteRule ^index\.php$ - [L]
                RewriteCond %{REQUEST_FILENAME} !-f
                RewriteCond %{REQUEST_FILENAME} !-d
                RewriteRule . /index.php [L]
                </IfModule>
        </Directory>
        ErrorLog ${APACHE_LOG_DIR}/example.com.error.log
        CustomLog ${APACHE_LOG_DIR}/example.com.access.log combined
RewriteEngine on # This line needed to be added manually
# Rewrite rule added by Letsencrypt - in this case, all traffic on
# this port is sent to the equivalent HTTPS.
RewriteCond %{SERVER_NAME} =example.com [OR]
RewriteCond %{SERVER_NAME} =www.example.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>

This isn’t a disaster, but it does mean that you’ll either need to amend the config (and reload the Apache configuration) or amend the root URL in your app (or database) to reference the HTTPS address.

I always amend the app URL in any case - this means that traffic will be sent straight to HTTPS rather than being redirected.

Renew Certificate

Test renewal:

letsencrypt renew --dry-run --agree-tos

If this works, run:

letsencrypt renew --agree-tos

let’s Encrypt suggest running this script twice per day as a cron job on a random minute in the hour. Certificates are valid for 90 days, and renewal won’t proceed unless the certificate is nearing it’s expiry date.

Add a New Site to an Existing Certificate

To achieve this, run:

sudo letsencrypt --apache

…and follow the on-screen instructions. All sites on the server will be listed, and pre-selected for inclusion in the cert. You are then given an option to expand the existing certificate to include the new site. Agree to this and the new site will be added. Don’t forget to check the port 80 Apache config file if redirects are not working.

Common Name

You may end up with the wrong domain name as the common name attached to the certificate. You can force this by running:

sudo letsencrypt --apache -d primary-domain.com -d www.primary-domain.com -d secondary-domain.com -d www.secondary-domain.com --force-renewal

This will result in a certifcate with the common name set to the first domain (in this case, primary-domain.com).

Moving Servers: Client Confusion!

During a recent server upgrade, I moved Let’s Encrypt certificates to the new server. Description here. This is a fairly straightforward process, and allows sites to be migrated without downtime.

However, I experienced a problem when renewing the Certificate.

The old server (Ubuntu 14.04) was running a version of Certbot cloned from the GitHub repo (which is actually a pretty good option - it self updates every time you run it). On the new server (Ubuntu 16.04), I followed the Certbot install instructions for Ubuntu 16.04 as outlined here. This caused problems because the config file (created on the original server) was actually created by a newer version of Certbot than that found in the Ubuntu repos for 16.04. The OLD server had newer software than the NEW server.

The error:

KeyError: 'server'

The fix: You could revoke the cert, remove the client, install Certbot as before and re-install the certificates. In other words, take off and nuke it from orbit (I always wanted to use that expression in a blog post). Alternatively append the following line to your renewal config files (see: /etc/letsencrypt/renewal):

server = https://acme-v01.api.letsencrypt.org/directory

If you have a [[webroot_map]] entry in the renewal config file, add the server line above that.

Kudos to Brad Warren for sharing this solution (ref below) in the Certbot issue tracker.

Remove Cert: The Nuclear Solution

Move into /etc/letsencrypt and remove everything apart from the accounts directory:

cd /etc/letsencrypt
sudo rm -rf archive csr keys live options-ssl-apache.conf renewal

Disable the SSL enabled sites by running:

sudo a2dissite example.com-le-ssl.conf
# ...

Remove all Apache ssl config files from /etc/apache2/sites-available and reload Apache:

cd /etc/apache2/sites-available
sudo rm example.com-le-ssl.conf # etc
sudo service apache2 reload

Note: Unless you’ve removed the redirect in the port 80 virtual host directive, and/or updated your app’s home URL, your site will now be broken. Take care. You should temporarily switch the site to HTTP whilst re-issuing the certificate - I’m assuming you know how to do that.

Run:

sudo letsencrypt --apache

This will reissue the certificate.

This is very much a nuclear option - there are better ways to use the letsencrypt client to amend certificates and settings.

References

Certbot Issue, GitHub: Error with renewal conf files that no longer have a “server” entry

Connect Bitbucket with Slack to Receive Notifications

Mon, 30 Jan 2017 09:40:00 +0000

Receive Slack notifications when activity occurs on a Bitbucket repo. This is a useful way for team members to monitor progress.

This assumes that you’ve added the Bitbucket app to Slack.

Slack

Go to https://YOURTEAM.slack.com/apps/manage
Select Bitbucket
“Add Configuration”
Choose channel, select “Add Bitbucket Integration”
Copy the provided webhook - this will be required in the steps outlined below

Bitbucket

Go to repo
Select “Settings” in left navigation
“Workflow” section, click “Add webhook” button
Give the webhook a unique name and paste in the hook generated by Slack as described above
Select “Choose from a full list of triggers” for additional utility

Gitignore Directory Contents

Wed, 25 Jan 2017 16:00:00 +0000

You may need to gitignore the contents of a directory, whilst including the empty directory in version control.

For example, you have a config directory that contains config files that are app specific. It may useful to include the config directory, so you can easily see where to place config files.

To achive this, add a .gitkeep file to the directory. Then in the repository’s .gitignore file:

config/*
# exception to the rule
!config/.gitkeep

MixItUp JavaScript Filter with Bootstrap Dropdown Toggle

Wed, 25 Jan 2017 14:39:00 +0000

MixItUp is a dependency-free JavaScript library for animated filtering, sorting, insertion, removal etc:

MixItUp is a pretty good library - easy to build nice filters. You need to buy the commercial license if you’re using it on a commercial project.

Bootstrap Conflict

There is a slight conflict with Bootstrap 3 dropdown elements. If you’re using a Bootstrap dropdown to display a list of filter elements, a MixItUp control is triggered when you click the dropdown. The data-toggle of the Bootstrap dropdown interferes wityh MixItUp This will probably hide all the elements that you’re trying to filter, as this element won’t have the required data-filter attribute.

Solution

For MixItUp >= 3.1, the solution is to specify the control element in the configuration object:

mixitup(container, {
    selectors: {
        control: '[data-mixitup-control]'
    }
}

… you then add the data-mixitup-control attribute to the filter controls.

JSFiddle

Resources

GitHub issue with solution

Apache Directives in Config vs .htaccess

Tue, 10 Jan 2017 14:53:00 +0000

Apache allows configuration rules to be overridden at the directory level by means of distributed configuration files. These files are usually called .htaccess, but the file name can be customised. Directives placed in .htaccess apply to the directory that contains the file, and changes to .htaccess have immediate effect for that directory. However, this can be wasteful of resources - .htaccess files are read on every request.

In general, you should only use .htaccess files when you don’t have access to the main server configuration file…

….htaccess files should be used in a case where the content providers need to make configuration changes to the server on a per-directory basis, but do not have root access on the server system…

…However, in general, use of .htaccess files should be avoided when possible. Any configuration that you would consider putting in a .htaccess file, can just as effectively be made in a section in your main server configuration file.

— Apache Docs, .htaccess

If you’re managing sites on a VPS, there’s no need to use distributed configuration files. I think I personally became conditioned to them from managing sites on shared hosting, with no root access.

Because of this, I’ve been moving Apache rewrite rules and access restrictions out of .htaccess files and into virtual host configs.

Global Rules

In some cases, the rules that might have gone into a .htaccess file can be applied at a server level - for example, preventing access to .git directories and restricting access to config files (e.g. wp-config.php) is probably best achieved by adding appropriate rules to /etc/apache2/conf-enabled/security.conf (the Ubuntu apache2 security config file). If you’re blocking access to these files in one directory, the chances are that you’ll want to apply this to the whole server.

Per-Directory Rules: WordPress

The normal WordPress rewrite block can be easily added to a virtual host config. This allows you to disallow overriding at the directory level which speeds things up.

The following sections contrast how to achieve the same effect with a .htaccess file and a virtual host configuration. In this case, a set of WordPress rewrites are needed, along with some restrictions based on IP address.

Typical WordPress .htaccess File: Production Site

The .htaccess file is placed in the site document root. If the file is writeable by WordPress, WP will generate the config block between # BEGIN WordPress and # END WordPress.

The following .htaccess file has an additional rule that restricts access to the main site login by IP.

# ==============================================================================
# Restrict access to WordPress login page by IP
# See: http://httpd.apache.org/docs/2.4/mod/core.html#files
# ==============================================================================
<Files "wp-login.php">
    Require ip x.x.x.x y.y.y.y
</Files>

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

Apache vHost Configuration with WordPress Specific Rules

To achieve the same thing without use of .htaccess, amend the site virtual host config file:

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerName example.com
        ServerAlias www.example.com
        ServerAdmin info@example.com
        DocumentRoot /var/www/html/example.com/web
        # Disallow overrides (.htaccess) in the document root
        <Directory /var/www/html/example.com/web>
                Options -Indexes +FollowSymLinks
                AllowOverride None
                Require all granted
        </Directory>
        # Restrict access to wp-login.php - space separated list of allowed IPs
        <Files "wp-login.php">
                Require ip x.x.x.x y.y.y.y
        </Files>
        # Restrict access to the site admin area by IP
        <Directory /var/www/html/example.com/web/wp/wp-admin>
                AllowOverride None
                Require ip x.x.x.x y.y.y.y
        </Directory>
        # WordPress Config - this must be placed in a <Directory> block
        <Directory "/var/www/html/example.com/web">
                <IfModule mod_rewrite.c>
                RewriteEngine On
                RewriteBase /
                RewriteRule ^index\.php$ - [L]
                RewriteCond %{REQUEST_FILENAME} !-f
                RewriteCond %{REQUEST_FILENAME} !-d
                RewriteRule . /index.php [L]
                </IfModule>
        </Directory>
        # Set custom logs
        ErrorLog ${APACHE_LOG_DIR}/example.com.error.log
        CustomLog ${APACHE_LOG_DIR}/example.com.access.log combined
        # SSL cert details (LetsEncrypt)
        SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

Typical .htaccess Rules for Private Staging Site

This .htaccess file is for a site that is in a sub-directory of the domain document root. For example: testdomain.com/testsite. This is often a convenient way of handling staging websites.

In this case, the main Apache vHost configuration for the “parent” domain needs to allow override by means of .htaccess. This is achieved by setting the AllowOverride directive to “All” in the virtual host config file. When this directive is set to None and AllowOverrideList is set to None, .htaccess files are completely ignored. In this case, the server will not even attempt to read .htaccess files in the filesystem.

# ==============================================================================
# Full block - private. This is a staging site!
# ==============================================================================
<RequireAll>
    Require ip x.x.x.x y.y.y.y
</RequireAll>

# ==============================================================================
# Restrict access to WordPress login page by IP
# See: http://httpd.apache.org/docs/2.4/mod/core.html#files
# ==============================================================================
<Files "wp-login.php">
    Require ip x.x.x.x y.y.y.y
</Files>

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /testsite/
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /testsite/index.php [L]
</IfModule>
# END WordPress

Reload Apache Configs

If you move your local config into virtual host configs, any changes you make will not take effect until you reload Apache:

sudo /etc/init.d/apache2 reload

# or use `force-reload`, an alias for the same command:
sudo /etc/init.d/apache2 force-reload

Convert ACF Fields Registered by PHP to Importable JSON Format

Sat, 07 Jan 2017 18:39:00 +0000

When using the WordPress Advanced Custom Fields plugin, you may need to switch between PHP registered custom fields and JSON registered fields.

Use Case

In older projects we registered ACF fields by importing to PHP. This is good for the deployed site - the fields load quicker, and users can’t amend the fields.

This can be problematic if you import a production database into a local development environment. The fields are not registered in the production database. When you import it, you no longer have access to the fields in the ACF GUI. The fields still exist (they are registered by PHP functions), but you can’t use the nifty ACF interface to edit them.

Because of this we have switched to ACFs local JSON option. ACF writes the field registration data in JSON format to a server-writable directory (acf-json, unless you override the default) in the theme root. You end up with a separate JSON file for each field group. The JSON configuration files can go into your version control system. You can deploy to staging or production, and if you accidentally delete files in your dev environment (for example, if you import a database) you can easily re-sync the JSON files to have them show up in the ACF GUI again.

If you have your fields registered by PHP (not in the database), how do you switch to the local JSON workflow?

Create a JSON configuration file that can be read by the ACF “Import” function - i.e. mimic the output of the ACF “Export” function
Create acf-json in your theme root and run sudo chown www-data acf-json
Import the JSON file to ACF

Creating the JSON File for Import

The following quick and dirty method is suitable for a local development environment:

Drop the code below into a suitable template
Create a suitable config file - touch acf-import.json and make it server-writable
Hit the page template in your browser - you’ll see the JSON data output, and this will be written to the config file
Import the config file: within the ACF “Tools” > “Import Field Groups” interface, select the file and click import

The field configurations are now held in the local database, and will be saved in the acf-json directory.

<?php

$groups = acf_get_local_field_groups();
$json = [];

foreach ($groups as $group) {
    // Fetch the fields for the given group key
    $fields = acf_get_local_fields($group['key']);

    // Remove unecessary key value pair with key "ID"
    unset($group['ID']);

    // Add the fields as an array to the group
    $group['fields'] = $fields;

    // Add this group to the main array
    $json[] = $group;
}

$json = json_encode($json, JSON_PRETTY_PRINT);
// Optional - echo the JSON data to the page
echo "<pre>";
echo $json;
echo "</pre>";

// Write output to file for easy import into ACF.
// The file must be writable by the server process. In this case, the file is located in
// the current theme directory.
$file = get_template_directory() . '/acf-import.json';
file_put_contents($file, $json );

?>

Jekyll Deploy Methodology

Fri, 30 Dec 2016 15:02:00 +0000

This post outlines a simple and secure method for deploying a Jekyll site to a server/VPS by means of SSH using Grunt and Bash.

The objective is to deploy a website by means of a Grunt task without exposing senstive information (SSH port, server IP address, username etc) in the Gruntfile. This allows the Gruntfile to be added to your VCS.

This is achieved by running a custom Bash script, which is executed from within Grunt.

Setting up Grunt Exec

The grunt-exec plugin allows shell commands to be executed from within Grunt.

Install grunt-exec with Yarn:

yarn add grunt-exec --dev

…or install grunt-exec with NPM:

npm install grunt-exec --save-dev

The configuration for grunt-exec might look like this:

/* /grunt-config/exec.js */
module.exports = {
  build: {
    cmd: 'bundle exec jekyll build'
  },
  dev: {
    cmd: 'bundle exec jekyll build --watch --config _config.yml,_config_dev.yml --drafts'
  },
  serve: {
    cmd: 'bundle exec jekyll serve --watch'
  },
  deploy: {
    cmd: 'deploy-my-website'
  }
};

Note that this config is separated out into a separate file, which requires the load-grunt-configs plugin.

The Gruntfile

Your deploy tasks can now be defined in the project Gruntfile.js:

module.exports = function(grunt) {

    // Define scope for `load-grunt-tasks` package.
    var tasks = { scope: ['devDependencies', 'dependencies'] };

    // Define location of config files for `load-grunt-configs`.
    var options = { config: { src: "grunt-config/*.js" } };

    // Require `load-grunt-configs` with the defined options.
    var configs = require('load-grunt-configs')(grunt, options);

    // Load the various task configuration files.
    grunt.initConfig(configs);

    // Auto-load tasks.
    require('load-grunt-tasks')(grunt, tasks);

    // Register tasks
    // -------------------------------------------------------------------------
    grunt.registerTask('default', [
        'sass:development',
        'concat:public',
        'uglify:public',
        'exec:dev',
        'watch'
    ]);

    grunt.registerTask('deploy', [
        'exec:build',
        'exec:deploy'
    ]);

};

Now from within your project directory running grunt deploy will run the exec:build task (bundle exec jekyll build), followed by the exec:deploy task.

The exec:deploy task runs a BASH script called deploy-my-website - which should be held locally, outside your version control system.

BASH Deploy Script

I like to organise deploy scripts into a dedicated directory, creating symlinks to the files from with my /usr/local/bin directory (or somewhere within your $PATH).

The deploy script should be executable:

# Create symlink
sudo ln -s ~/deploy-scripts/deploy-my-website /usr/local/bin

# Make executable
sudo chmod +x ~/deploy-scripts/deploy-my-website

The deploy script utilises Rsync to copy files across to the server:

#!/bin/bash
# deploy-my-website
# Deploy to example.com
# ------------------------------------------------------------------------------
PORT=4321                                           # Custom port for SSH
USERNAME=my-user                                    # Username for the server
SERVER_IP="XXX.XXX.XXX.XXX"                         # Server IP address
REMOTE_PATH="/var/www/html/example.com/public_html" # Server path
LOCAL_PATH="/var/www/html/example.com/_site/"       # Local path
SITENAME="My website"                               # Site name

function deploy() {

    # Rsync options
    OPT=(--progress -a -v -rz --checksum --delete -e "ssh -p ${PORT}")

    rsync "${OPT[@]}" ${LOCAL_PATH} ${USERNAME}@${SERVER_IP}:${REMOTE_PATH}

}

echo "Deploying ${SITENAME} to PRODUCTION..."

deploy;

In our case, SSH is configured to only allow connection by means of SSH Keys - Public Key Cryptography/Challenge Response Authemtication. This makes the deployment process very secure - the local private key (passphrase protected for extra security) is necessary to establish the connection to the server.

References

grunt-exec plugin

Moving Site to New Server with Letsencrypt Certificates

Mon, 19 Dec 2016 19:31:00 +0000

When moving a site to a new server, you can migrate the LetsEncrypt certificates. You need to be careful that the /etc/letsencrypt directory is installed on the new server with the proper ownerships and permissions.

This guide is intended for Ubuntu Xenial 16.04.

Important: Because of the nature of the files (security certificates), they should be transferred by means of a secure protocol such as SSH.

Source Server

The /etc/letsencrypt directory on the original server needs to be copied to /etc on the new server.

This is a bit tricky, because the directory is owned by root - you can’t just rsync in and pull down the directory. You could add rsync to the sudo group on the source server as described here (Note: I’ve not tried this) but this makes me nervous due to potential mistakes/mischief.

Instead, sudo rsync the directory to a suitable location and set your user as the owner of the copy. For example:

# Note -p is added
sudo rsync -az -p --progress /etc/letsencrypt ~/
sudo chown -R $USER:$USER ~/letsencrypt

Local Machine

Pull the directory:

# Assumes SSH connection over port 4321
# replace $REMOTEUSER and $SOURCE_IP with suitable values
rsync -az -p -e 'ssh -p 4321' $REMOTEUSER@$SOURCE_IP:~/letsencrypt ~/servers/staging

You can now push this directory to the new location:

# Assumes connection over port 4221
rsync -az -p -e 'ssh -p 4221' ~/servers/staging/letsencrypt $NEW_REMOTE_USER@$DESTINATION_IP:

This will copy the letsencrypt directory to your users home directory on the destination server.

Destination Server

Move the directory into place and set proper ownership:

# Lack of trailing slash on source will cause the directory to be created on the destination directory
sudo rsync -az --progress ~/letsencrypt /etc
sudo chown -R root:root /etc/letsencrypt

Firewall Notes: HTTPS

Your firewall needs to open port 443. To check open ports, use netstat:

netstat -tnl

# Typical output - shows that port 443 is open
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:4321            0.0.0.0:*               LISTEN
tcp6       0      0 ::1:25                  :::*                    LISTEN
tcp6       0      0 :::443                  :::*                    LISTEN
tcp6       0      0 :::4321                 :::*                    LISTEN
tcp6       0      0 :::80                   :::*                    LISTEN

Note: You need to have a service listening on a port for the port to be determined “open”. This initially confused me - I hadn’t yet set up Apache for SSL (i.e. to listen on 443), and the netstat output did not show an entry for 443 - you might assume that your firewall is blocking the port, when you just do not have any services listening on 443.

Enable SSL: Apache

The Apache ssl module needs to be enabled for SSL/HTTPS to work:

sudo a2enmod ssl
# Restart Apache:
sudo service apache2 reload

Note re: rsync -p

As far as I’m aware, the -p option is implicit in -a, which is equivalent to -rlptgoD - so probably -p is unecessary. However, I had a couple of transfers that did not preserve permissions - maybe due to an error on my part, but no harm to include -p. There seems to be quite a subtle set of permissions on the letsencrypt files, so messing them up is not a trivial thing.

Overriding Fail2Ban Settings

Sat, 17 Dec 2016 15:50:00 +0000

Fail2ban takes config data from files in the following order of precedence:

/etc/fail2ban/jail.conf
/etc/fail2ban/jail.d/*.conf, alphabetically
/etc/fail2ban/jail.local
/etc/fail2ban/jail.d/*.local, alphabetically

Don’t amend *.conf files - which are distributed by Fail2Ban, and may be overwritten if Fail2Ban is upgraded. Instead, make customizations in *.local files. For example, if you would like to enable the [ssh-iptables-ipset] jail specified in /etc/fail2ban/jail.conf, create /etc/fail2ban/jail.local and add the following lines:

# /etc/fail2ban/jail.local
[ssh-iptables-ipset]
enabled = true

You only need to specify settings that you would like to change - the rest of the configuration is taken from the corresponding .conf file which is parsed first.

Reference: Content of `/etc/fail2ban/jail.conf`

It’s handy to have this available on a web page - it saves opening and closing the parent config on the server you’re working on. This is the config for Fail2Ban v0.9.3

#
# WARNING: heavily refactored in 0.9.0 release.  Please review and
#          customize settings for your setup.
#
# Changes:  in most of the cases you should not modify this
#           file, but provide customizations in jail.local file,
#           or separate .conf files under jail.d/ directory, e.g.:
#
# HOW TO ACTIVATE JAILS:
#
# YOU SHOULD NOT MODIFY THIS FILE.
#
# It will probably be overwritten or improved in a distribution update.
#
# Provide customizations in a jail.local file or a jail.d/customisation.local.
# For example to change the default bantime for all jails and to enable the
# ssh-iptables jail the following (uncommented) would appear in the .local file.
# See man 5 jail.conf for details.
#
# [DEFAULT]
# bantime = 3600
#
# [sshd]
# enabled = true
#
# See jail.conf(5) man page for more information



# Comments: use '#' for comment lines and ';' (following a space) for inline comments


[INCLUDES]

#before = paths-distro.conf
before = paths-debian.conf

# The DEFAULT allows a global definition of the options. They can be overridden
# in each jail afterwards.

[DEFAULT]

#
# MISCELLANEOUS OPTIONS
#

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8

# External command that will take an tagged arguments to ignore, e.g. <ip>,
# and return true if the IP is to be ignored. False otherwise.
#
# ignorecommand = /path/to/command <ip>
ignorecommand =

# "bantime" is the number of seconds that a host is banned.
bantime  = 600

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 600

# "maxretry" is the number of failures before a host get banned.
maxretry = 5

# "backend" specifies the backend used to get files modification.
# Available options are "pyinotify", "gamin", "polling", "systemd" and "auto".
# This option can be overridden in each jail as well.
#
# pyinotify: requires pyinotify (a file alteration monitor) to be installed.
#              If pyinotify is not installed, Fail2ban will use auto.
# gamin:     requires Gamin (a file alteration monitor) to be installed.
#              If Gamin is not installed, Fail2ban will use auto.
# polling:   uses a polling algorithm which does not require external libraries.
# systemd:   uses systemd python library to access the systemd journal.
#              Specifying "logpath" is not valid for this backend.
#              See "journalmatch" in the jails associated filter config
# auto:      will try to use the following backends, in order:
#              pyinotify, gamin, polling.
#
# Note: if systemd backend is choses as the default but you enable a jail
#       for which logs are present only in its own log files, specify some other
#       backend for that jail (e.g. polling) and provide empty value for
#       journalmatch. See https://github.com/fail2ban/fail2ban/issues/959#issuecomment-74901200
backend = auto

# "usedns" specifies if jails should trust hostnames in logs,
#   warn when DNS lookups are performed, or ignore all hostnames in logs
#
# yes:   if a hostname is encountered, a DNS lookup will be performed.
# warn:  if a hostname is encountered, a DNS lookup will be performed,
#        but it will be logged as a warning.
# no:    if a hostname is encountered, will not be used for banning,
#        but it will be logged as info.
usedns = warn

# "logencoding" specifies the encoding of the log files handled by the jail
#   This is used to decode the lines from the log file.
#   Typical examples:  "ascii", "utf-8"
#
#   auto:   will use the system locale setting
logencoding = auto

# "enabled" enables the jails.
#  By default all jails are disabled, and it should stay this way.
#  Enable only relevant to your setup jails in your .local or jail.d/*.conf
#
# true:  jail will be enabled and log files will get monitored for changes
# false: jail is not enabled
enabled = false


# "filter" defines the filter to use by the jail.
#  By default jails have names matching their filter name
#
filter = %(__name__)s


#
# ACTIONS
#

# Some options used for actions

# Destination email address used solely for the interpolations in
# jail.{conf,local,d/*} configuration files.
destemail = root@localhost

# Sender email address used solely for some actions
sender = root@localhost

# E-mail action. Since 0.8.1 Fail2Ban uses sendmail MTA for the
# mailing. Change mta configuration parameter to mail if you want to
# revert to conventional 'mail'.
mta = sendmail

# Default protocol
protocol = tcp

# Specify chain where jumps would need to be added in iptables-* actions
chain = INPUT

# Ports to be banned
# Usually should be overridden in a particular jail
port = 0:65535

#
# Action shortcuts. To be used to define action parameter

# Default banning action (e.g. iptables, iptables-new,
# iptables-multiport, shorewall, etc) It is used to define
# action_* variables. Can be overridden globally or per
# section within jail.local file
banaction = iptables-multiport

# The simplest action to take: ban only
action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

# ban & send an e-mail with whois report to the destemail.
action_mw = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
            %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]

# ban & send an e-mail with whois report and relevant log lines
# to the destemail.
action_mwl = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]

# See the IMPORTANT note in action.d/xarf-login-attack for when to use this action
#
# ban & send a xarf e-mail to abuse contact of IP address and include relevant log lines
# to the destemail.
action_xarf = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath=%(logpath)s, port="%(port)s"]

# ban IP on CloudFlare & send an e-mail with whois report and relevant log lines
# to the destemail.
action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
                %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]

# Report block via blocklist.de fail2ban reporting service API
#
# See the IMPORTANT note in action.d/blocklist_de.conf for when to
# use this action. Create a file jail.d/blocklist_de.local containing
# [Init]
# blocklist_de_apikey = {api key from registration]
#
action_blocklist_de  = blocklist_de[email="%(sender)s", service=%(filter)s, apikey="%(blocklist_de_apikey)s"]

# Report ban via badips.com, and use as blacklist
#
# See BadIPsAction docstring in config/action.d/badips.py for
# documentation for this action.
#
# NOTE: This action relies on banaction being present on start and therefore
# should be last action defined for a jail.
#
action_badips = badips.py[category="%(name)s", banaction="%(banaction)s"]

# Choose default action.  To change, just override value of 'action' with the
# interpolation to the chosen action shortcut (e.g.  action_mw, action_mwl, etc) in jail.local
# globally (section [DEFAULT]) or per specific section
action = %(action_)s


#
# JAILS
#

#
# SSH servers
#

[sshd]

port    = ssh
logpath = %(sshd_log)s


[sshd-ddos]
# This jail corresponds to the standard configuration in Fail2ban.
# The mail-whois action send a notification e-mail with a whois request
# in the body.
port    = ssh
logpath = %(sshd_log)s


[dropbear]

port     = ssh
logpath  = %(dropbear_log)s


[selinux-ssh]

port     = ssh
logpath  = %(auditd_log)s
maxretry = 5


#
# HTTP servers
#

[apache-auth]

port     = http,https
logpath  = %(apache_error_log)s


[apache-badbots]
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port     = http,https
logpath  = %(apache_access_log)s
bantime  = 172800
maxretry = 1


[apache-noscript]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 6


[apache-overflows]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2


[apache-nohome]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2


[apache-botsearch]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2


[apache-fakegooglebot]

port     = http,https
logpath  = %(apache_access_log)s
maxretry = 1
ignorecommand = %(ignorecommands_dir)s/apache-fakegooglebot <ip>


[apache-modsecurity]

port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-shellshock]

port    = http,https
logpath = %(apache_error_log)s
maxretry = 1

[nginx-http-auth]

port    = http,https
logpath = %(nginx_error_log)s

[nginx-botsearch]

port     = http,https
logpath  = %(nginx_error_log)s
maxretry = 2

# Ban attackers that try to use PHP's URL-fopen() functionality
# through GET/POST variables. - Experimental, with more than a year
# of usage in production environments.

[php-url-fopen]

port    = http,https
logpath = %(nginx_access_log)s
          %(apache_access_log)s


[suhosin]

port    = http,https
logpath = %(suhosin_log)s


[lighttpd-auth]
# Same as above for Apache's mod_auth
# It catches wrong authentifications
port    = http,https
logpath = %(lighttpd_error_log)s


#
# Webmail and groupware servers
#

[roundcube-auth]

port     = http,https
logpath  = logpath = %(roundcube_errors_log)s


[openwebmail]

port     = http,https
logpath  = /var/log/openwebmail.log


[horde]

port     = http,https
logpath  = /var/log/horde/horde.log


[groupoffice]

port     = http,https
logpath  = /home/groupoffice/log/info.log


[sogo-auth]
# Monitor SOGo groupware server
# without proxy this would be:
# port    = 20000
port     = http,https
logpath  = /var/log/sogo/sogo.log


[tine20]

logpath  = /var/log/tine20/tine20.log
port     = http,https
maxretry = 5


#
# Web Applications
#
#

[drupal-auth]

port     = http,https
logpath  = %(syslog_daemon)s

[guacamole]

port     = http,https
logpath  = /var/log/tomcat*/catalina.out

[monit]
#Ban clients brute-forcing the monit gui login
filter   = monit
port = 2812
logpath  = /var/log/monit


[webmin-auth]

port    = 10000
logpath = %(syslog_authpriv)s


[froxlor-auth]

port    = http,https
logpath  = %(syslog_authpriv)s


#
# HTTP Proxy servers
#
#

[squid]

port     =  80,443,3128,8080
logpath = /var/log/squid/access.log


[3proxy]

port    = 3128
logpath = /var/log/3proxy.log


#
# FTP servers
#


[proftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(proftpd_log)s


[pure-ftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(pureftpd_log)s
maxretry = 6


[gssftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(syslog_daemon)s
maxretry = 6


[wuftpd]

port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(wuftpd_log)s
maxretry = 6


[vsftpd]
# or overwrite it in jails.local to be
# logpath = %(syslog_authpriv)s
# if you want to rely on PAM failed login attempts
# vsftpd's failregex should match both of those formats
port     = ftp,ftp-data,ftps,ftps-data
logpath  = %(vsftpd_log)s


#
# Mail servers
#

# ASSP SMTP Proxy Jail
[assp]

port     = smtp,465,submission
logpath  = /root/path/to/assp/logs/maillog.txt


[courier-smtp]

port     = smtp,465,submission
logpath  = %(syslog_mail)s


[postfix]

port     = smtp,465,submission
logpath  = %(postfix_log)s


[postfix-rbl]

port     = smtp,465,submission
logpath  = %(syslog_mail)s
maxretry = 1


[sendmail-auth]

port    = submission,465,smtp
logpath = %(syslog_mail)s


[sendmail-reject]

port     = smtp,465,submission
logpath  = %(syslog_mail)s


[qmail-rbl]

filter  = qmail
port    = smtp,465,submission
logpath = /service/qmail/log/main/current


# dovecot defaults to logging to the mail syslog facility
# but can be set by syslog_facility in the dovecot configuration.
[dovecot]

port    = pop3,pop3s,imap,imaps,submission,465,sieve
logpath = %(dovecot_log)s


[sieve]

port   = smtp,465,submission
logpath = %(dovecot_log)s


[solid-pop3d]

port    = pop3,pop3s
logpath = %(solidpop3d_log)s


[exim]

port   = smtp,465,submission
logpath = %(exim_main_log)s


[exim-spam]

port   = smtp,465,submission
logpath = %(exim_main_log)s


[kerio]

port    = imap,smtp,imaps,465
logpath = /opt/kerio/mailserver/store/logs/security.log


#
# Mail servers authenticators: might be used for smtp,ftp,imap servers, so
# all relevant ports get banned
#

[courier-auth]

port     = smtp,465,submission,imap3,imaps,pop3,pop3s
logpath  = %(syslog_mail)s


[postfix-sasl]

port     = smtp,465,submission,imap3,imaps,pop3,pop3s
# You might consider monitoring /var/log/mail.warn instead if you are
# running postfix since it would provide the same log lines at the
# "warn" level but overall at the smaller filesize.
logpath  = %(postfix_log)s


[perdition]

port   = imap3,imaps,pop3,pop3s
logpath = %(syslog_mail)s


[squirrelmail]

port = smtp,465,submission,imap2,imap3,imaps,pop3,pop3s,http,https,socks
logpath = /var/lib/squirrelmail/prefs/squirrelmail_access_log


[cyrus-imap]

port   = imap3,imaps
logpath = %(syslog_mail)s


[uwimap-auth]

port   = imap3,imaps
logpath = %(syslog_mail)s


#
#
# DNS servers
#


# !!! WARNING !!!
#   Since UDP is connection-less protocol, spoofing of IP and imitation
#   of illegal actions is way too simple.  Thus enabling of this filter
#   might provide an easy way for implementing a DoS against a chosen
#   victim. See
#    http://nion.modprobe.de/blog/archives/690-fail2ban-+-dns-fail.html
#   Please DO NOT USE this jail unless you know what you are doing.
#
# IMPORTANT: see filter.d/named-refused for instructions to enable logging
# This jail blocks UDP traffic for DNS requests.
# [named-refused-udp]
#
# filter   = named-refused
# port     = domain,953
# protocol = udp
# logpath  = /var/log/named/security.log

# IMPORTANT: see filter.d/named-refused for instructions to enable logging
# This jail blocks TCP traffic for DNS requests.

[named-refused]

port     = domain,953
logpath  = /var/log/named/security.log


[nsd]

port     = 53
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
logpath = /var/log/nsd.log


#
# Miscellaneous
#

[asterisk]

port     = 5060,5061
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
logpath  = /var/log/asterisk/messages
maxretry = 10


[freeswitch]

port     = 5060,5061
action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
logpath  = /var/log/freeswitch.log
maxretry = 10


# To log wrong MySQL access attempts add to /etc/my.cnf in [mysqld] or
# equivalent section:
# log-warning = 2
#
# for syslog (daemon facility)
# [mysqld_safe]
# syslog
#
# for own logfile
# [mysqld]
# log-error=/var/log/mysqld.log
[mysqld-auth]

port     = 3306
logpath  = %(mysql_log)s
maxretry = 5


# Jail for more extended banning of persistent abusers
# !!! WARNINGS !!!
# 1. Make sure that your loglevel specified in fail2ban.conf/.local
#    is not at DEBUG level -- which might then cause fail2ban to fall into
#    an infinite loop constantly feeding itself with non-informative lines
# 2. Increase dbpurgeage defined in fail2ban.conf to e.g. 648000 (7.5 days)
#    to maintain entries for failed logins for sufficient amount of time
[recidive]

logpath  = /var/log/fail2ban.log
banaction = iptables-allports
bantime  = 604800  ; 1 week
findtime = 86400   ; 1 day
maxretry = 5


# Generic filter for PAM. Has to be used with action which bans all
# ports such as iptables-allports, shorewall

[pam-generic]
# pam-generic filter can be customized to monitor specific subset of 'tty's
banaction = iptables-allports
logpath  = %(syslog_authpriv)s


[xinetd-fail]

banaction = iptables-multiport-log
logpath   = %(syslog_daemon)s
maxretry  = 2


# stunnel - need to set port for this
[stunnel]

logpath = /var/log/stunnel4/stunnel.log


[ejabberd-auth]

port    = 5222
logpath = /var/log/ejabberd/ejabberd.log


[counter-strike]

logpath = /opt/cstrike/logs/L[0-9]*.log
# Firewall: http://www.cstrike-planet.com/faq/6
tcpport = 27030,27031,27032,27033,27034,27035,27036,27037,27038,27039
udpport = 1200,27000,27001,27002,27003,27004,27005,27006,27007,27008,27009,27010,27011,27012,27013,27014,27015
action  = %(banaction)s[name=%(__name__)s-tcp, port="%(tcpport)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
           %(banaction)s[name=%(__name__)s-udp, port="%(udpport)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]

# consider low maxretry and a long bantime
# nobody except your own Nagios server should ever probe nrpe
[nagios]

enabled  = false
logpath  = %(syslog_daemon)s     ; nrpe.cfg may define a different log_facility
maxretry = 1


[oracleims]
# see "oracleims" filter file for configuration requirement for Oracle IMS v6 and above
enabled = false
logpath = /opt/sun/comms/messaging64/log/mail.log_current
maxretry = 6
banaction = iptables-allports

[directadmin]
enabled = false
logpath = /var/log/directadmin/login.log
port = 2222

[portsentry]
enabled  = false
logpath  = /var/lib/portsentry/portsentry.history
maxretry = 1

[pass2allow-ftp]
# this pass2allow example allows FTP traffic after successful HTTP authentication
port         = ftp,ftp-data,ftps,ftps-data
# knocking_url variable must be overridden to some secret value in filter.d/apache-pass.local
filter       = apache-pass
# access log of the website with HTTP auth
logpath      = %(apache_access_log)s
blocktype    = RETURN
returntype   = DROP
bantime      = 3600
maxretry     = 1
findtime     = 1

The above code is Copyright (c) 2004-2008 Cyril Jaquier, 2008- Fail2Ban Contributors Copyright of modifications held by their respective authors. Licensed under the GNU General Public License v2 (GPL).

Written by Cyril Jaquier cyril.jaquier@fail2ban.org. Many contributions by Yaroslav O. Halchenko debian@onerussian.com

Resources

Install LAMP Stack on Ubuntu 16.04 Xenial Xerus

Sat, 17 Dec 2016 10:43:00 +0000

Build notes for installing Apache, MariaDB, PHP.

Install Apache

sudo apt-get install apache2

Install PHP

Install PHP with extra modules.

sudo apt-get install php libapache2-mod-php php-mcrypt php-mysql

Install Maria DB

Use the online respository configuration tool. Select a local mirror.

Based on Ubuntu Xenial, stable release:

sudo apt-get install software-properties-common
sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
sudo add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://ftp.heanet.ie/mirrors/mariadb/repo/10.1/ubuntu xenial main'

# Once the key is imported, install MariaDB:
sudo apt-get update
sudo apt-get install mariadb-server

Secure MariaDB by removing anonymous users, removing the test database and disallowing external access:

sudo mysql_secure_installation

Install PHP Extras

For extensions available, enable with phpenmod:

sudo phpenmod mcrypt
sudo service apache2 restart

Extra extensions may be required. For example, if multibyte functions are used in apps, you will need php-mbstring:

sudo apt-get install php-mbstring
sudo phpenmod mbstring
sudo apt-get install php-xml
sudo service apache2 restart

Install PHPMyAdmin

Install phpMyAdmin:

sudo apt-get install phpmyadmin apache2-utils

An ncurses window will open that enables phpMyAdmin to be configured.

To select a server, hit space then tab/return
Choose “yes” when asked “Configure database for phpmyadmin with dbconfig-common?”
Enter a password for phpMyAdmin when prompted

Once installed, PHPMyAdmin needs to be added to the Apache config file:

sudo nano /etc/apache2/apache2.conf

# Add the following line to the end of apache2.conf:

Include /etc/phpmyadmin/apache.conf

Restart apache:

sudo service apache2 restart

When installing phpMyAdmin on Xenial Xerus, you may encounter a blank white screen of death. Try checking the Apache error log to determine the problem:

tail -f /var/log/apache2/error.log

In my case, the relevant error was:

[Fri Jun 17 20:04:50.205836 2016] [:error] [pid 8089] [client 127.0.0.1:34540] PHP Fatal error:  require_once(): Failed opening required '/usr/share/php/php-gettext/gettext.inc' (include_path='.:/usr/share/php') in /usr/share/phpmyadmin/libraries/select_lang.lib.php on line 477

This means that the php-gettext extension is required. Install this and restart Apache to fix:

sudo apt-get install php-gettext
sudo service apache2 restart

Grunt Uncss and Jekyll

Wed, 30 Nov 2016 10:34:00 +0000

Front-end frameworks like Bootstrap and Foundation provide a really good way to deliver high-quality modern responsive websites. They take care of the grunt work in terms of layout and common design elements and thereby drastically speed up development time.

I don’t buy the argument that they constrain design. You can style and override these frameworks as required.

They do have a major disadvantage -size. You’re often including (a lot of) rules in your stylesheet that are not used in the current project. Enter Uncss. This article describes how I use Uncss with Grunt on Jekyll projects.

Uncss

Uncss is a node module that removes unused CSS from stylesheets.

On a (fairly typical) recent Jekyll project, Uncss helped me to trim a 190 kB stylesheet down to 32 kB. This is great for user experience, as it speeds up page-load time. It’s probably better for the environment as well, as we’re zapping less bytes down the wire to get the same result.

How Uncss works:

HTML files are loaded by PhantomJS, a headless browser.
Stylesheets are parsed by PostCSS - a tool that allows JS plugins to transform styles.
If a selector is not found in the HTML files, it is removed.
Remaining rules are converted back to CSS.

You can specify rules to be ignired by Uncss - for example, styles that are added by JavaScript.

Grunt & Uncss

We use Grunt as a front end task runner on our Jekyll projects. No, I don’t care that it’s old-fashioned (!). It performs really well for us and there is no need to change it for a shinier tool in this context.

We use the grunt-uncss grunt task to run Uncss in development.

To install:

# npm
npm install grunt-uncss --save-dev

# better yet, use Yarn - it's quicker:
yarn add grunt-uncss --dev

Example grunt-uncss Task

We split out grunt tasks into separate files. Contents of a typical config/uncss.js file:

module.exports = {
  dist: {
    options: {
      ignore: [
        /\w\.in/,
        '.top-nav-collapse',
        'img-caption',
        '.top-nav-collapse',
        '.fade',
        '.collapse',
        '.collapsing',
        /(#|\.)navbar(\-[a-zA-Z]+)?/,
        /(#|\.)dropdown(\-[a-zA-Z]+)?/,
        '.bs.carousel',
        '.slid.bs.carousel',
        '.slide.bs.carousel',
        '.fade',
        '.fade.in',
        '.collapse',
        '.collapse.in',
        '.collapsing',
        '.alert-danger',
        '.carousel-inner > .next',
        '.carousel-inner > .prev',
        '.carousel-inner > .next.left',
        '.carousel-inner > .prev.right',
        '.carousel-inner > .active.left',
        '.carousel-inner > .active.right',
        '.navbar-fixed-top'
      ],
      stylesheets: ['../css/main.css'],
      ignoreSheets: [/fonts.googleapis/]
    },
    files: {
      'css/main.un.css': ['_site/index.html']
    }
  }
};

Grunt Uncss with Jekyll

We manage all our Jekyll builds by means of Grunt tasks. Specifically, we use the grunt-shell package to build and deploy Jekyll. This provides a convenient way of specifying different configurations for different environments.

We generally include the uncss task on staging and production builds, but not on development builds. Actually we keep the staging build pretty much identical to production for better quality assurance.

Before Grunt Uncss can work, it needs something to work on, which is why the staging grunt task below starts with a Jekyll build ('shell:jekyllNoMap'). More on the “NoMap” issue later…

// Taken from `/Gruntfile.js`

// Deploy to Staging with `grunt deployStaging` then reset the local copy.
    grunt.registerTask( 'deployStaging', [
        'shell:jekyllNoMap',    // Build the site (to _site)
        'sass:production',      // Run sass task to build CSS
        'uncss:dist',           // Run uncss on the built HTML/CSS
        'cssmin',               // Minify the processed CSS
        'uglify:dist',          // Uglify site JS
        'shell:jekyllStaging',  // Run the staging build (CSS previously generated)
        'copy:fonts',           // Copy fonts to _site
        'shell:rsyncStaging',   // Deploy to staging
        'sass:development',     // After deployment, run dev tasks (for convenience)
        'concat',
        'shell:jekyllDev',
        'copy:fonts',
        'watch'
    ]);

Problems with Google Maps

We recently experienced a problem with gulp-uncss and Google maps. After updating the project, the Uncss task was providing an incomplete style sheet - with no error messages.

Some ideas for debugging:

Look through the generated (un)css file and search for a style you know should be there: if it isn’t, uncss has encountered a problem
Make sure your HTML and CSS is valid
Comment out scripts and go through a process of elimination

In this particular case, the problem was caused by a Google maps call in the <head>:

<script src="https://maps.googleapis.com/maps/api/js?key={{site.data.map.api-key}}"></script>

With this line commented out, uncss worked perfectly. I have a feeling that this issue is caused by PhantomJS and the fact that the site is running on the http protocol, whereas the link to Google is https.

To be honest, sometimes you can’t justify the time to get to the bottom of issues like this - you just need a quick fix. Fortunately Jekyll makes it pretty easy to define environments by including multiple config files. For example, to build a dev environment you include a _config_dev.yml config file in your Jekyll build. For example, see our shell.js config:

module.exports = {
  jekyllNoMap: {
    command: 'rm -rf _site/*; jekyll build --config _config.yml,_config_staging.yml,_config_no_map.yml',
    stdout: true
  },
  jekyllStaging: {
    command: 'rm -rf _site/*; jekyll build --config _config.yml,_config_staging.yml',
    stdout: true
  },
  jekyllDev: {
    command: 'rm -rf _site/*; jekyll build --config _config.yml,_config_dev.yml',
    stdout: true
  },
  jekyllProduction: {
    command: 'rm -rf _site/*; jekyll build --config _config.yml,_config_production.yml',
    stdout: true
  },
  rsyncStaging: {
    command: 'deploy-staging'
  },
  rsyncProduction: {
    command: 'deploy-production'
  }
};

The noMap config file _config_no_map.yml looks like this:

url: "http://example.com/staging-site"
baseurl: "/staging-site"
production: true
environment: staging
include-map: false

You can then exclude the map script if include-map returns false:

<!-- in <head> -->

{% if page.map-height != nil and site.include-map != false %}
    {% include partials/map-scripts.html %}
{% endif %}

Laravel Reflection Error when Class Contains Errors

Tue, 15 Nov 2016 18:15:00 +0000

This is a quick note - intended to be a memory jogger.

Laravel will throw a ReflectionException error if a class contains an error.

For example:

ReflectionException in Route.php line 339: Class App\Http\Requests\CreateLeadRequest does not exist

This is pretty logical - if the class can’t be instantiated, it doesn’t exist.

It can be confusing if you’re looking at your project file tree and can see the class file (“I can see it - the class exists!”). In my case, I checked the calling class several times before bothering to check the content of the class that was being called - which was missing a comma.

To prevent this kind of thing:

Use a PHP linter - here’s one for Atom
Check all the classes involved, especially the one you’re calling - don’t assume you’ve got an autoloading issue

Generate URLs to Named Routes in Laravel

Tue, 15 Nov 2016 10:13:00 +0000

Named routes in Laravel allow you to easily generate specific URLs. Referring to the named route throughout your project makes the codebase more robust. If a URL needs to be amended, this can be done easily by editing the route definition. Without named routes you’d need to amend all URL references to the amended route.

Routes generated by the Route::resource() method will automatically have named routes assigned - taking the format resource.method.

Manually Specify Named Routes

This is achieved by chaining the name() method to the route definition:

<?php
// Define route in `routes/web.php`

Route::get('about', function() {
    return view('pages.about');
})->name('about');

// Or using a controller:
Route::get('about', 'PageController@showAbout')->name('about');

Generate URLs to Named Routes

Laravel provides a global route() function that gets the URL to a named route. The first Parameter is the route name (string). Depending on the route you’re trying to access you may also need to pass in an array of parameters as the second argument.

Simple example, as it would look as an anchor tag within in a Blade template:

<a href="{{ route('posts.show', [$id]) }}">Link to Resource {{ $id }}</a>

…this generates a URL that links to posts/{id}. For example: http://example.com/posts/10.

The route() function accepts three parameters:

Route name (string)
Parameters (mixed)
Absolute (boolean, true by default)

You could make the URL relative by passing in false as a third argument to the function:

<a href="{{ route('posts.show', [$id], false) }}">Link to Resource {{ $id }}</a>

…this generates: /posts/10.

Adding Additional Parameters

The example above passes a single parameter ($id) into the route function. In this case, the ‘show’ method that is being hit expects a single argument - and this is correctly passed in. If more arguments are required (for example in the case of nested resources) they will be passed in the order specified in the route() function.

Interestingly, Laravel is smart enough to know that additional parameters (beyond what is required by the accepting method) should be passed as GET parameters.

Example:

<?php
// Define route in `routes/web.php`
Route::get('posts/{post}/comments/{comment}', 'CommentController@show')->name('comment');

{-- Create link to named route in Blade --}
<a href="{{ route('comment', ['1', '2', 'par'=>'HELLO', 'par2'=>'Goodbye']) }}">The Comment</a>

The generated URL looks like: http://example.com/posts/1/comments/2?par=HELLO&par2=Goodbye

This might be handy of you were redirecting back to a page - you could use the amended GET parameters to alter elements of the page display.

Using Within Laravel Collective Forms

If your project requires the Laravel Collective Forms & HTML package, you can use named routes to set the form by passing an array to the Form::open() method with with the key ‘route’:

{!! Form::open(['method' => 'DELETE',
    'route' => ['leads.destroy', $id],
    'id' => 'form-delete-lead-' . $id])
    !!}

The ‘route’ array is comprised of ‘route.name’ followed by route parameters.

Redirects

The Redirect::to() method will also accept a named route:

<?php
...
// redirect to http://example.com/leads?ship=Rocinante
return redirect()->route('leads.index', ['ship' => 'Rocinante']);

Nested Resource Routes in Laravel

Mon, 14 Nov 2016 17:44:00 +0000

To create a resource controller, run:

php artisan make:controller PostController --resource

This will create a controller with stubbed out methods for handling typical CRUD actions.

Resourceful Route to the Controller

The Laravel resourceful route goes hand-in-hand with the resource controller. To generate typical CRUD routes to the controller, add this line to routes/web.php (Laravel 5.3+):

Route::resource('posts', PostController);

This route declaration sets up multiple routes to the controller. You can view these routes by running php artisan route:list:

+--------+-----------+--------------------+---------------+---------------------------------------------+------------+
| Domain | Method    | URI                | Name          | Action                                      | Middleware |
+--------+-----------+--------------------+---------------+---------------------------------------------+------------+
|        | GET|HEAD  | /                  |               | Closure                                     | web        |
|        | GET|HEAD  | about              |               | Closure                                     | web        |
|        | GET|HEAD  | posts              | posts.index   | App\Http\Controllers\PostController@index   | web        |
|        | POST      | posts              | posts.store   | App\Http\Controllers\PostController@store   | web        |
|        | GET|HEAD  | posts/create       | posts.create  | App\Http\Controllers\PostController@create  | web        |
|        | GET|HEAD  | posts/{posts}      | posts.show    | App\Http\Controllers\PostController@show    | web        |
|        | PUT|PATCH | posts/{posts}      | posts.update  | App\Http\Controllers\PostController@update  | web        |
|        | DELETE    | posts/{posts}      | posts.destroy | App\Http\Controllers\PostController@destroy | web        |
|        | GET|HEAD  | posts/{posts}/edit | posts.edit    | App\Http\Controllers\PostController@edit    | web        |

+--------+-----------+--------------------+---------------+---------------------------------------------+------------+

Nested Routes

Nested routes allow you to capture a relationship between resources within your routing. For example, resources from a Post model might be nested under a User model: users/{user}/posts/{post}. The resulting URL might look like this: http://example.com/users/1/posts/10.

To create nested routes, Laravel lets you use a dot notation. Sticking with the previous example:

Route::resource('users.posts', 'PostController');

This would create a set of routes for posts that include the user identifier. For example:

The ‘index’ route: http://example.com/users/1/posts
The ‘show’ route: http://example.com/users/1/posts/10

You can override the out-of-the-box routes that are set up with the resource() method, or add additional routes - you should add overrides before the resource method.

Nesting resources can make the URLs for your project unwieldy. They may also add complexity to your controllers - for example, the generated controller method stubs may require additional parameters, and you may need to pass additonal parameters when building forms.

Nested Resources: Forms

When building a form action in the Laravel Collective Forms & HTML package, you need to pass in the additional parameters that are needed to build the route.

Create New Resources

To set up a form action based on the route users/{user}/post/{post} which is used to store a new resource:

{{-- Blade template ----------------------------------------------------------}}
{{-- Open the form and use a named route for submissions ---------------------}}
{!! Form::open(['route'=>['users.posts.store', $user_id]]) !!}
    {{-- Include the form to keep things DRY ---------------------------------}}
    @include('posts.form', ['submitButtonText' => 'Create New Post'])
{!! Form::close() !!}

Note that only the user ID is required - this is a new post, so the post ID does not yet exist.

The store method on PostController might look like this:

    <?php
    /**
     * Store a newly created Post in storage.
     *
     * @param  \Illuminate\Http\Request  $request
     * @return \Illuminate\Http\Response
     */
    public function store(Request $request, $user_id)
    {

        $user = User::findOrFail($user_id);
        $user->posts()->create($request->all());

    }

This relies on the user parameter being passed in through the URL. You could also not bother with this and do Auth::user()->posts()->create($request->all()); to create the new post instead.

Edit Existing Resources

To set the form action so that it spoofs a PATCH method to the users/{user}/posts/{post}/edit endpoint:

{{-- Blade template ----------------------------------------------------------}}
{!! Form::model($post, ['method'=> 'PATCH', 'route'=>['users.posts.update', $user_id, $post->id]]) !!}
    @include('posts.form', ['submitButtonText' => 'Edit Post'])
{!! Form::close() !!}

To update the edited resource, you could use a controller update method like this:

    <?php
    /**
     * Store a newly created Post in storage.
     *
     * @param  \Illuminate\Http\Request  $request
     * @param  int  $user_id
     * @param  int  $post_id
     * @return \Illuminate\Http\Response
     */
     public function update(Request $request, $user_id, $post_id)
     {
         $post = Post::findOrFail($post_id);

         $post->update($request->all());

         return redirect('posts');
     }

Note that the parameters you pass in need to be ordered as they are in the URL.

Checking Routes

If in doubt, check registered routes by running php artisan route:list - which outputs a helpful table showing route names along with URLs and controller methods.

References

Laravel Docs on Controllers

Push to Named Stacks in Laravel Blade

Wed, 19 Oct 2016 21:07:00 +0100

Laravel blade usefully allows you to push to named stacks which can be rendered in different views. You can push either inline code, or a call to an external file.

Push to the Stack

You can push to a given stack multiple times. Pushing is achieved by wrapping your code in @push. This code triggers an alert confirmation if a delete form is submitted. The code is only required on one view, so pushing to the stack looks like a good option:

@push('scripts-footer')
  <script>
    $(function () {
      $('.data-delete').on('click', function (e) {
        var resourceTitle = $(this).data('title');
        if (!confirm('Are you sure you want to delete ' + resourceTitle + '?')) return;
        e.preventDefault();
        $('#form-delete-' + $(this).data('form')).submit();
      });
    });
  </script>
@endpush

This method can be quite nice for small bits of functionality as you can work on your script with the markup in front of you (you don’t have to remember data attributes etc). You could abstract your script out to an external file, and push a call to this file to the stack:

@push('scripts-footer')
    <script src="/build/js/are-you-sure.js"></script>
@endpush

Render the Stack

Then in another template, output the stack. For example, in the blade template for the footer:

@stack('scripts-footer')

More Stuff to Explore

Apparently if you @push from inside a form using model binding, then you have access to that context. I haven’t tried this but it looks interesting.

References

Add Body Class on Laravel Views

Sat, 15 Oct 2016 19:13:00 +0100

It is useful to add resource-specific classes to the body class of a page to allow elements to be targeted by CSS.

Pass Variable from Child View to Master Layout

Pass a relevant variable from the blade template:

{{-- /views/articles/index.blade.php --}}
...
@extends('layouts.master', ['body_class' => 'articles index'])
...

Then in /views/layouts/master.blade.php:

<body
  @unless(empty($body_class))
    class="{{$body_class}}"
  @endunless
  >

You need to check that the variable is not empty - otherwise, if $body_class isn’t passed in to the master template an error will occur.

Use @section

In the child blade template:

@section('pageClass', 'js-home-page')

In the master:

<body class="@yield('pageClass')">

References

https://www.reddit.com/r/laravel/comments/3i44nz/how_do_i_add_body_class_for_each_page/

Grep Commands Cheatsheet

Thu, 13 Oct 2016 10:56:00 +0100

Grep is a handy command-line utility that enables you to search text or files.

More formally: Grep (Global Regular Expression Print) is a utility for searching plain-text datasets for lines that match a given regular expression.

Grep was developed for Unix, but is now available for all unix-like systems (e.g. Linux).

Basic Command Syntax

# Output all lines that contain 'target string' in filename.log
grep 'target string' /var/log/filename.log

# Output all lines that contain 'target string' in multiple files
grep 'target string' filename1 filename2 filename3

Ignore Case

Use -i:

# Return lines containing 'David', 'david', 'DAVID' etc
grep -i 'david' /var/log/auth.log

Recursive Search

Use -r:

# Search all apache log files for example.com/about
grep -r "example.com/about" /var/log/apache2

# Sample Output:
/var/log/apache2/example.com.access.log.1:88.87.168.109 - - [12/Oct/2016:21:33:16 +0100] "GET /uploads/2015/08/example.jpg HTTP/1.1" 200 628726 "http://example.com/about/" "Mozilla/5.0 (Linux; Android 6.0.1; SM-G920F Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.124 Mobile Safari/537.36"

To suppress the filename. use the -h option. This example shows suppressed filenames and sends the result output to a file:

grep -h -r "example.com/about" /var/log/apache2 > ~/about-log

Search for Whole Words

Grep will return lines that contain the target string - it may be a fragment of another string. To return only lines containing the target string as a distinct word, use the -w option:

# Return lines containing 'example' as a distinct word
grep -w 'example' /var/log/logfile.log

Search for many words requires egrep (extended grep), or escaping the pipe character:

# Returns lines that contain either specified word using egrep
egrep -w "david|elaine" /var/log/auth.log

# Same:
grep -w "david\|elaine" /var/log/auth.log

# Recursively search apache logs for either specified string:
egrep -Rwi --color 'example.com/about|example.com/contact' /var/log/apache2

References

Variable Variables in PHP

Fri, 07 Oct 2016 18:36:00 +0100

PHP allows you to set dynamic variable names. This can be very useful when working with arrays and loops.

To use a variable in a variable declaration, wrap the declaration in curly brackets.

This is an example of a variable variable being used to build a form element in the context of a WordPress widget:

<?php

$options = ['mobile', 'landline']

foreach ($options as $option) {

  $fieldname = $option . '_text';

  ?>
  <label>
    Set the text for <?= $option; ?>
    <input
      type="text"
      name="<?= $this->get_field_name( $fieldname ); ?>"
      value="<?= ${$option . '_text'}; ?>" />
  </label><br>
  <?php

}
?>

In this example, during each loop the “value” attribute is set to a variable which is built on the fly. During the first iteration, the value attribute looks like value="<?= $mobile_text; ?>". This variable allows us to retrieve previously saved data that is represented by $mobile_text. On the next loop, the value will be set to $landline_text.

Resources

Variable variables for dummies PHP man page

Install Ruby & Ruby on Rails in Ubuntu Xenial

Sun, 02 Oct 2016 11:53:00 +0100

This guide involves:

Setting up Ruby using rbenv
Installing Node
Installing Rails

Install Ruby Dependencies

Packages required to build ruby.

sudo apt-get update
sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev

Install rbenv & ruby-build

git clone https://github.com/rbenv/rbenv.git ~/.rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc

# Add command completion scripts into path
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
exec $SHELL

git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc
exec $SHELL

rbenv install 2.3.1
rbenv global 2.3.1
ruby -v

Install Bundler

gem install bundler
rbenv rehash

Set Up Git Globally

If you haven’t already done it…

git config --global color.ui true
git config --global user.name "David Egan"
git config --global user.email "you@example.com"
ssh-keygen -t rsa -b 4096 -C "you@example.com on xenial-ghost"
cat ~/.ssh/id_rsa.pub

Copy and paste the contents of the public key to Github/BitBucket.

Check the ssh connection:

ssh -T git@github.com

Set Up Node

Install Node using NVM (Node Version Manager)

See: http://linoxide.com/ubuntu-how-to/installing-nodejs-ubuntu-16-04/

sudo apt-get update
sudo apt-get install build-essential libssl-dev curl
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.1/install.sh | bash
source ~/.profile
nvm ls-remote
nvm install v6.2.2
node -v

Build a quick hello world node app to check:

nano hello.js

# Enter:
a="Nice";
b="One!"
c=2;
d=2;
e=c+d;
console.log(c +' plus '+d+' equals:' +e);
console.log('Hello World! '+a+' '+b);

Run node hello.js in the terminal to output:

2 plus 2 equals:4
Hello World! Nice One!

Install Rails

gem install rails -v 4.2.6
rbenv rehash
rails -v

Build a New Rails App using MySQL(MariaDB)

Assuming that a MySQL server is already installed on the system, prepare Rails for MySQL:

# Requirement to compile the mysql2 gem
sudo apt-get install libmysqlclient-dev

# Gem used by Rails to connect to MySQL
gem install mysql2

Set up new Rails app:

rails new railsapp -d mysql
cd railsapp
rake db:create
rails server

References

Preventing User Enumeration in WordPress

Fri, 30 Sep 2016 15:54:00 +0100

We recently had to deal with a hacking attempt against a client WordPress site that had a few interesting aspects. The fix involved additional .htaccess rules to block user enumeration.

We experienced multiple failed login attempts against WordPress. This wasn’t particularly worrying - the originating IP address was automatically blocked by our Fail2Ban setup after three unsuccessful attempts. The attacker (probably a script) then switched IP address and repeated the process, trigerring a further ban and repeating the cycle. The attack lasted for approximately 10 minutes and triggered more than 50 bans.

The attack focused on usernames that were very close to (but not actually the same as) actual usernames on the site. It looked like a partially-successful user-enumeration attempt made up the initial phase of the attack. Puzzlingly, only some usernames had been enumerated.

User Enumeration

User Enumeration is when would-be attackers collect usernames by interacting with your app. Unfortunately, by default WordPress makes this process easy. Entering http://example.com/?author=1 in the browser will trigger display of all articles authored by the user with an ID of ‘1’ - along with their registered username. This provides would-be attackers with a toe-hold - they can attempt to log in to valid usernames rather than having to guess.

Our usual setup involves user-enumeration prevention measures - so it was surprising to see (almost valid) usernames cropping up in the log.

It turns out that our user-enumeration prevention relied on ‘redirect_canonical’ WordPress filter. This filter is triggered if you navigate to http://example.com/?author=1 - in this case, it performs a redirect to the Author archives for the author with an ID of 1.

The problem: If a registered user on the site has not authored any articles, the redirect will not take place. The user does not have an archive, the redirect doesn’t take place, and the user-enumeration can proceed.

In our case, the enumerated users had a custom membership role rather than an author role - so they will never have an archive page. In our context, these are pretty low risk users, with very few permissions on the site. Nevertheless, it’s a pain having to check when these attacks occur, and it places unecessary load on the server.

We verified the partially successful enumeration attempt by doing some penetration testing using WPScan - this turned up the exact “usernames” that were tried during the hack attempt.

The solution involved extra .htaccess rules to prevent user-enumeration. We also added some extra rules to block login attempts using the enumerated (incorrect) usernames - just in case the attacker is logging them for future usage.

.htaccess Rule to Prevent User enumeration

RewriteEngine	On
RewriteCond	%{REQUEST_URI}	!^/wp-admin [NC]
RewriteCond	%{QUERY_STRING}	author=\d
RewriteRule	(.*)		$1? [L,R=301]

Explanation

Line One

Turn on rewriting functionality - the Apache mod_rewrite module must be installed on the server. This module rewrites requested URLs on the fly by means of a rule-based rewriting engine. The rewrite engine is based on a Perl Compatible Regular Expressions(PCRE) parser.

Line 2

Apply a rewrite condition such that the rule will be ignored if the REQUEST_URI begins with /wp-admin.

REQUEST_URI

The path component of the requested URI, such as “/index.html”. This notably excludes the query string which is available as its own variable named QUERY_STRING. — Apache mod_rewrite Docs

REQUEST_URI in simple terms is the bit after your domain.

The author=\d string that we’ll use to match the user enumeration attempt is used legitimately to display author posts in back end - so the rewrite rule should not apply if the request takes place in the WordPress admin area.

Line 3

Specify the rewrite condition - the target query string must include 'author=\d', where \d means a single digit.

This means that http://example.com/?dummy&author=1 will trigger the rewrite, as will http://example.com/?dummy&author=100 - provided we’re not in the admin area, as specified by the previous condition.

Note that the rule doesn’t specify that the ‘author’ variable is at the start of the query string (e.g. ^/?author=([0-9]*) - a query string that starts with /?author= followed by any number of digits).

Line 4

The rewrite rule: replace the entire path (.*) with itself $1 but with an empty query string ?.

Make this the last rule and specify that it is a permanent redirect [L,R=301].

TLDR: .htaccess Rules

Add these rules to .htaccess to prevent all malicious user-enumeration attempts. Such attempts will redirect to the site home page:

RewriteEngine	On
RewriteCond	%{REQUEST_URI}	!^/wp-admin [NC]
RewriteCond	%{QUERY_STRING}	author=\d
RewriteRule	(.*)		$1? [L,R=301]

Note that preventing user-enumeration is only one component of an effective security policy.

References

Inject Menu Item into a Specific WordPress Menu

Wed, 28 Sep 2016 16:30:00 +0100

The ‘wp_nav_menu_items’ WordPress filter allows you to filter the HTML list content for nav menus.

The filter can accept up to two parameters:

The HTML list content (<li>s)
An object containing wp_nav_menu() arguments

Passing in the second parameter allows you to target the nav menu. The following code appends a new menu item to the ‘Primary Navigation’ menu:

<?php
add_action ('wp_nav_menu_items', function( $menu_items, $menu_object ){

  $menu_name = $menu_object->menu->name;

  // Only append the new li to the 'Primary Navigation' menu.
  // Change 'Primary Navigation' for the name of the menu that you'd like to amend.
  if ( 'Primary Navigation' === $menu_name ) {

    $new_li = "<li><a class='btn btn-default' href='#'>The New Link</a></li>";

    return $menu_items . $new_li;

  }

}, 10, 2 );

References

‘wp_nav_menu_items’ documentation

WordPress Basic Ajax

Wed, 28 Sep 2016 11:50:00 +0100

The WordPress way of handling AJAX requests is to use a specially constructed hook name built from the ‘action’ property of your AJAX request prepended with ‘wp_ajax_’ or ‘wp_ajax_nopriv_’.

In a namespaced setup (like Roots\Sage) you’d set up hooks and callbacks like this:

<?php
add_action( 'wp_ajax_my_custom_action', __NAMESPACE__ . '\\ajax_processor' );
add_action( 'wp_ajax_nopriv_my_custom_action', __NAMESPACE__ . '\\ajax_processor' );

function ajax_processor() {

  // Validate incoming POST data before further processing - return errors if necessary
  $currency = $_POST['currency'];

  // Do stuff, return true on success
  $processed_data = data_processor($currency);

  if( true === $processed_data ) {

    // Success condition: send back stuff to alter in the browser
    $response = [];
    $response['status'] = 'success!!';
    $response['postedData'] = $_POST;
    $response['currencySelected'] = $currency;
    $response['price'] = $price;

  } else {

    // Error condition: Build an informative error message
    $response = ['error'] = 'foo';

  }

  // send $response back as a JSON object
  return wp_send_json( $response );

}

In this case, when you send an AJAX request to WordPress with a request ‘action’ property set to ‘my_custom_action’ the hook callback will be automatically executed.

If you’re using the ‘nopriv’ option (for users who are not logged in) your ajaxurl property won’t be automatically defined, so you need to include it manually or pass it in via wp_localize_script() within the enqueuing script. You should enqueue the javascript in any case.

Enqueue JavaScript

In the context of Sage, create a JavaScript file in /assets/js. You then need to specify that this file should be built in with your front end processor. In Sage, you do this by providing the relevant info in manifest.json. This instructs Gulp (Sage 8.x) to process your JS file and build it into the dist directory - where it can be accessed.

Simplified manifest.json, adding test.js to our build directory:

{
  "dependencies": {
    "main.js": {
      "files": [
        "scripts/main.js"
      ],
      "main": true
    },
    "test.js": {
      "files": [ "scripts/test.js" ]
    },
    "main.css": {
      "files": [
        "styles/main.scss"
      ],
      "main": true
    },
  "config": {
    "devUrl": "http://localhost/sage-test"
  }
}

Next up, enqueue the script. Sage provides a useful helper function to access the file path: Roots\Sage\Assets\asset_path(). Depending on the location of your enqueuing script, you may need to import the namespace.

Enqueuing from /lib/extras.php might look like this:

<?php
use Roots\Sage\Assets;

add_action('wp_enqueue_scripts', function () {

  wp_register_script( 'your_custom_handle', Assets\asset_path('scripts/test.js'), ['jquery'], null, false );
  wp_enqueue_script( 'your_custom_handle' );
  wp_localize_script( 'your_custom_handle', 'yourVars', [
    'yourAjaxURL' => admin_url( 'admin-ajax.php' )
    ]
  );
}, 100);

The JavaScript

A simple jQuery AJAX handler might look like this:

jQuery(document).ready(function($) {

  var url = yourVars.yourAjaxURL;

  $('#currency').change( function() {

    var data = {
      'action': 'my_custom_action',
      'currency': $('#currency').val()
    };

    $.post(url, data, function(response) {

      console.log(response);
      alert("Currency Selected: " + response.currencySelected);

    });

  });

});

This is designed to take and submit data from an on-page form that might look like this:

<form method="post" id="currencyselector" name="currencyselector" action="">
  <select name="currency" id="currency">
    <option value = "USD">USD</option>
    <option value = "GBP">GBP</option>
    <option value = "EUR">EUR</option>
  </select>
</form>

Summary

This example is just a stripped down version - mainly it’s a memory jogger for the hook name setup. I would generally build this functionality in an Object Oriented context - so the callbacks would be held within class methods rather than the functions shown here.

Embed Google Slides in Jekyll

Mon, 26 Sep 2016 11:23:00 +0100

To embed a Google Slides presentation in a Jekyll post or page, open the presentation and select “Publish to Web” from the “File” menu.

Choose the options you require, then select the “Embed” tab. You’ll be provided with an embed code for an iframe, that you simply paste into your Jekyll post or page. Generate your site as usual, and the iframe with your presentation will be embedded. When Jekyll parses markdown files, it will render any HTML that it finds.

It won’t be responsive out of the box, so you’ll need a bit of CSS. I’ve shown this inline, but you could add the CSS to your stylesheet.

<style>
.responsive-wrap iframe{ max-width: 100%;}
</style>
<div class="responsive-wrap">
<!-- this is the embed code provided by Google -->
  <iframe src="https://docs.google.com/presentation/d/1F0DQTNPg3YG_By6LMGcgwT3icJ3eMhCiupAZm76CIfE/embed?start=false&loop=false&delayms=3000" frameborder="0" width="960" height="569" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe>
<!-- Google embed ends -->
</div>

Working Example

Laravel Front End Assets Quickstart

Thu, 22 Sep 2016 20:41:00 +0100

Laravel is pre-configured to use Node and Gulp (wrapped in Laravel elixir) to manage and build front-end assets.

Node is used to manage dependencies.

Quick Start Bootstrap

In the project root run npm install to fetch the packages defined in package.json
Enter gulp to run the default elixir task
Bootstrap SASS files will be built in to resources/assets/sass/app.scss
There is a font @import rule in resources/assets/sass/app.scss - more efficient to import fonts via HTML
Starter variables file at resources/assets/sass/variables.scss

Set Node Version

A modern node is required - if necessary use nvm to load in a suitable version:

nvm ls # List available local node versions
nvm ls-remote # List available remote node versions
nvm install v6.5.0 # Install v6.5.0
nvm use v6.5.0 # Use v6.5.0

See nvm for more.

Versioning

Amend Gulpfile.js:

const elixir = require('laravel-elixir');

require('laravel-elixir-vue-2');

/*
 |--------------------------------------------------------------------------
 | Elixir Asset Management
 |--------------------------------------------------------------------------
 |
 | Elixir provides a clean, fluent API for defining some basic Gulp tasks
 | for your Laravel application. By default, we are compiling the Sass
 | file for your application as well as publishing vendor resources.
 |
 */

elixir(mix => {
    mix.sass('app.scss')
       .webpack('app.js')
       .version(['css/app.css', 'js/app.js']); // app.css and app.js are now cache-busting;
    // Add glyphicons from Bootstrap
    mix.copy('node_modules/bootstrap-sass/assets/fonts/bootstrap/','public/build/fonts/bootstrap');
    mix.browserify('app.js');
});

Reference the versioned CSS file in a blade view - possibly resources/views/partials/head.blade.php:

<link rel="stylesheet" href="{{ elixir ('css/app.css') }}" media="screen" title="no title" charset="utf-8">

Reference the versioned JavaScript file in a blade view - possibly resources/views/partials/footer.blade.php:

<script src="{{ elixir ('js/app.js') }}"></script>
@stack('script-footer')

Laravel Form Validation Primer

Thu, 22 Sep 2016 16:37:00 +0100

Form validation logic can be held in an extension of the Request class.

The Request class allows your script to get an instance of the current HTTP request. This is achieved by type hinting the Request class on the controller method. Dependency injection of The HTTP request then happens automatically.

Fortunately, Laravel has a built-in Artisan command that facilitates the generation of custom requests.

Create a Custom Request

Create a new form request class using Artisan:

php artisan make:request ArticleRequest

This stubs out a form request class under /app/Http/Requests - in this case the new file is: app/Http/Requests/ArticleRequest.php.

The file contains a new ArticleRequest class, which extends FormRequest.

FormRequest in turn extends the basic Laravel Request class - which in turn extends the Symfony Request component.

It’s probably better to keep the class name a bit general - ArticleRequest is probably better than CreateArticleRequest - as the same validation logic can be used for forms used to create and edit resources.

Set Up Validation Rules

The new form request class has an authorize() and a rules() method stubbed out. The authorize() method can be used to check if the user is authorized to make the request - we might add a check for article ownership here.

The rules() method allows you to set rules which must be passed. The controller form processing method will receive an instance of the request class - and will not progress if any rules fail.

Example Rules with Custom Error Messages

<?php

// app/Http/Requests/CreateArticleRequest.php

...
  /**
   * Get the validation rules that apply to the request.
   *
   * This method should return an array of validation rules.
   *
   * @return array
   */
    public function rules()
    {
      return [
        // The 'title' field is required and must be at least 10 characters
        'title' => 'required|min:10',

        // The 'body' field is required
        'body' => 'required',

        // The 'published_at' field is required and must be a date
        'published_at' => 'required|date'
      ];

      // If necessary, create a $rules array to return (rather than returning
      // the array directly). This will allow conditional validation to be
      // built in - for example, a form that updates a record may require
      // additional (or relaxed) validation rules. These can be added
      // conditionally to the $rules array before it is returned.
    }

    /**
     * Override the built-in messages.
     *
     * @return array
     */
    public function messages()
    {
        return [
            'title.required' => 'Er, you forgot to add a title!',
            'title.min' => 'Don\'t be stingy - make the title LONGER THAN 3 CHARACTERS!',
            'body.required' => 'No body, no article.'
        ];
    }

In the Controller

The new form request instance is passed into form processing methods on the Controller.

Dependency injection happens automatically by means of Laravel’s service container, but this can only take place if the request class is type-hinted on the controller method.

The store/update methods will not proceed until validation rules pass.

Example:

<?php
// app/Http/Controllers/ArticlesController.php
...
  /**
   * Store a newly created resource in storage.
   *
   * @param  \Illuminate\Http\Request  $request
   * @return \Illuminate\Http\Response
   */
  public function store(Requests\ArticleRequest $request)
  {

      $request = $request->all();
      Article::create($request);

      return redirect('articles');
  }

...

References

Laravel Accessors - Format or Prepare Data for Access

Wed, 21 Sep 2016 10:56:00 +0100

Laravel Accessors are Model methods that allow Eloquent attributes to be formatted or processed when they are retrieved.

To set up an Accessor, add a suitable method in the Model class. The database column is determined by the method name, which hence needs to be in a certain format:

get{ColumnName}Attribute

Note that the column name needs to be written in StudlyCaps (PascalCase) - the first letter of each sub-word (including the first) is capitalised.

An accessor that takes a string saved in the post_title column and converts it to uppercase in the Eloquent attribute would look like this:

<?php
public function getPostTitleAttribute($title)
    {
      $this->attributes['post_title'] = strtoupper($title);
    }

The method receives the value of the attribute, and is automatically called when the value of the attribute is retrieved.

In the view for this example:

@foreach($posts as $post)
  <h2>{{$post->title}}</h2>
@endforeach

…titles will be capitailsed.

When the attribute is accessed, it will be in uppercase. The string stored in the database remains unchanged.

References

Laravel Docs: Accessors & Mutators

Laravel Mutators

Wed, 21 Sep 2016 09:48:00 +0100

Laravel mutators are Model methods that process data in some way before it is stored in the database.

Potential use cases:

Capitalising names before saving
Title-casing titles
Slugifying slugs
Change date strings to Carbon objects
Hashing passwords
JSON encoding (possible in MySQL 5.7.8, coming soon in MariaDB)

To set up a Mutator, add a suitable method in the Model class. The database column is determined by the method name, which hence needs to be in a certain format:

set{ColumnName}Attribute

Note that the column name needs to be written in StudlyCaps (PascalCase) - the first letter of each sub-word (including the first) is capitalised.

A mutator that converts a string to title case, saved in the post_title column would look like this:

<?php
public function setPostTitleAttribute($title)
    {
      $this->attributes['post_title'] = ucwords($title);
    }

Note that the method doesn’t return anything - it just directly accesses the atribute and “mutates” the value.

References

Resource Routes - RESTful Routes in Laravel

Mon, 19 Sep 2016 14:36:00 +0100

Laravel allows you to quickly and easily set up controllers and routes so that you can interact with data in a RESTful way..

Create a Model and Database Table

Create a Model in the usual way (the -m flag creates a migration for the model, allowing you to set up a new database table):

php artisan make:model Article -m

Amend the migration as necessary and run php artisan migrate to set up the new table.

Create a Resourceful Controller

Create a controller using Artisan using the --resource option:

php artisan make:controller ArticleController --resource

This creates a new controller under app/Http/Controllers, with stubbed out methods to allow interaction with the database table for this resource:

index(): Display a listing view for the resource
create(): Show the form for creating a new resource
store(): Store a newly created resource in storage
show(): Display the specified resource
edit(): Update the specified resource in storage
destroy(): Remove the specified resource from storage

Ruby on Rails has avery similar command: rails g scaffold_controller Article - which stubs out a similar controller.

Create Route for the Resource

In your routes/web.php file (Laravel 5.3), create a new route using the resource() method:

Route::resource('articles', 'ArticleController');

If you check the registered routes now using php artisan route:list, you’ll see the new (named) routes automatically set up:

+--------+-----------+-------------------------+------------------+------------------------------------------------+--------------+
| Domain | Method    | URI                     | Name             | Action                                         | Middleware   |
+--------+-----------+-------------------------+------------------+------------------------------------------------+--------------+

|        | GET|HEAD  | articles                | articles.index   | App\Http\Controllers\ArticleController@index   | web          |
|        | POST      | articles                | articles.store   | App\Http\Controllers\ArticleController@store   | web          |
|        | GET|HEAD  | articles/create         | articles.create  | App\Http\Controllers\ArticleController@create  | web          |

|        | GET|HEAD  | articles/{article}      | articles.show    | App\Http\Controllers\ArticleController@show    | web          |
|        | PUT|PATCH | articles/{article}      | articles.update  | App\Http\Controllers\ArticleController@update  | web          |
|        | DELETE    | articles/{article}      | articles.destroy | App\Http\Controllers\ArticleController@destroy | web          |
|        | GET|HEAD  | articles/{article}/edit | articles.edit    | App\Http\Controllers\ArticleController@edit    | web          |

The resource has been set up so that the correct controller methods are triggered when you access a specified route with a given HTTP verb.

Mount and Transfer Data From an Encrypted Filesystem in Ubuntu

Sun, 18 Sep 2016 16:15:00 +0100

This article describes the steps necessary to recover data from a LUKS encrypted filesystem under Ubuntu 16.04. We needed to also move encrypted user home directories.

Mount the old encrypted disk:

Identify the LUKS encrypted volume
Open device/decrypt
Mount the decrypted filesystem
Copy data from source to destination

Mount Encrypted LVM Logical Volume

Identify the encrypted device:

sudo fdisk -l

# Follow with:
sudo lsblk -f /dev/sdb

sudo file -s /dev/sdb3

# Output:
/dev/sdb3: LUKS encrypted file, ver 1 [aes, xts-plain64, sha256] UUID: xxxx

Open the Encrypted Device

sudo cryptsetup luksOpen /dev/sdb3 encrypted_device
# This returns:
Enter passphrase for /dev/sdb3:

# Enter the encryption passphrase at this point

You then need to identify the volume group and list logical volumes:

sudo vgdisplay --short

# Typical return:
"ubuntu-vg" 464.78 GiB [464.78 GiB used / 0    free]

# List logical volumes:
sudo lvs -o lv_name,lv_size -S vg_name=ubuntu-vg

# Typical return:
LV     LSize
  root   456.86g
  swap_1   7.91g

At this point, you may hit trouble - if you are working on a Ubuntu logical and trying to mount a Ubuntu logical volume, it’s likely that they’ll have the same volume group name (“ubuntu-vg”).

To fix this, you can rename one of the volumes - but beware - this may prevent that disk from booting without major intervention.

Rename the logical volume:

# Get the UUID
sudo vgdisplay

# Typical return
--- Volume group ---
  VG Name               ubuntu-vg

  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               464.78 GiB
  PE Size               4.00 MiB
  Total PE              118983
  Alloc PE / Size       118983 / 464.78 GiB
  Free  PE / Size       0 / 0
  VG UUID               xxxx

# Rename by referencing the Volume Group UUID:
sudo vgrename -v XcFAZ7-3iBG-BnQ7-GoS4-egLP-29YY-u8mcXY old-drive

# The volume group is now "old-drive"

You then need to activate the desired volume group - in this case, root on old-drive:

# Check the new volume name
sudo vgdisplay --short

# Check the logical volumes on this volumne
sudo lvs -o lv_name,lv_size -S vg_name=old-drive

# Activate the desired root
sudo lvchange -ay old-drive/root

Mount the Encrypted Filesystem

sudo mkdir /mnt/old-drive
sudo mount /dev/old-drive/root /mnt/old-drive

The decrypted filesystem is now available under /mnt/old-drive.

In our case, this system contains user home directories that are also encrypted. These need to be moved to /home on the destination filesystem.

On the new filesystem, don’t create users. First, move the encrypted directories to the usual location. For example:

# Move a user home directory
sudo rsync -ra --progress /mnt/old-drive/home/rory /home

# Also move the Apache root directory:
sudo mkdir -p /var/www
sudo rsync -ra --progress /mnt/old-drive/var/www/html /var/www

Then create corresponding users. The -m flag on useradd denotes that a home directory will be created if one doesn’t already exist. In our case, we’ve just created the directory by moving the old one into position from the old drive.

# Create the user "rory"
sudo useradd -m rory
sudo passwd rory
sudo adduser rory sudo

The user can now log on using their exisiting passphrase for decryption.

GRUB Problems

I encountered a major problem when completing this - GRUB became borked and the system would not boot - it hung on a initramfs prompt (which was pretty useless - don’t waste time in this limited shell).

The Initramfs prompt suggests that GRUB 2 began the boot process (the initial Ubuntu loading screen was visible) - but couldn’t pass control to the OS.

In our case, I suspect this was related to renaming the logical volume on the old disk.

The solution involved:

Booting Ubuntu from a live disk
Decrypting and mounting the main volume
Downloading and running ‘boot-repair’

Running boot-repair on an encrypted volume requires a bit of work - the volume needs to be properly decrypted and activated. In particular, you need to activate the encrypted drive using the correct name - I determined this by trial and error, with some clues from this useful article.

The procedure:

In the live environment, navigate to /etc/crypttab and take a peek:

cat /etc/crypttab

# Typical output:
sdb3_crypt UUID=xxxxx none luks,discard
cryptswap1 UUID=xxxxx /dev/urandom swap,offset=1024,cipher=aes-xts-plain64

This gives you the name that you need for your cryptsetup - in this case, sdb3_crypt. Decrypt the volume using this name:

cryptsetup luksOpen /dev/sdb3 sdb3_crypt
# Enter decryption passphrase

# Activate:
sudo lvchange -ay sdb3_crypt/root

Then download and run Boot Repair - this should allow you to reinstall GRUB 2:

sudo add-apt-repository ppa:yannubuntu/boot-repair
sudo apt-get update
sudo apt-get install -y boot-repair && boot-repair

References

Check Site Backup Integrity: WordPress/PHP MySQL

Fri, 16 Sep 2016 11:28:00 +0100

If you’ve set up a backup solution for WordPress or other dynamic PHP websites, you will probably be backing up site files as well as the site database. For a proper backup solution, you need to check that the backup copy is viable.

You may have a copy of the site files, along with a (hopefully properly) dumped database, but unless you connect these up, how do you know that your backup copy is sound?

The integrity of your backups is not something that you should discover during an emergency recovery situation.

Manually rebuilding a working copy of a dynamic website is time consuming. For each site database, internal site URLs all relate to production domains, complicating the rebuild process. When you have a server backup with ten or twenty important client sites, verification of backups looks pretty daunting - and I suspect that a lot of people just don’t bother.

This article describes how to partially automate this process.

If you want to hack on this, get the files on GitHub.

Context

In our case, site files from the Apache root of a production server are backed up incrementally on a daily basis to a date-stamped directory. This contains:

A subdirectory html - which in turn contains a subdirectory for each site under the document root
A subdirectory sql which contains a collection of dumped databases for the sites in question

Important config files are also backed up, but that is beyond the scope of this article.

Production server backup procedure
Secure rsync between servers - describes consolidating backup copies on a central backup server
Rsync between backup server and local machine

This article assumes that the backup has been downloaded to a local machine.

Checking Integrity: Overview

To test the integrity of backed up sites, one option is to build working clones of the sites on a virtual machine. To avoid the need to change URLs on the backup copies, the /etc/hosts file is amended on the guest VM.

Obviously, the guest VM needs to run a server that broadly matches the original backed up server (in this case Apache), and the virtual hosts settings for the guest VM server need to be set up correctly (this is a one-time import from the backed-up config directory).

You don’t necessarily need to use a VM - you could use any machine on the local network. The reason this is done on a VM/separate machine is so that the main host computer can access the actual live sites for maintenance purposes.

This method also keeps seperation between backed up clones and ongoing development websites - which are two different things.

This article assumes that a backup archive is available. Building working copies involves:

One-time setup of a suitable Virtual Machine - in this case, a Ubuntu Xenial, Apache, MariaDB and PHP 7 LAMP stack
A one-time import of relevant database users to the VM
Exporting files from the Host machine backup archive to the Guest VM (run command in Host)
Importing databases in the Guest (run command in Guest)

Aims

Check the integrity of multiple site backups by building working local copies. This is achieved by:

Moving site files and databases for a backed-up production server from a host machine into a local virtual machine
Import MySQL/MariaDB databases and set up working sites on the VM

Backup integrity should be checked regularly, so this should be a simple process.

Ideally, once the system has been setup it should be run by administrators rather than developers.

Requirements

These BASH scripts have been tested on Ubuntu Xenial Xerus 16.04 Desktop.

Zenity is used to create user dialogues.

VirtualBox is required for the Virtual Machine. In this case, the VM runs Ubuntu 16.04 Xenial Xerus desktop - desktop rather than server because it allows easy checking of the moved sites. To achieve this, the guest machine hosts file (/etc/hosts) must be set up properly to point at the local copies.

The VM also runs Ubuntu 16.04 Desktop. The database server is MariaDB, but the commands would work on a standard MySQL database server.

The sql backups directory includes the performance_schema.sql, phpmyadmin.sql, mysql.sql and log files from the original server. These aren’t necessary to build clones from backups, and if imported will probably mess up the VM MySQL configuration. Because of this, we exclude these files from the transfer - see the sql-verification-exclude file in the linked repo for an example.

Note: For the backed up sites on the guest machine to work properly, the MySQL users from the original server should be imported in a one-time operation.

Move Files to the VM

This is achieved with the move-backups script. This script prompts the user to choose a directory to move. The script is tightly coupled to our requirements, but would be easy to amend.

The directory to be moved is a datestamped directory that contains the entire html directory (i.e. document root) from a backed-up Apache server. It also contains backed up MySQL files (originally created by mysqldump) in a sql directory.

Move Backups Script

Run on the Host computer.

#!/bin/bash
#
# Move a directory into a local Virtual Machine for testing purposes.
#
# This file should be executable and in your path. E.g.:
# - `mv move-backups /usr/local/bin`
# - `chmod +x /usr/local/bin/move-backups`
# - Run: `move-backups`
#
# Add username@network-IP for your VM in place of `david@192.168.1.145`
# Add root@network-IP for your VM in place of `root@192.168.1.145`
# ------------------------------------------------------------------------------
STORAGE=/media/david/storage/servername
SQL_DESTINATION=david@192.168.1.145:staging
HTML_DESTINATION=root@192.168.1.145:/var/www/html
VM=Xenial
SQL_EXCLUDE=/media/david/storage/sql-verification-exclude # rsync excludes are controlled in a file

cd $STORAGE;

# Select the Directory to move
# ------------------------------------------------------------------------------
zenity --info \
--text="Begin the build process for backup up client websites. Click \"OK\" to begin. Then select the date-stamped directory in the backup storage area."

SOURCE=`zenity --file-selection --directory --title="Select a Directory to Sync"`

case $? in
  0)
  echo "\"$SOURCE\" selected.";;
  1)
  echo "No file selected.";;
  -1)
  echo "An unexpected error has occurred.";;
esac

# Start the Virtual Machine - for headless, append --type headless
# ------------------------------------------------------------------------------
VBoxManage startvm "$VM" --type headless | zenity --progress \
  --pulsate --width="320" --height="150" \
  --text="Starting $VM Virtual Machine" \
  --title="Please Wait while $VM is started" --auto-close

# Sync HTML directories INDIVIDUALLY
# ------------------------------------------------------------------------------
HTML_DIRS=($SOURCE/html)

# Loop through Directories only
for DIR in $HTML_DIRS/*/; do

  # For our rsync setup, the source directory MUST NOT have a trailing slash -
  # so that if the directory doesn't exist, it will be created.
  SOURCE_DIR=${DIR%/}

  # basename of the $DIR - used as the destination directory, under `/var/www/html`
  DEST_DIR= $(basename $DIR)

  rsync -azv --progress --delete $SOURCE_DIR $HTML_DESTINATION/$DEST_DIR | zenity --progress \
    --pulsate --width="320" --height="150" \
    --text="Syncing the HTML directory: $SOURCE_DIR" \
    --title="Please Wait" --auto-close

done

# Sync SQL backups to a staging directory
# ------------------------------------------------------------------------------
rsync -azv --exclude-from=$SQL_EXCLUDE --progress --delete $SOURCE/sql/ $SQL_DESTINATION/sql | zenity --progress \
  --pulsate --width="320" --height="150" \
  --text="Syncing the SQL directory" \
  --title="Please Wait" --auto-close

# Tidy up
zenity --question \
--text="Sync complete. Do you want to shut down the VM?"

case $? in

  0)
  echo "0"
  # Close up the VM, maintain state
  VBoxManage controlvm $VM savestate | zenity --progress \
    --pulsate --width="320" --height="150" \
    --text="Shutting down $VM Virtual Machine" \
    --title="Please Wait while VMs are Saved" \
    --auto-close
    zenity --info\
     --window-icon="info" \
     --text="The VM $VM has been shut down."
    echo "$vm was closed to a saved state"

  ;;

  1)
  echo "1"
  zenity --info\
  --window-icon="info" \
  --text="Your VM $VM is Running - though it may be in a headless[GitHub repo with scripts](https://github.com/DavidCWebs/check-backups).

  -1)
  echo "An unexpected error has occurred."
  ;;

esac

Usage:

Add move-backups to usr/local/bin on the Host computer: mv move-backups /usr/local/bin
Make executable: chmod +x /usr/local/bin/move-backups
Run move-backups in a terminal and follow instructions

When prompted, you should select a directory that contains the backed-up html directory from the Apache doc root - the directory that is normally located at /var/www/ in a standard Apache setup.

Note that the moved files won’t do anything unless you also import the associated databases on the guest machine.

Import Databases

Add import-databases to usr/local/sbin on the Guest computer/VM: mv import-databases /usr/local/sbin
Make executable: chmod +x /usr/local/sbin/import-databases
Run sudo import-databases in a terminal on the Guest VM

#!/bin/bash
#
# The purpose of this script is to import databases so that working copies of
# backed up PHP/WordPress websites can be quickly and easily checked.
#
# The script loops through all databases in a staging directory and imports them
# into MySQL/MariaDB. Existing databases having the same name will be overwritten.
#
#-------------------------------------------------------------------------------

SOURCE=/home/david/staging/sql
PASSWORD=thenicelongpassword
DATABASES=($SOURCE/*)

for (( i = 0; i < ${#DATABASES[@]}; i++ )); do

  # The file extension - in our case, there are *.log files that should be ignored
  EXT=${DATABASES[$i]#*.}

  if [[ "sql" == $EXT ]]; then

    DB_SOURCE=${DATABASES[$i]}
    DB_NAME=$(basename ${DATABASES[$i]} .sql)

    # If a Databse exists with this name, DROP it
    mysql --user=root --password=$PASSWORD -e "DROP DATABASE IF EXISTS \`$DB_NAME\`"

    # Create new DB with the name of the DB backup file
    mysql --user=root --password=$PASSWORD -e "create database \`$DB_NAME\`; GRANT ALL PRIVILEGES ON \`$DB_NAME\`.* TO root@localhost IDENTIFIED BY '$PASSWORD'"

    # Import the Database
    mysql --user=root --password=$PASSWORD $DB_NAME < $DB_SOURCE

  fi

done

# Set proper ownership of site files
chown -R www-data /var/www/html/*

TODO

These scripts are a good start, and allow us to build and check backup copies quite easily. There is room for further automation - ideally we’d like the process to be fully automated, integrated naturally into the backup process.

Our setup includes passwordless SSH keys which allows for easier rsync’ing, and this has not been documented.

Other enhancements might include:

Prevent selection of the ‘wrong’ backup directory
Auto creating the staging directory for the sql files transfer
Trigger the import-databases script from the host, so working copies are built with a single command
Better feedback on the import-databases script (there’s none at the moment!)
Document how to import users from original server to the guest machine

Pull requests welcome.

Resources

Display WordPress Term Links

Thu, 15 Sep 2016 14:28:00 +0100

Display a list of links to associated WordPress terms for a given taxonomy/custom taxonomy.

Two options for class methods to achieve this:

<?php
...

  /**
   * Output post term links.
   *
   * The `get_the_terms()` function returns an array of term objects for the
   * given taxonomy.
   *
   * @uses get_the_terms()
   * @param  string The custom taxonomy to target
   * @return string HTML markup for a list of term links
   */
  public function the_terms( $taxonomy = 'category' ) {

    $terms = get_the_terms( $this->post_ID, $taxonomy );

    ob_start();

    foreach ($terms as $key => $term) {

      echo '<a href="' . get_term_link($term) . '">' . $term->name . '</a>';
      echo ( $key !== count( $terms ) -1 ) ? ", " : NULL; // Comma separated, but not last item

    }

    echo ob_get_clean();

  }

  /**
   * Alternative method.
   *
   * The `wp_get_object_terms()` function retrieves the terms associated with
   * the given object(s) for the specified taxonomy.
   *
   * @uses wp_get_object_terms()
   * @param  string The custom taxonomy to target
   * @return string HTML markup for a list of term links
   */
  public function alt_post_terms( $tax = 'category' ) {

    $post_terms = wp_get_object_terms( $this->post_ID, $tax, array( 'fields' => 'ids' ) );

    if ( ! empty( $post_terms ) && ! is_wp_error( $post_terms ) ) {

      $term_ids = implode( ',' , $post_terms );

      $terms = wp_list_categories(
        [
          'title_li' => '',
          'style'    => 'none',
          'echo'     => false,
          'taxonomy' => $tax,
          'include'  => $term_ids
        ]
       );

      $terms = rtrim( trim( str_replace( '<br />',  ',', $terms ) ), ',' );

      return $terms;

    }

  }

...

You could also simply drop this into the template, within the loop to return a list of links to ‘project-category’ terms for the current post:

<?= get_the_term_list( $post->ID, 'project-category', 'Project Categories: ', ', ' ); ?>

References

Target the Last Item in a PHP foreach Loop

Thu, 15 Sep 2016 11:53:00 +0100

To target the last item in a PHP foreach loop without using a counter, check for count( $array ) -1 within the loop.

This simple example assigns appropriate classes to the rendered elements:

<?php

foreach ($array as $key => $value) {

  $class = ( $key !== count( $array ) -1 ) ? " class='not-last'" : " class='last'";

  echo "<div{$class}>";
  echo "$value['the_title']";
  echo "</div>";

}
?>

WordPress Pagination with WP_Query - Page Slug Causing Error

Wed, 14 Sep 2016 20:16:00 +0100

Pagination can be tricky. This post outlines a problem that can arise due to a page slug that conflicts with a custom post type rewrite base.

Hopefully this post will help me to avoid repeating the error in a months time (I’ve just spent an hour fixing this). If you find this post after a Google search for “WordPress pagination 404 error on page 2” - hopefully you’ll find it useful.

Scenario

A static page that calls WP_Query() to build a custom loop. In this case, the page is called ‘People’ (the page slug is ‘people’) and WP_Query() builds a loop of ‘person’ custom post types.

Pagination works, but clicking the link to “page 2” results in a 404 error.

The Custom Loop

I use a wrapper to simplify generation of WP_Query().

Rather than hunt through the wrapper classes (which apply overrides by merging arrays), it’s usually easiest to check the arguments passed in to WP_Query() by examining them just before they are used:

<?php
// Log the args to `debug.log`
error_log( json_encode($args) );

// The Custom Query
$custom_query = new \WP_Query( $args );

This outputs the arguments to debug.log. Your site must be configured to log errors - your config should include this:

<?php
define('WP_DEBUG', true);
define('WP_DEBUG_LOG', true);

…and wp-content/debug.log should be writable by your server process. In my case (Ubuntu 16.04, Apache 2.4), the file should be owned by www-data.

The Arguments passed to WP_Query() in this case are as follows (JSON encoded for readability):

{
  "post_type":["person"],
  "post_status":["publish"],
  "posts_per_page":2,
  "order":"ASC",
  "orderby":"menu_order",
  "nopaging":false,
  "paged":1
}

These aruments look pretty OK to me.

The Permalink

Switching permalinks to WP plain option fixes the issue. This is a clue.

It’s not a solution, because the URLs look terrible.

The Custom Post Type

In this case, the custom post type “person” had a rewrite rule so that the individual custom post type URLs were: http://example.com/people/joe-bloggs/.

And that’s the issue right there.

You can verify this by temporarily amending the page slug for the static page that calls the custom loop. Do this so that is is anything other than ‘people’ and pagination will work.

Solution

Options:

Amend the rewrite rule for the custom post type
Amend the page slug

I went with option 1. The client can live with http://example.com/person/joe-bloggs - the whole point of the static page with custom loop is to have a more controllable (pseudo) archive page, so the rewrite is pretty redundant - the http://example.com/people was originally set up as a more semantically meaningful archive URL.

Upgrading Letsencrypt Auto to Certbot

Tue, 13 Sep 2016 20:38:00 +0100

To upgrade from an older LetsEncrypt to Certbot on Ubuntu 14.04, install Certbot and run the auto-upgrading script.

Install Certbot

wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto

Move the script to a suitable location:

sudo mv certbot-auto /opt

Run the Installer

Move into the containing directory and run the installer:

cd /opt
./certbot-auto

Renew Certs

Certbot picks up on existing configurations pretty well. To run the auto-renew dialog:

sudo ./certbot-auto renew

# or:

sudo ./certbot-auto renew --dry-run

This brings up an ncurses window.

Note that all domains on the server will be pre-selected for domain renewal.

Be careful here - if you inadvertently leave a domain selected where the site hasn’t been set up for SSL, you can end up with a broken site. This is because all internal links will be pointing to the “http://” version, and Certbot will have created and enabled a vhosts configuration for the “https://” version.

After selecting domains for cert renewal/registration, you will also get the option to choose whether HTTPS access is required or optional. “Required” is generally a good choice.

Success message:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/example.com/fullchain.pem. Your cert
   will expire on 2016-12-12. To obtain a new or tweaked version of
   this certificate in the future, simply run certbot-auto again with
   the "certonly" option. To non-interactively renew *all* of your
   certificates, run "certbot-auto renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

PHP 7 Null Coalescing Operator

Sat, 10 Sep 2016 10:10:00 +0100

One of the goodies in PHP 7 is the null coalescing operator.

This provides an easy method of setting a variable if it exists (i.e. is not null), or setting it to a default if null.

Remember - PHP 7 is required!

Examples

The expression (expr1) ?? (expr2) evaluates to expr2 if expr1 is NULL, and expr1 otherwise — PHP Manual

<?php
// Null coalescing operator
$foo = $_POST['bar'] ?? 'baz';

// The pre PHP 7 way!
if(isset($_POST['bar'])){
  $foo = $_POST['bar'];
} else {
  $foo = 'baz'
}

// ...or pre-PHP 7 with a ternary:
$foo = isset($_POST['bar']) ? $_POST['bar'] : 'baz';

This is going to make setting defaults much neater and more readable.

Using Masonry With WordPress and Sage 8

Wed, 07 Sep 2016 09:46:00 +0100

Masonry is a very popular open-source grid layout library by David DeSandro. It provides a great way of laying out client images, especially when the sizes and aspect ratios are all over the place.

You can use Masonry as a jQuery plugin, or initialize it with vanilla JavaScript - though I haven’t tried the latter.

Installing Masonry when you’re using the Sage starter theme is probably best done with Bower. There are a few important issues that you need to be aware of.

Most importantly, you need to reference the masonry.pkgd.js file (rather than masonry.js) from the Masonry bower package for the script to run properly.

Install Masonry with Bower

In the project root, build Masonry with Bower:

bower install masonry --save

You may also need the imagesloaded plugin:

bower install imagesloaded --save

Conditionally Load Masonry

Masonry is generally only required on some pages, so we need to prevent masonry being auto-built into main.js - to do this, we need to amend assets/manifest.json:

manifest.json

{
  "dependencies": {
    "main.js": {
      "files": [
        "scripts/main.js",
        "scripts/sticky-navbar.js"
      ],
      "main": true,
      "exclude": {
        "bower": [
          "outlayer",
          "fizzy-ui-utils",
          "get-size",
          "ev-emitter",
          "masonry",
          "imagesloaded"
        ]
      }
    },
    ...
    "masonry.js": {
      "bower": [
        "masonry",
        "imagesloaded"
      ],
      "files": [
        "scripts/masonry-control.js"
      ]
    }
...

…now the bower packages for masonry & imagesloaded, along with the masonry-control.js script will be concatenated into dist/masonry.js. This file can then be enqueued where it is needed.

Masonry Dependencies

Masonry needs the masonry.pkgd.js file to pull in necessary dependencies. Don’t try to do this manually - it will result in a mess.

Instead set up an override in the project bower.json:

bower.json

{
  ...
  "overrides": {
    "bootstrap-sass": {
      "main": [
        ...
      ]
    },
    "font-awesome": {
      "main": [
        "./scss/font-awesome.scss",
        "./fonts/\*"
      ]
    },
    "masonry": {
      "main": ["./dist/masonry.pkgd.js"],
      "dependencies": {}
    }
  }
}

The Future: Sage 9

This article probably has a short shelf-life - Sage 9 is coming up, and this will drop Bower entirely - project dependencies will be managed by Webpack/NPM.

Laravel Models with Artisan

Sun, 28 Aug 2016 11:52:00 +0100

Laravel is an open source MVC framework for PHP. Laravel uses the concept of models to store and manage data. Laravel has a command line interface called artisan that amongst other things allows you to easily set up model stubs. The framework uses the concept of “migrations” to manage changes to the app database - and migrations can be created in conjucntion with models.

Create a Database

If you don’t already have one set up, your project will need a database:

Set up a new database with a unique user. Add database user credentials to Laravel’s /.env file.

# Create Database
mysql -u root -p -e "create database new_DB; GRANT ALL PRIVILEGES ON new_DB.* TO new_DB_user@localhost IDENTIFIED BY 'nicelongpassword'"

Create a Model

To make a model called “Experiment”:

php artisan make:model Experiment

Running this command stubs out a basic model under the /app directory.

Make a model with a corresponding migration:

php artisan make:model Experiment -m

Create DB Tables

Run the migrations to set up DB tables:

php artisan migrate

Migration Problems

Sometimes the migration fails because the laravel.log file is not writable by the current user. It needs to be writable by the server user (www-data in the case of Ubuntu/Apache), but the migration is run by your current user. You’ll get an error message like this:

david@david-desktop:$ php artisan migrate
PHP Fatal error: Uncaught UnexpectedValueException:
The stream or file "/var/www/html/your-app/storage/logs/laravel.log"
could not be opened: failed to open stream: Permission denied in
/var/www/html/your-app/vendor/monolog/monolog/src/Monolog/Handler/
StreamHandler.php:107

To fix this you could add your user’s primary group to the file ownership:

sudo chown -R www-data:$USER storage/logs/laravel.log

# Allow the file to be group-writable:
sudo chmod 664 storage/logs/laravel.log

This is probably pretty safe on a local development environment - and in most cases I think would be acceptable for production - you’re giving additional write access to your primary user only after all - and if this user has been compromised, file permissions on a log file are the least of your worries.

TODO: It would probably be better to set this up as an ACL as described in the Symfony documentation

References

Set Up & Seed Laravel Database

Sat, 27 Aug 2016 18:53:00 +0100

These instructions are for local development of Laravel on Ubuntu 16.04 Xenial Xerus (Desktop). The instrauctions have been tested on 10.1.16-MariaDB, but they should also work for MySQL.

Create Database

Create a Database with a unique user:

mysql -u root -p -e "create database database_name; GRANT ALL PRIVILEGES ON database_name.* TO user_name@localhost IDENTIFIED BY 'urehfh577hg5hrpefh7'"

Connect Laravel

Enter the Databse connection details in .env:

DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=database_name
DB_USERNAME=user_name
DB_PASSWORD=urehfh577hg5hrpefh7

Test the Database

Laravel comes with a couple of migrations defined out of the box.

To check that Laravel is properly connected to the new database:

# Open a mysql prompt, working on the new DB:
mysql -u root -p database_name
# Enter show tables:
SHOW TABLES;
# Returns Empty set (0.00 sec)

Open a new terminal and run the pending Laravel migrations:

php artisan migrate

This will run the default migrations (under Laravel 5.3, a “users” table and a “password_resets” table will be created, as well as a “migrations” table).

You can also seed the database. Create a seeder class:

php artisan make:seeder UsersTableSeeder

This will create database/seeds/UsersTableSeeder.php, which will be stubbed out with an empty run() method.

This method can be used to insert records into the database. Laravel includes the handy Faker factory class, which allows dummy data to be created.

To determine which fields to seed, check the table:

# At the mysql prompt:
SHOW COLUMNS FROM users;

Next, amend the run() method of the seeder class that you’ve just created:

<?php
public function run()
    {
      $faker = Faker\Factory::create();

        for( $i = 0; $i < 25; $i++ ) {

          DB::table('users')->insert([
            'password' => bcrypt('secret'),
            'name' => $faker->name,
            'email' => $faker->email

          ]);
        }
    }

Now, when you run:

php artisan db:seed --class=UsersTableSeeder

…the seeder will create 25 dummy users complete with realistic data.

Note: running php artisan db:seed will only run the default DatabaseSeeder class - which will be empty. This is a bit different to the php artisan migrate command, which runs all migrations under the database/migrations directory.

To check the seeder results, run the following command from the mysql prompt:

SELECT * FROM users;

Note that Laravel has a much better way of interacting with your database on the fly - Artisan tinker:

# Get the tinker prompt:
php artisan tinker

# Show all users:
App\User::all()->toArray();

Install Laravel on Ubuntu 16.04 Desktop

Sat, 27 Aug 2016 17:48:00 +0100

This article describes how to get Laravel up and running in a Ubuntu Xenial Xerus development environment.

Minimum requirements for Laravel:

PHP >= 5.6.4
OpenSSL PHP Extension
PDO PHP Extension
Mbstring PHP Extension
Tokenizer PHP Extension

These notes describe installing Laravel on PHP 7 with MariaDB and Apache.

Laravel Installer

Use Composer to download the Laravel installer. It will then be available globally:

composer global require "laravel/installer"

This will place the installer in ~/.composer/vendor/laravel/installer/, symlinked in ~/.composer/vendor/bin.

This latter directory shoudl be in your $PATH so that the laravel installer can be located by the system. You may need to add the follwoing to your ~/.bashrc file:

# Add to ~/.bashrc for global access to Laravel installer
export PATH=$HOME/.composer/vendor/bin:$PATH

The Laravel installer calls the Symfony\Component\Console\Application class to work it’s magic.

Installing Laravel

The development site will be located in the Apache docroot - and the installer will install Laravel in the current directory:

cd /var/www/html
laravel new project-name

Accessing http://localhost/project-name/ in a browser will display the app directory structure.

Post-Installation Permissions

Assuming the Apache user is www-data, the following ownership settings need to be applied:

# Move into the new project root
cd /var/www/html/project-name

# Apply ownership
sudo chown -R www-data storage
sudo chown -R www-data bootstrap/cache

The Public Directory

The front controller for Laravel is public/index.php file - this is the file that all requests hit. The public directory will be the document root for the project.

If you’ve set permissions correctly, you should be able to access the new app: http://localhost/project-name/public/

It’s abit cumbersome to use this URL in the development environment. You should instead configure a virtual host directive for a suitable URL, and adjust your /etc/hosts file so that your chosen domain directs to 127.0.0.1 (localhost).

Localhost and the Trailing Slash

There is a “gotcha” if you decide to devlop locally under Apache localhost - adding a trailing slash to the URL http://localhost/your-app/page/ triggers Laravel to redirect to a non-trailing slashed home URL, which will be http://localhost/page rather than http://localhost/your-app/page.

This can be worked around (maybe by means of .htaccess rules) - but it’s probably easier all round to just set up a local Apache virtual host config and amend etc/hosts to point to your localhost for a specified domain.

Post Install Checks

Check the 32 character application key in .env has been set to a random string.

Moving a Site to a New Domain - Apache Redirects/Rewrites

Fri, 26 Aug 2016 15:45:00 +0100

When moving a site to a new domain, it’s important to redirect visitors to the new domain. This provides a courtesy to your users, helps retain search engine ranking, and prevents broken links to your site.

Apache provides several methods for redirecting requests and this article documents several approaches.

The rules should be tested properly before applying them. In our case, we set up a virtual machine with cloned versions of both new and old sites (with an appropriately modified hosts file) to allow full testing of all redirects.

Context

This article refers to a site that is being moved to a new domain as part of a re-branding exercise.

The site structure will remain broadly the same, but for two quite important differences. The old site permalink structure for the ‘projects’ and ‘people’ index pages is:

http://olddomain/category/people/
http://olddomain/category/projects/

On the new site, these index pages will be located at the following URLs:

http://newdomain/people/
http://newdomain/projects/

In addition, project categories (a custom taxonomy) are to be rationalised as part of the move/rebuild. The old URL structure involved having ‘projects’ as a category, with projects further grouped into sub-categories. For the new site, these sub-categories have been mapped into a more logical ‘project-category’ structure.

The resource found on the old domain at http://olddomain/category/projects/term/ should be available at http://newdomain/project-category/term/ on the new domain.

Approach

The redirects can be achieved with a combination of Redirect and Rewrite rules, or Rewrite rules only. Rules can be added on a Directory basis (either .htaccess or in a <Directory> block in the vHost config).

A better approach would probably be to include all necessary rewrite rules outside a <Directory> block within the vHost configuration. This allows the redirect to happen without even the presence of a directory corresponding to the old domain - so when the move is complete, the old site can be collapsed completely.

Per-Directory Rules: `.htaccess`

These rules are added to a .htaccess file in the root directory of the old site.

Redirect 301 /category/people/ http://newdomain.com/people/
Redirect 301 /category/projects/ http://newdomain.com/projects/

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} !newdomain.com$ [NC]
RewriteCond %{REQUEST_URI} !^/category/people/$ [NC]
RewriteCond %{REQUEST_URI} !^/category/projects/$ [NC]
RewriteRule ^(category/projects/)(.*)$ http://newdomain.com/project-category/$2
RewriteRule ^(.*)$ http://newdomain.com/$1 [L,R=301]

Alternatively, the rewrite engine only could be used:

RewriteEngine On
Options FollowSymLinks
RewriteBase /
RewriteCond %{HTTP_HOST} !newdomain.com$ [NC]
RewriteRule ^(category/people/)$ http://newdomain.com/people/ [L,R=301]
RewriteRule ^(category/projects/)$ http://newdomain.com/projects/ [L,R=301]
RewriteRule ^(category/projects/)(.*)$ http://newdomain.com/project-category/$2 [L,R=301]
RewriteRule ^(.*)$ http://newdomain.com/$1 [L,R=301]

Note that the first string to be matched should not have a leading slash. This is because in the directory context (within either a <Directory> block or a per-directory .htaccess file) the removed prefix always ends with a slash - so a Pattern with ^/ never matches in the per-directory context.

Bracketed sections of the source string are available as indexed variables in the destination string:

...
# Rewrite http://olddomain.com/first-string/second-string/...
# ...to http://newdomain/second-string:
RewriteRule ^(first-string/)(second-string)$ http://newdomain/$2`

NOTE for testing purposes, you should designate these rules as “302” - a temporary redirect. These should be amended to “301” to denote a permanent redirect when you deploy the rules to the production server.

These rules could be placed in either a .htaccess file in the olddomain.com document root or a <Directory> block in the site virtual host configuration file for the old site (e.g. /etc/apache2/sites-available/olddomain.com.conf). If adding the rules to an Apache config file, you’ll need to restart Apache for the rules to take effect.

Simple vHost Redirect

This block added to the virtual host configuration of the old site provides a one-to-one mapping of the old domain to the new. If the site retains the exact same permalinks (apart from the domain of course), this might be an adequate solution. Note that a <Directory> block is not necessary:

<VirtualHost *:80>
        ServerName oldsite.com
        ServerAlias www.oldsite.com
        ServerAdmin info@olddomain.com

        Redirect 301 / http://newdomain.com

        ErrorLog ${APACHE_LOG_DIR}/oldsite.com.error.log
        CustomLog ${APACHE_LOG_DIR}/oldsite.com.access.log combined
</VirtualHost>

This kind of simple redirect isn’t going to work in our case, because the structure of some of our URLs will be altered.

Complete vHost Redirect

Here’s the final virtual host config file solution:

<VirtualHost *:80>
        ServerName olddomain.com
        ServerAlias www.olddomain.com
        ServerAdmin info@olddomain.com
        RewriteEngine On
        RewriteCond %{HTTP_HOST} !newdomain.com$ [NC]
        RewriteRule ^(/category/people/)$ http://newdomain.com/people/ [L,R=301]
        RewriteRule ^(/category/projects/)$ http://newdomain.com/projects/ [L,R=301]
        RewriteRule ^(/category/projects/)(.*)$ http://newdomain.com/project-category/$2 [L,R=301]
        RewriteRule ^(.*)$ http://newdomain.com/$1 [L,R=301]
</VirtualHost>

Don’t forget to restart Apache for the rewrite rules to take effect.

References

Page Sections in Jekyll - Seperating Content from Layout

Wed, 10 Aug 2016 19:23:00 +0100

It’s usually considered good practice to keep content separate from logic and layout. I think this is especially important with static site generators like Jekyll, where it’s easy to mix content and layout.

Achieving separation is pretty straightforward with single page/post/collection views - just define a custom template and inject the content from a markdown file (from yaml frontmatter and the main content field).

But what about displaying section content in a single page context? This is a common requirement for landing pages, home pages and single page sites.

Ideally, all content - including text, background image, foreground image etc should be controlled in the content markdown file. I’d go a step further and suggest that some style elements should also be controllable here - for example, if you add a background image which results in poor text/background contrast, you should be able to set the image overlay opacity in the same location that you’re setting the image.

Define Collections

Jekyll collections are an ideal candidate for managing repeating content that is not naturally date ordered.

Our use case here is for a very simple site, and our section content will be grouped using a jekyll collection that we’ll call “sections”.

To begin, define the site collections in _config.yml:

collections:
- sections

Then create a directory called _sections in the project root. This directory will contain section files, stored in markdown format.

The Index Page

Iterating through the sections in index.html looks like this:

{% assign sections = site.sections | sort: 'order' %}
{% for section in sections %}
  {% if section.include != null %}
    {% include {{ section.include }} %}
  {% else %}
    {% include sections/default.html %}
  {% endif %}
{% endfor %}

This does the following:

Assign a ‘sections’ object, by iterating through site sections (i.e. the _sections/*.md files) and ordering by the “order” attribute (which is set in the section front matter)
Loop through these objects - for each one, if there is a specified include (again, set in the section frontmatter) then call it
If no include is specified in the section, call a default include as a fallback

The Includes

In our case we store includes under _includes/sections.

The includes are html files that have the section content injected. This include shows how:

Inline style is included as defined by the section yaml frontmatter
Section content is injected into the section layout

{% capture bg-image-style %}
 style="background:{% if section.bg-opaque-overlay != nil %} linear-gradient(
    {{ section.bg-opaque-overlay }},
    {{ section.bg-opaque-overlay }}
  ),{% endif %} {% if section.bg_image != nil %}url('{{ section.bg_image }}') center center fixed; background-size: cover;{% endif %}"
{% endcapture %}
<section id="{{ section.title | slugify }}" class="content-section"{{ bg-image-style }}>
  <div class="row">
    <div class="col-md-12">
      <h1>{{ section.title }}</h1>
      {{ section.content }}
    </div>
  </div>
</section>

In this case the style rule builds in a background image with an opaque overlay.

Obviously you can define as many layout includes as your project needs.

The Content

Content is held as a collection markdown file stored in the /_sections directory.

For example, this is _sections/about.md:

---
title: About
image: path/to/img
# Set the display order for this section
order: 3
# Specify the layout for this section
include: sections/two-column.html
# Set style variables
bg-opaque-overlay: rgba(0,200,0,0.3)
bg_image: http://loremflickr.com/2000/600/robot
---
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

### Ut enim ad minim veniam

quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Duis aute irure dolor in reprehenderit in voluptate velit...

I like this method because content editing on the site is very simple, and site editors can have quite a few options available in the content markdown file. It also breaks apart the layout/HTML which I think makes projects more maintainable.

There are probably better ways of organising section content - if you know of any, please leave a comment!

References

Secure rsync Between Servers

Wed, 10 Aug 2016 12:40:00 +0100

This method allows for automatic incremental backups between servers.

Backup Server Pulls from Production Server

Either server could initiate the sync - but if Production initiates, it would need write access to the Backup server. It is probably safest to give the Backup server read access to the Production server, so this guide assumes that the Backup machine initiates the sync.

The files to be synced have been built by a local automatic rsync-based incremental backup script that runs on the Production server. This is because:

There is enough space on the Production machine (approx 8G out of 40G is in use)
Having an incremental backup on the Production server allows for quick restores in the event of non-catastrophic failure
Having backups on the production server is “belt and braces” - backups are stored in multiple locations
The script has root access and can build a complete set of backup files: /var/www/html/, Apache config files, MySQL config files, Database files (built by mysqldump)

This method uses the rrsync perl script (restricted rsync) to restrict rsync to a subdirectory declared in .ssh/authorized_keys. This works really well since our local incremental backup has written all the files we need to a single backup directory.

The Incremental Backup that is being synced is described here.

Set up a Backup User on the Production Server

Set up a user without sudo privileges. This user is only going to be used to store backups - which provides another level of security.

The backup user home directory will receive incremental backups, and hold the SSH public key to allow the remote backup server to connect - and that’s about it.

Set up an SSH Key Pair on the Backup Server

Because the backup script is designed to be triggered automatically by a cron job, the user that will be running the rsync command on the production server will be root.

It is therefore important to transfer the public SSH key for the root user to the backup target machine.

In addition, the SSH key should NOT have a passphrase, so that connection can be made automatically. We will also limit the actions that can be carried out by this SSH connection.

# Switch to root
su root

# Generate an SSH key pair, WITHOUT a passphrase
ssh-keygen -t rsa

# copy the public key to the backupuser account on the target Production machine
ssh-copy-id -p 1234 backupuser@123.456.789.0

Failure to Copy Public Key

If the public key doesn’t copy across, you’ll see an error message like this:

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Permission denied (publickey).

This means that the remote server has refused the connection.

In my case, this is because I usually set up servers such that access is via SSH key only - with password authentication disabled.

To copy a public SSH key to a remote server, password SSH login must be enabled on the remote server

To enable password login:

SSH into the remote server
Configure SSH: sudo nano /etc/ssh/sshd_config
Edit to allow password authentication: PasswordAuthentication yes
Save & exit, then reload the SSH server: sudo reload ssh

Once the key has been copied across (which will require password entry for the relevant user), test SSH access (as prompted). If it works, you can safely disable password login on the remote machine.

Limit Actions for this SSH Connection to Restricted rsync

Using an SSH key without a passphrase makes it possible to automate remote tasks - a user-entered passphrase is not required for key decryption.

Automation of backup tasks is essential - if backup relies on human intervention, sooner or later it will get missed or messed up. However, using SSH keys without a passphrase is a security risk - is someone had access to the backup server, they would be easily able to access the remote Production server.

To prevent this, SSH allows restriction of the commands that can be executed by a specific set of keys. This is accomplished by editing the /home/backupuser/.ssh/authorized_keys file for the backup user on the Production server.

A script called rrsync (which stands for restricted rsync) is provided with rsync specifically to ease the restricting keys to be used only for rsync via .ssh/authorized_keys.

On Ubuntu, the script is located here: /usr/share/doc/rsync/scripts/rrsync.gz. The script needs to be unzipped and installed under user/local/bin:

# Copy the archive rrsync script to /usr/local/bin
sudo cp /usr/share/doc/rsync/scripts/rrsync.gz  /usr/local/bin/

# Unzip the script
sudo gzip -d   /usr/local/bin/rrsync.gz

# Give it correct permissions
sudo chmod 755 /usr/local/bin/rrsync

Once rrsync is in place, we can lock down access for our SSH key by amending the /home/backupuser/.ssh/authorized_keys file.

Open the public key for the backup user on the Production Server. This will constitute a single line of text:

sudo nano /home/backupuser/.ssh/authorized_keys

Prepend the following:

command="/usr/local/bin/rrsync -ro /home/path-to/backup/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding

The command="/usr/local/bin/rrsync ..." restricts access of that particular public key - only the given command can be executed
The ...-ro /home/path-to/backup/" gives read-only access to the specified directory
The no-* options further restrict what actions can be carried out with the public key.
Note that the full path is need to reference rrsync

If you try and SSH into the Production machine from the Backup server you should now see a message like this:

PTY allocation request failed on channel 0
/usr/local/bin/rrsync: Not invoked via sshd
Use 'command="/usr/local/bin/rrsync [-ro] SUBDIR"' in front of lines in /home/backupuser/.ssh/authorized_keys
Connection to 123.45.67.89 closed.

Command rsync from Backup Server

Note: After the above amendment to the authorized_keys file, the final path from the Backup servers point of view is now “/”:

Sample rsync command from the Backup server:

rsync --log-file=$HOME/.rsyncd.log --progress -az -H -e "ssh -p 1234" backupuser@123.456.789.0:/ ~/backup-target-directory

This specifies:

-a: archive mode; equals -rlptgoD (no -H,-A,-X)
-z: Compress file data during the transfer
-H: Maintain hardlinks (important due to our incremental backup)
-e “ssh -p 1234”: connect via SSH on port 1234
See rsync man

This is a more comprehensive script that sends an email with success/fail messages. This could be added to a nightly cronjob. It would be better if it looped through the servers:

#!/bin/bash
#
# Collect all of the backups!
#
#

# Server 1 Backup
# ------------------------------------------------------------------------------
rsync -aqz -H -e "ssh -p 22222" backupuser@111.111.111.111:/ /home/david/server-1 2> /home/david/server-1-rsync.err

if [[ $? -gt 0 ]]
  then

    # Server 1 errors
    # ----------------------------------------------------------------------------
    SERVER_1_MSG=false
    echo "There was an error in the nightly backup for Server 1 to the backup server: ${TIMESTAMP}"| mail -s "Backup Error, Server 1 to Backup" info@example.com

  else

    # Server 1 success
    # --------------------------------------------------------------------------
    SERVER_1_MSG="rsync was successful for Server 1"

fi

# Server 2 Backup
# ------------------------------------------------------------------------------
rsync -aqz -H -e "ssh -p 22222" backupuser@222.222.222.222:/ /home/david/server-2 2> /home/david/server-2-rsync.err

if [[ $? -gt 0 ]]
  then

    # Server 2 errors
    # ----------------------------------------------------------------------------
    SERVER_2_MSG=false
    echo "There was an error in the nightly backup for Server 2 to the backup server: ${TIMESTAMP}"| mail -s "Backup Error, Server 2 to Backup" info@example.com

  else

    # Server 2 success
    # --------------------------------------------------------------------------
    SERVER_2_MSG="rsync was successful for Server 2"

fi


REPORT="Backup Report, ${TIMESTAMP}
===========================================
$SERVER_1_MSG
$SERVER_2_MSG
"

echo "$REPORT"| mail -s "Backup Status" info@example.com

Problems with Permissions

Some files that you’re backing up may have restrictive permissions. For example, permissions on WordPress configuration file (wp-config.php) may be set to 660 - read/write for owner, read/write for group, and no privilege for others. Most files will have permission set to 644.

Because your backup server is logging in as the backup user on the production server, it won’t have read privilege on the files with 660 permission. Trying to rsync these will result in a permission error.

To prevent this, add the production server’s backup user to the file’s group - typically www-data:

# Production server: server that is being backed up
# Add backup user to the group
usermod -a -G www-data backupuser

This will allow the backup server to log in as the backup user and copy files.

Resources

Export MySQL Users

Sun, 07 Aug 2016 17:52:00 +0100

Build a SQL command that can be used to reconstruct MySQL users - useful for migrating servers, or building a verification check server for backups.

Run this command on the source server:

# Replace PASS with your root password for MySQL
MYSQL_CONN="-uroot -pPASS"

mysql ${MYSQL_CONN} --skip-column-names -A -e "SELECT CONCAT('SHOW GRANTS FOR ''',user,'''@''',host,''';') AS query FROM mysql.user WHERE user NOT IN ('root','pma','phpmyadmin','debian-sys-maint')" | mysql ${MYSQL_CONN} --skip-column-names -A | sed 's/$/;/g' > ~/MySQLUserGrants.sql

See: http://serverfault.com/a/399875

Output:

GRANT USAGE ON *.* TO 'xyz_user'@'localhost' IDENTIFIED BY PASSWORD '*rehoorufiuhrfourur8';
GRANT ALL PRIVILEGES ON `xyz`.* TO 'xyz_user'@'localhost';
GRANT SELECT, RELOAD, REPLICATION CLIENT, SHOW VIEW, EVENT, TRIGGER ON *.* TO 'backup_user'@'localhost' IDENTIFIED BY PASSWORD 'yguirfuiyrfurfygufr';
GRANT USAGE ON *.* TO 'abc_user'@'localhost' IDENTIFIED BY PASSWORD '*yguguyrfeguirefgiuryfegu';
GRANT ALL PRIVILEGES ON `abc`.* TO 'abc_user'@'localhost';
...

This is a set of valid SQL commands that can be run on the destination server to create the necessary users. You can then import files, import databases, adjust /etc/hosts if necessary and the migrated sites should work.

Find All Users for a Database

# Log in as mysql root user
mysql -u root -p
SELECT USER FROM mysql.db WHERE db='DB_NAME';

Resources

Persistent Iptables Rules in Ubuntu 16.04 Xenial Xerus

Sun, 07 Aug 2016 15:35:00 +0100

The process of persisting firewall rules in Ubuntu 16.04 is different to the procedure for 14.04.

The Firewall setup is broadly the same as for 14.04 as described here.

This article briefly describes how to import a set of rules for IPtables and make these rules persist across reboots.

Export Rules

If you’re exporting a ruleset from an existing Ubuntu 14.04 server, log in to this machine. Assuming that the iptables-persistent package is installed, run the following commands:

sudo iptables-save > ~/iptables-rules/ruleset-v4
sudo ip6tables-save > ~/iptables-rules/ruleset-v6

Copy these ruleset files across to a temporary location on your Ubuntu 16.04 server.

Install iptables-persistent

# Install package
sudo apt-get install iptables-persistent

# Start - Thanks to Dave Wood for the correction in the comments section
sudo service netfilter-persistent start

#Add to startup
sudo invoke-rc.d netfilter-persistent save

# Stop the service
sudo service netfilter-persistent stop

Import Rules

# Import Rules
sudo iptables-restore < ~/serenity-iptables-rules/ruleset-v4
sudo ip6tables-restore < ~/serenity-iptables-rules/ruleset-v6

# Check iptables
sudo iptables -S

Save Rules

To save the imported rules, run the iptables-persistent dpkg-reconfigure script:

sudo dpkg-reconfigure iptables-persistent

NOTE: The commands sudo netfilter-persistent save and sudo netfilter-persistent reload should work, but we’ve had problems with these commands and resorted to the dpkg-reconfigure option. It may be that a restart of the service is necessary after running these commands.

The dpkg-reconfigure causes iptables-persistent to repeat the install procedure - it will prompt for you to save the current rules. The current iptables rules will be saved into a file by means of iptables-save >/etc/iptables/rules.v4 and ip6tables-save >/etc/iptables/rules.v6. You should see your rules in /etc/iptables/rules.v4 and /etc/iptables/rules.v6.

The iptables-persistent package causes the following to run on reboot:

iptables-restore < /etc/iptables/rules.v4
ip6tables-restore < /etc/iptables/rules.v6

Persistent Rules and Fail2Ban

If you save iptables rules for restoration on reboot, and they contain rules added by Fail2Ban, Fail2Ban will duplicate the rules on boot. After a few reboots, the iptables can potentially get very messy.

To avoid this, stop the fail2ban service before saving the reconfiguration, and manually edit the saved rules to remove references to Fail2Ban. Rebooting should result in the correct rules being added, as Fail2Ban adds it’s own:

# Stop Fail2Ban
sudo fail2ban-client stop

# Configure Persistent Rules
sudo dpkg-reconfigure iptables-persistent

# Make a backup copy (repeat for the v6 ruleset)
sudo cp /etc/iptables/rules.v4 /etc/iptables/rules.v4.bak

# Edit, remove references to Fail2Ban:
sudo nano /etc/iptables/rules.v4

# Reboot to apply proper rules & restart Fail2Ban
# (probably better to reload rules and restart Fail2Ban - these are rough notes)
sudo reboot

Resources

Apache Virtual Host Setup

Fri, 05 Aug 2016 11:29:00 +0100

Apache Virtual Hosts allows multiple sites to be hosted on a single server/VPS.

These guidelines are for configuration of virtual hosts on Apache 2.4.

Create Directory Structure

sudo mkdir -p /var/www/html/example.com/public_html
sudo mkdir -p /var/www/html/example2.com/public_html

The -p flag creates intermediate directories as required.

Give permissions:

sudo chown -R $USER:$USER /var/www/example.com/public_html
sudo chown -R $USER:$USER /var/www/test.com/public_html

Note WordPress is going to need the www-data to have ownership of the public_html subdirectories to allow file upload etc. - for the time being, give ownership to the current user - pass ownership to www-data later.

Set Permissions to 755 for directories:

sudo chmod -R 755 /var/www

Add index.html demo pages if necessary.

Create Config Files

Create a config file for each site:

sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/yoursite.com.conf

Use this as a template:

<VirtualHost *:80>
        ServerName example.com
        ServerAlias www.example.com
        ServerAdmin info@example.com
        DocumentRoot /var/www/html/example.com/public_html
        <Directory /var/www/html/example.com/public_html>
                Options -Indexes +FollowSymLinks
                AllowOverride All
                Require all granted
        </Directory>
        ErrorLog ${APACHE_LOG_DIR}/example.com.error.log
        CustomLog ${APACHE_LOG_DIR}/example.com.access.log combined
</VirtualHost>

This config block sets the correct server name, alias and document root. Directory browsing is disallowed, and .htaccess files are allowed.

Site specific error reporting is added - log files are located here:

/var/log/apache2/yoursite.com.error.log

Enable the site using a2ensiteand restart Apache:

sudo a2ensite yoursite.com.conf
sudo service apache2 restart

Enable Apache Rewrites

Enabling the Apache rewrite module will be essential it you’re using pretty permalinks. Enable the module & restart Apache:

sudo a2enmod rewrite && sudo service apache2 restart

sudo a2enmod rewrite && sudo /etc/init.d/apache2 restart

Set Up Permanent Redirect

Configure a new Virtual Host on the server to intercept requests for the old domain, and permanently redirect them to the new domain:

<VirtualHost *:80>
        ServerName olddomain-example.com
        ServerAlias www.olddomain-example.com
        ServerAdmin info@example.com

        Redirect 301 / http://new-example.com/

        ErrorLog ${APACHE_LOG_DIR}/olddomain-example.com.error.log
        CustomLog ${APACHE_LOG_DIR}/olddomain-example.com.access.log combined
</VirtualHost>

Bulk Import Images Using WP-CLI

Sun, 31 Jul 2016 17:10:00 +0100

Use WP-CLI to bulk import images to the WordPress media library.

The wp media import command creates attachments from local files or URLs. The utility has the ability to recurse through subdirectories, making it very useful.

A typical workflow might involve:

Getting the raw images
Bulk renaming image files (e.g. to include client details and an index number)
Bulk image processing using a custom script
Use wp media import to import images, either to a local dev site or a staging site

Bulk Import

WP-CLI makes it easy to run a bulk import from the command line:

# Import all images from a local directory, without attaching to posts
wp media import ~/Pictures/processed-images-medium/\*\*\/\*.jpg

The command works recursively, but relies on the glob capabilities of the current shell.

“Globbing” is what the shell does when it encounters a wildcard (like “*”) in a command. Bash 4.0 offers an option called globstar which is used for “**”. As seen in the command above, the “**” pattern allows the command to recurse all directories.

Under Ubuntu 16.04, the globstar option must be turned on for this to work. By default, it isn’t - and the command will not act recursively. To turn globstar on:

shopt -s globstar
wp media import ~/Pictures/processed-images-medium/\*\*\/\*.jpg

Attach an Image to a Post

The WP-CLI wp media import command can also be used to import images as featured images for specified posts:

wp media import ~/Pictures/image.png --post_id=123 \
--title="A downloaded picture" --featured_image
# See: http://wp-cli.org/commands/media/import/#examples

Import an Image as the Featured Image for All posts

This is a pretty handy feature for setting up development sites:

# Line 1
ATT_ID="$(wp media import ~/Pictures/swimming.jpg --porcelain)"
# Line 2
wp post list --post_type=post --format=ids | \
# Line 3
xargs -0 -d ' ' -I % wp post meta add % \_thumbnail_id $ATT_ID

Walkthrough:

The --porcelain option causes the new attachment ID to be output, so this line runs the import and sets the $ATT_ID shell variable to the imported image ID
wp post list --post_type=post --format=ids gets a list of post IDs in a space-separated string format and passes this to xargs
xargs operates on each ID, setting the post meta _thumbnail_id field for each post ID. The xargs options set a space as the delimiter

Resources

Add MixitUp to a Sage Project

Sat, 30 Jul 2016 09:53:00 +0100

MixItUp is a very useful jQuery filtering plugin.

MixItUp is free to use in non-commercial projects under the terms of the creative commons CC-BY-NC license. However, for use in commercial projects and products we require that you purchase a commercial license.

– mixitup.kunkalabs.com/licenses

At the time of writing, the single developer commercial license is $25.

Install Using Bower

Run the following line in your project root - Mixitup will be downloaded as a bower_component, and the project bower.json will be amended:

bower install bower-mixitup --save

The project bower.json now looks like this, with a new dependency:

{
  "name": "your-projectname",
  "homepage": "http://carawebs.com/",
  "authors": [
    "David Egan <david@carawebs.com>"
  ],
  "license": "MIT",
  "private": true,
  "dependencies": {
    "mixitup": "bower-mixitup#^2.0.4"
  }
}

Sage 8 uses the Asset Builder to pipeline assets. This results in an almost magical connecting up of resources, but it needs the imported project’s bower.json to reference the “main” file for the package. The Mixitup bower.json doesn’t do this, so an override must be added to the PROJECT bower.json, which now looks like this:

{
  "name": "your-projectname",
  "homepage": "http://carawebs.com/",
  "authors": [
    "David Egan <david@carawebs.com>"
  ],
  "license": "MIT",
  "private": true,
  "dependencies": {
    "mixitup": "bower-mixitup#^2.0.4"
  },
  "overrides": {
    "mixitup": {
      "main": ["./src/jquery.mixitup.js"]
    }
  }
}

In this example, the main file referenced is an un-minified version - which can make debugging easier. You may as well add the unminified version, since the Sage asset pipeline handles minification.

Adding this allows Asset Builder to grab the imported file and build it into the asset pipeline - which lead us on to the next step.

Add to manifest.json

Adding a proper reference to the bower-installed project in the /assets/manifest.json file will result in a minified asset being built into the dist directory when gulp is run.

Now the asset can be conditionally enqueued where it is needed (it could alternatively be built into main.js).

Enqueuing

To stick with the Sage convention, add your enqueue to the assets() function in /lib/setup.php. For example:

<?php
if( is_post_type_archive( 'project' ) ) {
    wp_enqueue_script('mixitup', Assets\asset_path('scripts/mixitup.js'), ['jquery'], null, true);
    wp_enqueue_script('filter', Assets\asset_path('scripts/filter.js'), ['jquery'], null, true);
  }

Setting Up

Create HTML with the right container and filterable elements
Add/enqueue JavaScript to initiate and control mixitup
Create a HTML filter

Note: By default, MixitUp looks for elements with the class .mix and designates these as filterable elements. You can override this, but if you don’t, and your filter elements don’t have the .mix class, MixItUp isn’t going to work.

Delete files in Atom under Ubuntu 16.04

Tue, 26 Jul 2016 19:54:00 +0100

Atom is my editor of choice - I really enjoy working in it, to the point where I even write content and prepare client reports from within Atom (I can build in git logs, write in Markdown, and convert to PDF for the client). Going back to Tiny MCE in WordPress feels horrible! Atom is free, open source and very hackable - but I recently had a problem with file deletion.

Trashing Files

When trying to delete files in Atom 1.8.0 running on Ubuntu Xenial, you may experience a problem whereby file deletion is not possible. The error message returned by Atom suggests that gvfs-trash is not installed:

The following file couldn’t be moved to trash (is gvfs-trash installed?)

Debugging Steps

Check that gvfs-trash is working by opening a terminal & moving to a development directory - in this case, var/www/html/devsite - and attempting to delete a file:

gvfs-trash filename

# this returns:
Error trashing file: Unable to find or create trash directory

This gives a clue - it looks like:

Atom uses gvfs-trash to send files to Trash on delete
When the file to be deleted is located outside the home directory, there is a problem - the gvfs-trash process fails

It would be good if Atom could return the gvfs-trash error message, as this gives just enough information to hint towards a solution.

Solution

Create a new Trash directory that can be accessed from outside the user’s home directory:

# Create a Trash directory (with some subdirectories) in root
sudo mkdir -p /.Trash-1000/{expunged,files,info}

# Give ownership of this to your user:
sudo chown -R $USER /.Trash-1000

That’s it - problem solved!

The answer had been staring me in the face for weeks - some of my repos are in my home directory (mainly BASH scripts and system utilities) and files in these directories were deleting without issue. It didn’t dawn on me that the issue only cropped up in my Apache doc root (/var/www/html/*). Facepalm.

Posting this here to prevent repeating myself the next time I set up a development environment.

Resources

Useful `gvfs-trash` Commands

# Move oldfile to trash
gvfs-trash oldfile

# See contents of trash:
gvfs-ls trash://

# Take out the trash:
gvfs-trash --empty

Sync a Forked Git Repo to Upstream

Sun, 03 Jul 2016 10:55:39 +0100

When you have forked a repo, it can be useful to pull in changes that have been made to the original.

Git makes this pretty straightforward.

Configure Git Remote

To list the current remotes in your project, cd into the repo directory and run:

git remote -v

# Outputs:
origin  git@bitbucket.org:Username/project.git (fetch)
origin  git@bitbucket.org:Username/project.git (push)

Configure a git remote that points to the upstream repo (the original that you forked from):

git remote add upstream git@bitbucket.org:vendor/original-project.git

Sync to the Upstream Repo

Fetch data from the upstream repo - commits to master will be stored in a local upstream/master branch
Checkout the fork’s master branch (if you’re not already on master)
Merge changes from upstream/master:

git fetch upstream

git checkout master

git merge upstream/master

Automatic Synchronization

Bitbucket allows automatic fork synchronization - this is enabled in the Bitbucket GUI when you first fork the repo. I’ve not used this feature.

References

Using git Branches

Sun, 03 Jul 2016 10:55:39 +0100

Git branches are a good way to work on a feature or a bugfix.

You can create and switch to a new branch, which allows you to work without affecting the production code. Once happy wit changes, the branch can be merged back and deployed.

Determine & Switch Branches

Enter git branch. Returns branches, current branch indicated.

To switch:

git checkout <branchname>

Create New Branch (Local)

To create and switch to a new branch:

git checkout -b <newbranchname>

Don’t forget to git push origin <newbranchname> rather than pushing to master, which will return an “up to date” message.

The remote repo will add the new branch when the local is pushed.

See Which Files Have Changed

Check to see which files have changed:

git diff --name-status master..<branchname>

# even more info:
git diff --stat --color master..<branchname>

Merge Branch into Master

Switch to master branch: git checkout master
Merge: git merge <branchname>

If the master has changed since the branch was created, this will result in a merge conflict.

Dealing With Merge Conflicts

Merge conflicts generally result when the same part of a file has been changed by different branches.

During the merge, git will generate a useful merge-conflict message highlighting the file with the conflict. Running git status will show information about the conflict.

Open the relevant file - git adds conflict markers to the relevant code block, highlighting the different code by branch:

<?php
// organise-posts.php
<<<<<<< HEAD
$colour = "red";
=======
$colour = "green";
>>>>>>> bugfix

Just delete the code you don’t want, or add an entirely new block. Then git add and commit the file with a new commit message:

git add organise-posts.php
git commit -m "Resolve merge conflict: set things up properly."
git push origin master

Delete Branch

Delete local branch: git branch -d <branchname>
Delete remote branch: git push origin --delete <branchName>

More obtuse way of remote deletion: git push origin :<branchname>

Remove a Directory

Remove from git (e.g. Bitbucket, GitHub) but leave in place on local filesystem:

git rm -r --cached my-image-folder

Combine this with an entry into .gitignore to exclude directories (or files) from the repo.

ArrayAccess PHP Interface

Sun, 26 Jun 2016 20:08:20 +0100

The PHP ArrayAccess class is an interface that allows objects to be accessed as arrays.

Classes that extend the ArrayAccess interface must include four core methods:

<?php
ArrayAccess {
  /* Methods */
  abstract public boolean offsetExists ( mixed $offset )
  abstract public mixed offsetGet ( mixed $offset )
  abstract public void offsetSet ( mixed $offset , mixed $value )
  abstract public void offsetUnset ( mixed $offset )
}

When the ArrayAccess class is implemented by a child class with the appropriate methods, data held in the resulting object can be accessed using array syntax.

Background: PHP Arrays

Arrays in PHP are really ordered maps - a data type that associates keys and values.

…This type is optimized for several different uses; it can be treated as an array, list (vector), hash table (an implementation of a map), dictionary, collection, stack, queue, and probably more. As array values can be other arrays, trees and multidimensional arrays are also possible.

—PHP Manual

PHP arrays hold groups of values which can be identified by position or an associative index.

Associative array:

<?php
$container = [
  'first'   => 'one',
  'second'  => 2,
  'third'   => 3,
  'fourth'  => 4,
  'fifth'   => 5
];

// To access:
echo $container['first']; // echoes 'one'
?>

Numerically Indexed Array:

<?php
$container = ['foo', 'bar', 'chimi', 'changa'];

// To access (arrays are zero-indexed):
echo $container[0]; // outputs 'foo'
echo $container[2]; // outputs 'chimi'
?>

Background: PHP Objects

PHP Objects incorporate one or more functions (methods) along with data (properties) into a structure called a class. Many objects can be instantiated from a single class - the class simply provides the blueprint for the object. Classes are defined with the class keyword, and new objects are instantiated from classes using the new keyword.

PHP also includes a generic empty class called stdClass - which can be useful for anonymous objects and dynamic properties. This kind of class is useful for storing key-value pairs. You could do this in an array, but value objects have some advantages.

The stdClass class can be used to create stand-alone objects without first defining a class.

Value Object Example:

<?php
$hygiene_obj = new StdClass;
$hygiene_obj->armpits = 'hairy';
$hygiene_obj->feet_status = 'smelly';
echo "<pre>", json_encode($obj, JSON_PRETTY_PRINT), "</pre>";

// Adding more data is just a matter of creating a new class property. E.g.
$hygiene_obj->hands = 'grubby';

Output:

{
    "armpits": "hairy",
    "feet_status": "smelly"
}

The same thing can be achieved by means of an associative array:

<?php
$hygiene_status = [
  'armpits' => 'hairy',
  'feet status' => 'smelly'
];
echo "<pre>", json_encode($hygiene_status, JSON_PRETTY_PRINT), "</pre>";

// To add another element to the array:
$hygiene_status['hands'] = 'grubby';

The output is the same as the object example, though there is more leeway with the key names, which can contain spaces:

{
    "armpits": "hairy",
    "feet status": "smelly"
}

There are a few ways of creating a stdClass object in modern PHP:

<?php
$obj1 = new stdClass;
$obj2 = new class{}; // Instantiate anonymous class
$obj3 = (object)[];

Mistake pointed out by Rajesh in comments:

~~$Object = {}~~

Array Access

It can sometimes be useful to access objects as arrays. ArrayAccess is a built in interface class that allows this.

The interface requires that extending classes use these methods:

ArrayAccess::offsetExists
ArrayAccess::offsetGet
ArrayAccess::offsetSet
ArrayAccess::offsetUnset

The methods allow us to define a handy data container which can be accessed and amended using the array syntax.

ArrayAccess: Simple Implementation

<?php
class Foo implements ArrayAccess {

  public $container = [
    'first'   => 1,
    'second'  => 2,
    'third'   => 3
  ];

  public function offsetExists ($offset) {

    return isset($this->container[$offset]);

  }

  public function offsetGet ($offset) {

    return isset($this->container[$offset]) ? $this->container[$offset] : null;

  }

  public function offsetSet ($offset, $value) {

    if (is_null($offset)) {
      $this->container[] = $value;
    } else {
      $this->container[$offset] = $value;
    }

  }

  public function offsetUnset ($offset) {

    unset($this->container[$offset]);

  }

}

This allows us to access the Object data structure using array syntax:

<?php
  $object = new Foo;
  $object['test key'] = 'value';
  $object['chimi'] = 'changa';
  $object['first'] = 'changed';

The Updated object:

<?php
// Output of var_dump($object)
object(Foo)[1]
  public 'container' =>
    array (size=5)
      'first' => string 'changed' (length=7)
      'second' => int 2
      'third' => int 3
      'test key' => string 'value' (length=5)
      'chimi' => string 'changa' (length=6)

References

Mount Disk Drives in Ubuntu & Raspbian

Tue, 10 May 2016 00:00:00 +0100

Automatically mount disk drives on boot.

We recently set up a Raspberrry pi as a local fileserver. The Pi is connected to a powered USB hub and a number of hard drives. We use this to backup our systems overnight - the Pi is not the fastest, but the entire system draws less power than a lightbulb.

To work properly, the connected drives need to be mounted at boot, and this guide outlines how to achieve this.

You could follow the same procedure for adding drives to any Debian based distro.

Set Up Drives

Use the Disks utility to format and label drives.

Ext4 is a good choice for a filesystem format if the disk is intended for Linux use only.

Mount Points

To mount disk drives, first create a mount point(s) and set permissions:

sudo mkdir /mnt/disk-1
sudo mkdir /mnt/disk-2

# set permissions:
sudo chmod -R 755 /mnt

Find Drives

There are many ways to determine the drives available to the system:

# Get name, size & model, but no UUID
lsblk -ido KNAME,TYPE,SIZE,MODEL
# Output:
david@raspberrypi:~ $ lsblk -ido KNAME,TYPE,SIZE,MODEL
KNAME   TYPE   SIZE MODEL
sda     disk 931.5G External USB-3.0
sdb     disk 931.5G TOURO Mobile    
mmcblk0 disk   7.3G

# Get disk name and partition data
sudo fdisk -l

# Get UUID
ls -l /dev/disk/by-uuid/
# Output:
david@raspberrypi:~ $ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 15 May 10 10:36 04FE-51E9 -> ../../mmcblk0p1
lrwxrwxrwx 1 root root 10 May 10 10:36 76ef1fc2-1bc5-4a2c-950f-43e99c975da8 -> ../../sdb1
lrwxrwxrwx 1 root root 15 May 10 10:36 e6e7f776-11a4-4cd7-b4fd-c44ecdbfcf90 -> ../../mmcblk0p2
lrwxrwxrwx 1 root root 10 May 10 10:36 fba9c513-2117-4245-a41c-f813497b0826 -> ../../sda1

# RECOMMENDED - show UUID & disk label for connected drives
# ------------------------------------------------------------------------------
sudo blkid
# Typical output:
david@raspberrypi:~ $ sudo blkid
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="boot" UUID="04FE-51E9" TYPE="vfat" PARTUUID="8f1eafaf-01"
/dev/mmcblk0p2: UUID="e6e7f776-11a4-4cd7-b4fd-c44ecdbfcf90" TYPE="ext4" PARTUUID="8f1eafaf-02"
/dev/sda1: LABEL="intenso-backup-1" UUID="fba9c513-2117-4245-a41c-f813497b0826" TYPE="ext4" PARTUUID="972d07b8-01"
/dev/sdb1: LABEL="Backup Data" UUID="76ef1fc2-1bc5-4a2c-950f-43e99c975da8" TYPE="ext4" PARTUUID="ef9e4c2c-01"
/dev/mmcblk0: PTUUID="8f1eafaf" PTTYPE="dos"

Disk drives are distinguished by a UUID(universally unique identifier) - we’ll use this number to mount the disks rather than the assigned device name (e.g. /dev/sda1), since the latter might change and the UUID will not.

Manual Mount

To mount a drive manually:

sudo mount /dev/sda1 /mnt/disk-1

This won’t be persistent - when the system reboots, the drive will no longer be available.

Auto Mount on Boot

Amend /etc/fstab by adding a line for each required disk. For example:

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1

# additional lines follow - referencing the correct UUID,
# mount point and format for the required disks
UUID="76ef1fc2-1bc5-4a2c-950f-43e99c975da8"       /mnt/disk-1       ext4    defaults          0       0
UUID="fba9c513-2117-4245-a41c-f813497b0826"       /mnt/disk-2       ext4    defaults          0       0

Save, reboot and disks will be mounted automatically on the defined mount points.

WordPress Pages for Posts - Show the Content Editor

Thu, 05 May 2016 00:00:00 +0100

When WordPress has a static front page (as set in “Settings” > “Reading”), it is often useful to display introductory content on the page that has been selected for posts.

The posts page is determined by the ‘page_for_posts’ value in the ‘option_name’ column in the WordPress options table. The ID of this page can be retrieved by:

<?php
$page_for_posts_ID = get_option( 'page_for_posts');

This allows the page content and featured image to be displayed in the template.

Display Page Content

By default, the page for posts uses the WordPress home.php template. To access the page content in this template:

<?php
$posts_page = get_post( get_option( 'page_for_posts' ) );
echo apply_filters( 'the_content', $posts_page->post_content );

Where the Hell is the Editor?

WordPress (as of version 4.5.1 anyway) has an extremely annoying characteristic whereby the content editor is not shown on the page for posts if the post_content field is empty. Instead, you get an annoying message telling you what you already know - that this is the page selected to display blog posts.

There are two fixes:

Temporarily set a different page for blog posts, add some content to your original posts page, and then switch the setting back to the original posts page
Remove the annoying message and add back the editor by hooking onto edit_form_after_title

When I first noticed this, it bugged the hell out of me - since my development site had page content, the editor displayed normally (not so in the staging site).

The culprit is this little diamond in /wp-admin/edit-form-advanced.php:

<?php
if ( $post_ID == get_option( 'page_for_posts' ) && empty( $post->post_content ) ) {
	add_action( 'edit_form_after_title', '_wp_posts_page_notice' );
	remove_post_type_support( $post_type, 'editor' );
}

The second, less hackish fix:

<?php
/**
 * Add the wp-editor back into WordPress after it was removed in 4.2.2.
 *
 * @see https://wordpress.org/support/topic/you-are-currently-editing-the-page-that-shows-your-latest-posts?replies=3#post-7130021
 * @param $post
 * @return void
 */
 function fix_no_editor_on_posts_page($post) {

   if( $post->ID != get_option( 'page_for_posts' ) ) { return; }

   remove_action( 'edit_form_after_title', '_wp_posts_page_notice' );
   add_post_type_support( 'page', 'editor' );

 }

 // This is applied in a namespaced file - so amend this if you're not namespacing
 add_action( 'edit_form_after_title', __NAMESPACE__ . '\\fix_no_editor_on_posts_page', 0 );

Credit for this goes to crgeary

I can’t imagine why this “functionality” has been built in. It would surely be better for WP core to be less opinionated - but at least it’s fairly easy to undo the damage.

References

Connect Raspberry Pi Backup Server to Remote Server

Wed, 04 May 2016 00:00:00 +0100

This article describes how to provide a local Raspberry Pi access to a remote server in order to automatically download backup files through an encrypted channel. The connection is limited to read-only access of a specified directory on the remote server.

Context

The objective is to hold a local copy of backup files. The solution needed to be fully automatic and secure.

A Raspberry Pi with an external SSD storage drive was used as the local client. Once downloaded, files are accessible across the LAN by SSHing into the Pi.

The Pi acts as an always-on local fileserver - this periodically connects to a remote server (in this case, a Digital Ocean Ubuntu box) via SSH collecting backup data using rsync. The setup is low power, cost-effective, secure and runs automatically.

Because the backup script runs automatically and is unattended, it has to be passwordless. The scope of action on the target machine must be limited to read-only rsync for a specified directory. This limitation is the focus of this article, though I also provide a sample backup script.

Key Generation

A public-private key pair is required, with the public key added to the target server.

If one doesn’t already exist, or you need a unique key for some reason, you can generate an RSA key with a specific name:

ssh-keygen -t rsa

# Outputs this:
Enter file in which to save the key (/home/david/.ssh/id_rsa):

Specify a filename if required. Enter the full path, e.g. /home/username/.ssh/local-fileserver

Comments in the Key File

It can be helpful to have a comment in the key file. When transferred to the target machine, the public key will be appended to the /home/username/.ssh/authorized_keys file. The comment will help distinguish the key.

If the client machine is compromised, it allows the relevant line to be easily removed from the server authorized_keys file. If no comment is added, the public key file has username@hostname appended to the key as a default comment.

To add a comment, just enter text at the end of the public key, separated from the key content by a space.

Specify the Correct Key

If you’re not using the default key file, create an entry in an SSH config file specifying the correct key file for the server that you’re going to connect to.

You may need to create the config file it it doesn’t exist. The following command will do this in any case, with the file being created when you save (ctrl + o in nano):

sudo nano /home/username/.ssh/config

# Enter the following and save
Host 192.168.1.XXX
    IdentityFile ~/.ssh/local-fileserver

Uploading the Key

This can be achieved easily using the ssh-copy-id command - though the server must obviously allow the connection before the key is set up, so you may need to temporarily amend the servers SSH configuration to reflect this.

If no keyfile is specified, this command will copy all public keys from the /home/username/.ssh directory, which may not be what you intend.

The -i flag can be used to set the specific key file that should be transferred:

ssh-copy-id -i ~/.ssh/local-fileserver.pub username@192.168.1.XXX

Once the key has been transferred, set the SSH config to disallow password login (PasswordAuthentication no) and restart SSH.

Security

Using an SSH key without a passphrase makes it possible to automate remote tasks - a user-entered passphrase is not required for local key decryption.

Automation of backup tasks is essential - if backup relies on human intervention, sooner or later it will fail. However, using SSH keys without a passphrase is a security risk - is someone had access to the local backup server, they would be easily able to access the remote server.

Realistically, this is a pretty low risk. The bad guy would have to break in to your office and know his way round a headless Linux box. We don’t get too many roving bands of rogue systems administrators in this neck of the woods, but just in case…

SSH allows restriction of the commands that can be executed by a specific set of keys. This is accomplished by editing the /home/username/.ssh/authorized_keys file on the target server.

A script called rrsync (which stands for restricted rsync) is provided with rsync specifically to ease the restricting keys to be used only for rsync via .ssh/authorized_keys.

On Ubuntu, the script is located here: /usr/share/doc/rsync/scripts/rrsync.gz. The script needs to be unzipped and installed under usr/local/bin:

# Copy the archive rrsync script to /usr/local/bin
sudo cp /usr/share/doc/rsync/scripts/rrsync.gz  /usr/local/bin/

# Unzip the script
sudo gzip -d   /usr/local/bin/rrsync.gz

# Give it correct permissions
sudo chmod 755 /usr/local/bin/rrsync

Once rrsync is in place, we can lock down access for our SSH key by amending the /home/backupuser/.ssh/authorized_keys file on the target server.

Open the public key for the backup user on the target server. This will constitute a single line of text:

sudo nano /home/backupuser/.ssh/authorized_keys

Prepend the key data with the following:

command="/usr/local/bin/rrsync -ro /home/path-to/backup/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding

The command="/usr/local/bin/rrsync ..." restricts access of that particular public key - only the given command can be executed
The ...-ro /home/path-to/backup/" gives read-only access to the specified directory
The no-* options further restrict what actions can be carried out with the public key.
Note that the full path is need to reference rrsync

If you try and SSH into the Production machine from the Backup server you should now see a message like this:

PTY allocation request failed on channel 0
/usr/local/bin/rrsync: Not invoked via sshd
Use 'command="/usr/local/bin/rrsync [-ro] SUBDIR"' in front of lines in /home/backupuser/.ssh/authorized_keys
Connection to 123.45.67.89 closed.

Your script however will be able to gain read-only access, which is good enough to pull-down backup files.

rsync Command rsync from the Client

Note: After the above amendment to the authorized_keys file, the final path from the Backup server’s point of view is now /

Sample rsync command from the local server:

rsync --log-file=$HOME/.rsyncd.log --progress -az -H -e "ssh -p 1234" backupuser@123.456.789.0:/ ~/backup-target-directory

This specifies:

-a: archive mode; equals -rlptgoD (no -H,-A,-X)
-z: Compress file data during the transfer
-H: Maintain hardlinks (important due to our incremental backup)
-e “ssh -p 1234”: connect via SSH on port 1234
See rsync man page

Connect Via a Scheduled Script

Save the following script in /usr/local/sbin (or create an appropriate symlink:

#!/bin/bash
#
# On remote, run `rrsync` and prepend this command to the relevant line in `/home/username/.ssh/authorized_keys`:
# `command="/usr/local/bin/rrsync -ro /home/path/to/backup/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding`
#
# Run this file from the root crontab on the client machine.

# Name of the current script
NAME=$(basename "$0")
# The remote server username@ip address, appended with `:/`
SRC="remote-username@111.222.111.222:/"
TRG="/mnt/usb1/server-backups/local-backups-dir"
LOGFILE="/mnt/usb1/server-backups/pull-backups.log"
TIMESTAMP=$(date '+%Y-%m-%d at %H:%M:%S')
ERRORLOG="/mnt/usb1/server-backups/pull-backups-error.log"

# ------------------------------------------------------------------------------
# rsync options: follow the symlinks to make a hard backup. Note the custom port
# specified for SSH which needs to be configured on the remote server.
# ------------------------------------------------------------------------------
OPT=(-az -H --exclude='*.zip' --exclude='debian.cnf' -e 'ssh -p 3333')

# ------------------------------------------------------------------------------
# Execute the backup
# ------------------------------------------------------------------------------
rsync "${OPT[@]}" $SRC $TRG 2> $ERRORLOG

if [[ $? -eq 0 ]]
then
  ENDTIME=$(date '+%H:%M:%S')
  printf -v MSG "SUCCESS! %s started on: %s, successfully finished at: %s" "$NAME" "$TIMESTAMP" "$ENDTIME" echo "$MSG" >> "$LOGFILE"
else
  ENDTIME=$(date '+%H:%M:%S')
  printf -v MSG "ERROR! There was a problem running %s. Started on: %s, ended at: %s. See: %s" "$NAME" "$TIMESTAMP" "$ENDTIME" "$ERRORLOG"
  echo "$MSG" >> "$LOGFILE"
fi

Edit the crontab:

sudo crontab -l

Enter the following on a separate line:

# m h  dom mon dow   command

# Run script `my-backup-script` every day at 03:30 am
30 03 * * * /usr/local/sbin/my-backup-script

References

Ubuntu Manpage for ssh-copy-id

Accessing Local Fileserver in Ubuntu

Mon, 02 May 2016 00:00:00 +0100

A headless always-on server can be a good way of managing office backups and shared directories.

These instructions assume that:

A linux server has been set up on the network and is accessible via SSH from the client machine (we use a raspberry pi with attached external storage)
The machines accessing the server’s filesystem are running Ubuntu desktop (14.04 in this case)
The remote filesystem (in this case, the external drive attached to the Pi) is mounted on /mnt - this is the directory that will be accessible by the client

Install sshfs on the Client Ubuntu Machine

SSHFS stands for SSH File System. It is a filesystem client that allows you to interact with files on a remote server or workstation by means of an SSH connection.

Install SSHFS:

sudo apt-get install sshfs

Add your non-sudo user to the fuse group:

sudo usermod -a -G fuse username

Activate group membership change:

exec su -l $USER

Make and Mount the Directory

test -e ~/local-backup-server || mkdir --mode 700 ~/local-backup-server
sshf pi@192.168.1.XXX:/mnt local-backup-server

You’ll need to enter password for the remote host.

You could also set up passwordless ssh login to simplify this process.

Unmounting the Filesystem

To unmount a ssh-mounted directory:

fusermount -u <local_mount_point>

Auto Mount

To automatically mount over ssh on boot:

Set up passwordless ssh login
Append the following in /etc/fstab:

sudo nano /etc/fstab
sshfs pi@192.168.1.123:/mnt <local_mount_point> fuse user 0 0

Connect Via Nautilus

In Nautilus, open ‘File’ > ‘Connect to Server’ in ‘Server Address’ bar, add sftp://pi@192.168.1.XXX:/mnt.

Resources

Mount Remote Directory Over SSH on Linux - nice article, but the Nautilus information is superceded by the info contained in the link below
Connect via Nautilus

Deregister Open Sans from WordPress Admin

Mon, 11 Apr 2016 00:00:00 +0100

WordPress enqueues the Open Sans font in the admin area. This requires making a request to Google Fonts, which obviously isn’t going to work if you are offline.

This can be very problematic if you’re developing locally and you lose the internet connection. The local site becomes extremely slow as the request to Google Fonts does it’s thing.

This caught me off guard at first - I blocked my own custom script calls and couldn’t understand what was messing up the local site. It’s WP core calling Google.

The following hack removes Open Sans from the admin area. Add this code to your theme and uncomment the add_action().

Remove Open Sans

<?php
namespace Company\Project\Extras;

/**
 * In case of internet outage
 * @return void
 */
function remove_open_sans() {

    wp_deregister_style( 'open-sans' );
    wp_register_style( 'open-sans', false );
}
// Un-comment these lines in case of internet outage
// Admin area
add_action( 'admin_enqueue_scripts', __NAMESPACE__ . '\\remove_open_sans' );
// Front end - this may be necessary if using default WP themes
add_action('wp_enqueue_scripts', __NAMESPACE__ . '\\remove_open_sans');

If you don’t use namespacing you won’t need __NAMESPACE__ . '\\ in the callback reference.

Seems like a lot of hassle just for a pretty basic admin font. Maybe the enqueuing function could be wrapped in a conditional that pings Google Fonts?

I think core should use system fonts by default - if we need to prettify the admin area, that should be a theme-level decision.

References

The answer by Michael Dance Sort it with a plugin

Manipulate Paths and Filenames

Sun, 10 Apr 2016 16:35:00 +0100

BASH scripts often require manipulation of the pathname of files/directories.

Get Any Path in a Filename

You may need access to the target file’s parent directory, or you may need the file/directory name seperated from it’s path.

# Example pathname
pathname=parent/child/target

echo $(basename $pathname)
# returns target

echo $(basename $(dirname $pathname))
# returns child

echo $(basename $(dirname $(dirname $pathname)))
#returns parent

Strip Trailing Slash

DIR=test/
SOURCE_DIR=${DIR%/}
echo $SOURCE_DIR
# returns test

Bootstrap 3 Header Markup for Sage WordPress Theme

Sun, 10 Apr 2016 00:00:00 +0100

Sage is an excellent starter theme for WordPress. It comes with Bootstrap connected out-of-the-box, but it does not have a “standard” Bootstrap nav markup. The old Roots theme (Sage’s parent) used to have a NavWalker() class that extended the WordPress Walker_Nav_Menu() class to give cleaner menu markup and Bootstrap friendly classes.

To add the NavWalker() class, from your theme root, run:

wget https://gist.githubusercontent.com/DavidCWebs/445308a3beeffef3d4cf/raw/2187f99bf3e1e9b0f571e507b614ca95ed7df286/nav.php -O lib/nav.php

NB: see update below for a potentially better option.

Then replace the code in your templates/header.php with this (or similar Bootstrap-style markup):

<?php require_once( get_template_directory() . '/lib/nav.php' ); ?>
<header class="banner navbar navbar-inverse navbar-fixed-top navbar-shrink" role="banner">
  <div class="container">
    <div class="navbar-header">
      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse">
        <span class="sr-only"><?= __('Toggle navigation', 'sage'); ?></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>
      <a class="navbar-brand" href="<?= esc_url(home_url('/')); ?>"><?php bloginfo('name'); ?></a>
    </div>
    <nav class="collapse navbar-collapse" role="navigation">
      <?php
      if (has_nav_menu('primary_navigation')) :

        wp_nav_menu(['theme_location' => 'primary_navigation', 'walker' => new Roots\Sage\Nav\NavWalker(), 'menu_class' => 'nav navbar-nav']);

      endif;

      ?>
    </nav>
  </div>
</header>

If you are using the navbar-fixed-top class on the navbar element, you’ll need to add the following style rules:

// Offset navbar from top if logged in to allow for the WP admin toolbar
// -----------------------------------------------------------------------------
.logged-in .navbar-fixed-top {
  margin-top: 30px;
  // At the extra small screen and smaller, WP toolbar is taller
  @media (max-width: $screen-xs-max) {
  	margin-top: 46px;
  }
}
// If the navbar is fixed, push down the wrap
// -----------------------------------------------------------------------------
.navbar-fixed-top + .wrap {
  margin-top: 78px;
}

Update September 2016

The original article didn’t provide the license for the NavWalker. I’ve now included the license in the file docblock.

The NavWalker above is from quite an old version of Roots - I can’t remember the exact version. The Roots NavWalker class was dropped in March 2015. The link below is for the most recent version, and it may be better for your purposes: https://github.com/roots/sage/blob/9efeb67daffa477873d33adafd8b12386c4298ba/lib/nav.php

I haven’t tried this class, but I will on my next WP project. It looks like it handles custom post types in a better way than the older NavWalker referenced in my original article.

To copy it into your project, run:

wget https://raw.githubusercontent.com/roots/sage/9efeb67daffa477873d33adafd8b12386c4298ba/lib/nav.php -O lib/nav.php

Parse YAML in PHP Using Symfony YAML

Mon, 21 Mar 2016 00:00:00 +0000

I do a lot of work in WordPress. I also build a lot of static websites - both for rapid design in-the-browser and as a low-cost small-business website solution. I mainly use the excellent Jekyll static site generator.

Jekyll uses YAML for config and data files (it can also use CSV format, but that’s another story). WordPress doesn’t use YAML.

I like YAML because it is very human friendly - the whole team (including non-developers) can easily build YAML config files in a way that you’re not going to see with formats like JSON or XML. This is an example of a YAML array used to create JavaScript variables for use in a Google map:

# Map centre Latitude & Longitude
# ------------------------------------------------------------------------------
latitude: 52.7157856867271
longitude: -8.8741735070805

zoom: 15
#height: 548px

# Set custom colour variables
# ------------------------------------------------------------------------------
waterColour: "#398A8D"
landColour: "#dec7c7"
mainRoadColour: "#777777"
minorRoadColour: "#a9a9a9"

# A nested array
# ------------------------------------------------------------------------------
test:
  - One
  - Two
  - Three

In the context of Jekyll, you could use this as a data file (e.g. map.yml), writing JavaScript variables into <head> something like this:

<script>
var cwCentre = {
  latitude:{{ site.data.map.latitude }},
  longitude:{{ site.data.map.longitude }},
  zoom:{{ page.map-zoom }},
  mainMarker:"{{ site.baseurl}}/{{ site.data.map.mainMarker }}",
  secondaryMarker:"{{ site.baseurl}}/{{ site.data.map.secondaryMarker }}",
  waterColour:"{{ site.data.map.waterColour }}",
  landColour:"{{ site.data.map.landColour }}",
  mainRoadColour:"{{ site.data.map.mainRoadColour }}",
  minorRoadColour:"{{ site.data.map.minorRoadColour }}",
  title: "{{ site.data.map.title | escape }}",
  description:'{{ map_description | markdownify | strip_newlines }}',
};
{% if "multi-centre" == page.map-type %}
var markers = [
  {% for location in site.data.secondary-map-coords %}
    [
      '{{ location.name | escape }}',
      {{ location.latitude }},
      {{ location.longitude }},
      '{{ location.description | escape }}'
    ]
    {% unless forloop.last %},{% endunless %}
  {% endfor %}
];
{% endif %}

</script>

YAML in WordPress

I recently needed to convert a Jekyll site to a WordPress theme. Moving the map config settings required parsing YAML data into a PHP array. Fortunately this can be achieved pretty easily thanks to the Symfony YAML component.

I’m a recent convert to Composer, and find it amazingly powerful. You can add the Symfony YAML component with a single composer command.

Add Symfony/YAML Using Composer

composer require symfony/yaml

When you run this, composer will add a new symfony/yaml directory under the project vendor directory. It will also add the relevant namespace to the autoload_psr4.php file, so that the new class will be autoloaded.

Using the YAML parser

To read the YAML contents of the config fiel into a PHP array:

<?php
use Symfony\Component\Yaml\Parser;

$yaml = new Parser();

$value = $yaml->parse( file_get_contents( get_template_directory() . '/assets/map.yml' ) );

For the YAML content presented above, the following will be output:

$value = array (
  'latitude' => 52.715785686727102,
  'longitude' => -8.8741735070804992,
  'zoom' => 15,
  'waterColour' => '#398A8D',
  'landColour' => '#dec7c7',
  'mainRoadColour' => '#777777',
  'minorRoadColour' => '#a9a9a9',
  'test' =>
  array (
    0 => 'One',
    1 => 'Two',
    2 => 'Three',
  ),
);

This array can be passed to wp_localize_script() when enqueuing the map script.

The WordPress/PHP way would be to collect such data from a form on an admin page, storing the data in the wp_options table. However taking variables from YAML files can be a good way to quickly port settings, which might even be used as defaults. It might also be a good way to configure certain project settings.

Undo a Git add

Tue, 01 Mar 2016 00:00:00 +0000

git add Adds file contents to the index - preparing the content staged for the next commit.

The “index” holds a snapshot of the content of the working tree, and it is this snapshot that is taken as the contents of the next commit.

git docs

Before running the commit command, you use add to add new/modified files to the index.

It’s common that you accidentally add a file - and it can be very important to remove it again.

Remove a Single File

To remove a given file only from the current index:

git reset filename

Undo `git add` Althogether

git reset

Exim4 Send Only Mailserver

Tue, 01 Mar 2016 00:00:00 +0000

Exim4 is a lightweight mail transfer agent that is easy to set up and configure.

These instructions focus on Ubuntu 14.04 Server.

Setting up a send-only mailserver on Ubuntu is relatively straightforward. The tricky bit is ensuring that mail sent by the server is not labelled as spam.

Exim4 Installation

Upgrade system: sudo apt-get update && sudo apt-get -y upgrade
Install Exim4 and dependencies: sudo apt-get -y install exim4
Run the config script: sudo dpkg-reconfigure exim4-config

Exim4 Configuration

Make selections with arrow keys - select “Ok” with tab key

Select: ‘internet site; mail is sent and received directly using SMTP’
Enter FQDN: hostname.yourdomain.com
SMTP Listener: enter 127.0.0.1 or 127.0.0.1; ::1(for IPv6 support)
Mail destinations - list FQDN, local hostname, localhost.localdomain, localhost (see NOTE below)
Relay Options: Leave blank - hit tab to highlight “Ok”, then enter
Follow up Screen to Relay Options: Leave blank, hit “Ok”
DNS Queries: Keep DNS queries to a minimum? select “No”
Delivery method: Select “Maildir format in home directory”
Choose default unsplit config file by entering “No”

In Ubuntu 14.04, that’s it - there is no option to specify postmaster mail recipients. This must be set - or email clients like gmail will place sent emails in the recipient’s spam folder. Most tutorials refer to a final configuration window - but this will not appear on Ubuntu 14.04.

Specify Postmaster Address

Amend /etc/email-addresses to include:

username: no-reply@yourdomain.com

Replace “username” with the system username that is running the mail process.
Enter new username/email values on separate lines

Test email from Command Line:

echo "Hello - this is a test!" | mail -s Testing you@yourweb.com

Access Logs

The main log is held in: /var/log/exim4.

This directory is owned by Debian-exim:adm - and is hence inaccessible, unless you log in as root.

Add your user to the adm group to access logs:

sudo usermod -a -G adm username

You can then access the main exim4 log at: /var/log/exim4/mainlog

Use eximstats to see a text report:

eximstats /var/log/exim4/mainlog

Post Installation Configuration

The initial configuration settings are stored in the /etc/exim4/update-exim4.conf.conf file.

To re-configure the program, either re-run the configuration wizard: sudo dpkg-reconfigure exim4-config or manually edit this file:

sudo nano /etc/exim4/update-exim4.conf.conf

After editing the settings, you need to generate the master configuration file:

sudo update-exim4.conf

Restart the exim4 service:

sudo /etc/init.d/exim4 restart

Set Up Reverse DNS

Reverse DNS maps an IP address to a domain name - and is crucial to avoid having outgoing mail labelled as spam.

The reverse DNS zone must be created on the “authoritative DNS nameserver for the main IP address of your server” - so to set reverse DNS, you need control over the nameserver for your main IP address.

For Linode VPS:

Log in to Linode Manager
Select the Linode
Select “Remote Access”
Enter the domain name in the hostname field and click “lookup”
A message should appear stating that a match has been found between your domain and your IP address
Select “yes” for the desired address (IPv4 & IPv6 will need to be selected separately)

That’s it!

SPF Records

Publishing a Sender Policy Framework (SPF) Record in your domain’s DNS specifies which server IP addresses are allowed to send emails from your domain.

Having a properly set up SPF record makes it less likely that outgoing mail will be tagged as spam.

Although it is best practice is to publish SPF records via DNS as both a SPF and TXT record, the DNS management utilities for many registrars (e.g. Heart Internet, Blacknight) only provide for the addition of txt records. For a good description of SPF as TXT records, see here.

The following SPF record, added as a TXT record is a reasonable sample:

v=spf1 a mx ip4:123.45.678.90 mx:mail.yourdomain.com ~all

Bizzarre Spam Issues

After following the above setup, I noticed the following weird behaviour with regard to gmail recipients.

Send mail from WordPress to a gmail account (the www-data user, not aliased in /etc/email-addresses): Success
Send mail from command line (echo "This is a test." | mail -s Testing emailaddress68@gmail.com) to same address: Designated as Spam
Send the same mail from command line to a different gmail account: Success

There were two SPF records on the DNS settings for the domain - which isn’t allowed. Deactivated the old one.

In addition, the header in gmail showed that gmail was looking for an IP address in the ipv6 format…so I added this to the SPF record.

To determine ipv6 for the server: ip -6 addr show - the public IP is on the line labelled ‘scope global’.

You can only add a single SPF record - but ipv6 and ipv4 IP addresses can be included. Sample TXT record for SPF:

v=spf1 a mx ip4:123.45.67.890 ip6:3a00:9a00::b14d:12aa:ab67:a3c5/64 mx:mail.yourdomain.com ~all

Send Email From Command Line

echo "Hello - this is a test!" | mail -s Testing you@yourweb.com

Resources

Exim4 wiki
Brad the Mad’s Exim Cheatsheet - switch exim4 for exim if necessary
Setup Reverse DNS
Digital Ocean tutorial on SPF
Unlock the Inbox: Info on email configuration
SPF Wizard
Unlock the Inbox Sender ID Wizard
Unlock the Inbox SPF Wizard
~~MS SPF Wizard~~: Page removed, thanks to Henry for pointing this out
Exim4 setup & config
Send email from Command Line

iptables Ubuntu Firewall

Thu, 25 Feb 2016 00:00:00 +0000

iptables is a firewall programme that allows you to control server traffic on a Linux system.

By default, iptables allows all traffic.

Note: UFW is the default firewall configuration tool for Ubuntu - UFW (disabled by default) can simplify firewall setup.

Display iptables Info

# Quick list
sudo iptables -S

# iptables list with numbered lines
iptables -nL --line-numbers

Add Rules

The -A flag denotes that the rule will be appended - added as the last rule in the chain.

The -I flag denotes that the rule will be inserted - the precise line-number for insertion can also be controlled.

# Allow connection on post 80 by appending a rule
sudo iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

# Allow incoming traffic on the default SSH port(22):
sudo iptables -A INPUT -p tcp --dport ssh -j ACCEPT

# Allow SSH connection on a non-standard port (1234)
sudo iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 1234 -j ACCEPT

# Allow ping from a specified IP address, inserted at a specific line in the chain:
sudo iptables -I INPUT 9 -s x.x.x.x -p ICMP --icmp-type 8 -j ACCEPT

# Insert a rule as the first rule in the INPUT chain
# This rule allows already established (and related) connections
iptables -I INPUT 1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Delete Rules

Delete a rule in a chain using the -D option. You need to specify the chain, and the rule. The rule is either specified by entering the whole rule to match, or the line number of the rule to delete. Rules are numbered from the top of each chain, starting with 1.

# Delete by specifying rule
sudo iptables -D INPUT -p tcp -m tcp --dport 80 -j ACCEPT

# Delete by specifying the line number in the INPUT chain - deletes the 9th rule in the chain
sudo iptables -D INPUT 9

View iptables Log

Logs are generated by the kernel, so they go to the file that receives kernel logs: /var/log/kern.log.

To view iptables denial in real time, run:

tail -f /var/log/kern.log

Blocking an IP

To block an attackers ip address (e.g. 1.2.3.4) enter:

iptables -A INPUT -s 1.2.3.4 -j DROP
iptables -A INPUT -s 192.168.0.0/24 -j DROP

Make iptables Rules Persistent

The iptables rulesets are held in memory, which means that they are not persistent across system reboots.

The easiest way to persist rules on Ubuntu is probably the iptables-persistent package.

To install:

sudo apt-get install iptables-persistent

# Start the service!
sudo service iptables-persistent start

After starting, new ruleset files will be created: /etc/iptables/rules.v4 and /etc/iptables/rules.v6. These contain IPv4 and IPv6 rules respectively.

After amending iptables (or ip6tables), save changes to those files:

sudo iptables-save | sudo tee /etc/iptables/rules.v4
sudo ip6tables-save | sudo tee /etc/iptables/rules.v6

# Restart service to read the changes in:
sudo service iptables-persistent start

See http://askubuntu.com/questions/119393/how-to-save-rules-of-the-iptables

Import or Restore a Ruleset

# Save ruleset
sudo iptables-save > /tmp/ruleset-v4
sudo ip6tables-save > /tmp/ruleset-v6

# Restore ruleset
sudo iptables-restore < /tmp/ruleset-v4
sudo ip6tables-restore < /tmp/ruleset-v6

There is a good basic ruleset from Linode here.

Sample iptables Ruleset

Restoring this ruleset may provide a decent starting point. The default policy is DROP - this means that the rules contain exceptions - if the specific condition is met, the packet is accepted.

The ruleset below allows:

Loopback traffic
Inbound traffic from established connections
HTTP & HTTPS connections (internet)
SSH connection on a custom port number

*filter
-N LOGGING
# Allow all loopback traffic (lo0)
-A INPUT -i lo -j ACCEPT
# Reject traffic to localhost that doesn't originate from lo0
-A INPUT ! -i lo -s 127.0.0.0/8 -j REJECT

# Allow inbound from established connections
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Allow HTTP and HTTPS connections from anywhere
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -m state --state NEW -j ACCEPT

# Allow ping from the home server only
-A INPUT -s 1.1.1.1 -p icmp -m icmp --icmp-type 8 -j ACCEPT

# Allow SSH traffic on custom port 4321
-A INPUT -p tcp -m state --state NEW -m tcp --dport 4321 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 4321 -j ACCEPT

# If the packet has made it this far without being accepted, it's going to be
# dropped. The logging chain will record some data about the packet first.
# The relevant log file is `/var/log/kern.log`.
-A INPUT -j LOGGING
-A LOGGING -p icmp -m limit --limit 5/min -j LOG --log-prefix "iptables-denied ICMP: " --log-level 7
-A LOGGING -p udp -m limit --limit 5/min -j LOG --log-prefix "iptables-denied UDP: " --log-level 7
-A LOGGING -p tcp -m limit --limit 5/min -j LOG --log-prefix "iptables-denied TCP: " --log-level 7
-A LOGGING -j DROP
COMMIT

Notes Re iptables and UFW

If iptables rules have been added, they will appear before the UFW rules - and the first rules will take precedence.

Packets drop through a chain of rules. If the iptables INPUT chain rule has a default drop policy, it matches every remaining packet that reaches it. With iptables, order is important. The first rule that natches a packet of data is the one that will be applied.

Once a decision is made to accept a packet, no more rules affect it. As our rules allowing ssh and web traffic come first, as long as our rule to block all traffic comes after them, we can still accept the traffic we want. All we need to do is put the rule to block all traffic at the end.

Ubuntu Community iptables HowTo

Procedure to Flush Rules

If you need to flush firewall rules, first ensure that the remote connection can be maintained:

# Set default policies to "ACCEPT"
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

# Flush the `nat` and `mangle` tables:
sudo iptables -t nat -F
sudo iptables -t mangle -F

# Flush all chains
sudo iptables -F

# Delete all non-default chains
sudo iptables -X

If you’ve been trying UFW, you may end up with a number of extraneous statements prepended “ufw-“. To remove these:

# Remove ufw rules from iptables
sudo iptables -D ufw-after-forward
sudo iptables -X ufw-after-forward
sudo iptables -X ufw-after-input
sudo iptables -X ufw-after-logging-forward
sudo iptables -X ufw-after-logging-input
sudo iptables -X ufw-after-logging-output
sudo iptables -X ufw-after-output
sudo iptables -X ufw-before-forward
sudo iptables -X ufw-before-input
sudo iptables -X ufw-before-logging-forward
sudo iptables -X ufw-before-logging-input
sudo iptables -X ufw-before-logging-output
sudo iptables -X ufw-before-output
sudo iptables -X ufw-reject-forward
sudo iptables -X ufw-reject-input
sudo iptables -X ufw-reject-output
sudo iptables -X ufw-track-forward
sudo iptables -X ufw-track-input
sudo iptables -X ufw-track-output

You might need to do something similar for ip6tables. Bash script for this procedure.

Resources

Ubuntu Community iptablesHowTo - ggood resource, with good info on logging
Digital Ocean - list & delete iptables rules
Digital Ocean - Set Up Firewall using iptables on Ubuntu 14.04
Firewall with UFW on Ubuntu - good tutorial, UFW as alternative to amending iptables directly
iptables Essentials - Digital Ocean
Good iptables tutorial
Testing with nmap - attacking WordPress
Log dropped packets
Use psad to detect network intrusion attempts
Intro to Fail2ban
Intro to iptables setup
Good Linode article on securing a server

Wget Commands

Tue, 16 Feb 2016 00:00:00 +0000

Wget is a free utility for download of files from the web. It is non-interactive, so can work in the background.

Download a file from GitHub or Gisthub

Click “raw” to access the URL of the raw file.

To download into the current directory:

wget https://link-to-raw-file.php

Download a List of Pages

This can be a useful way to collect images from a service like Unsplash. Image download links can be added on a separate line in a manifest file, which can be used by wget:

wget -i manifest

Download All Images From a webpage

wget -nd -H -p -A jpg,jpeg,png,gif -e robots=off http://www.website.com

-nd or --no-directories When retriving recursively, wget will not create a hierarchy of directories - all files will be saved to the current directory.
-H Allow spanning across hosts when doing recursive retrieving - be careful since foreign hosts may link to other hosts, sucking down more data than intended.
-p Page requisites - allows wget to downlaod all files necessary to display a HTML page, including inline images, sounds and stylesheets.
-A Specify a comma separated list of files to accept - in this example, image files.
-e Execute command as if it were part of the file .wgetrc. Executed after .wgetrc, therefore taking precedence - needed for the robots=off flag.
robots=off Turns off robots exclusion - by default wget respects robot exclusion rules set in the site /robots.txt
wget syntax guide
wget man page

Semantic Bootstrap Layout Classes with an Offset

Sat, 13 Feb 2016 00:00:00 +0000

Making CSS classes descriptive, or semantic, can help improve code maintainability by describing an elements purpose, rather than it’s presentational function.

People level this as a criticism against Bootstrap (and other CSS frameworks) - where column names are presentational rather than semantic. In other words, in a typical Bootstrap project the main content area might have a class name like .col-md-8, which is not semantic.

Fortunately, Bootstrap makes it pretty easy to define semantic classes that apply the built-in presentational logic, by use of the make-x-column() mixins.

Sage - Example of Semantically Defining Classes

The Sage WordPress starter theme (a great starting point for WordPress projects) defines a .main and a .sidebar class out of the box.

The column widths for these elements can then be set in a _variables.scss file.

I’ve modified the Sage definition of .main and .sidebar to add an offset to the sidebar, using the additional Bootstrap mixin make-sm-column-offset().

I find the offset really good for user experience - constraining the main content area generally makes for greater readability, and the extra whitespace between content and sidebar can reduce the sensation of clutter.

// Grid system

.main {

  // No sidebar, `.main` is full width
  // ---------------------------------------------------------------------------
  @include make-sm-column($main-sm-columns);

  // `.main` is contained by the `.sidebar-primary` parent class - this class is
  // added to the <body> of pages that display the sidebar. If a sidebar displays,
  // `.main` can take a reduced width.
  // ---------------------------------------------------------------------------
  .sidebar-primary & {

    // `.main` is narrower by 2 columns, to give room for the offset
    // -------------------------------------------------------------------------
    @include make-sm-column($main-sm-columns - $sidebar-sm-columns - 2 );

  }

}

.sidebar {

  // Set the width, then the offset, using Bootstrap mixins
  // ---------------------------------------------------------------------------
  @include make-sm-column($sidebar-sm-columns);
  @include make-sm-column-offset(2);

}

Resources

Get Column Names from a WordPress Database Table

Tue, 09 Feb 2016 00:00:00 +0000

Get a comma-separated string of all column names (Field names) for a given MySQL table.

Useful when working with custom tables.

Method 1: Using `$wpdb->get_col()`

<?php
// An array of Field names
$existing_columns = $wpdb->get_col("DESC {$table}", 0);

// Implode to a string suitable for inserting into the SQL query
$sql = implode( ', ', $existing_columns );

Method 2: Using `$wpdb->get_results()`

<?php
$cols_sql = "DESCRIBE $table";
$all_objects = $wpdb->get_results( $cols_sql );
$existing_columns = [];
foreach ( $all_objects as $object ) {
  // Build an array of Field names
  $existing_columns[] = $object->Field;
}
$sql = implode( ', ', $existing_columns );

Example

<?php
namespace Company\Project\Includes;
class Get {
  public static function rows( $table, $fields = 'all' ) {
    global $wpdb;
    $table = $wpdb->prefix . $table;

    // Build an array of all field names for this table
    // -------------------------------------------------------------------------
    $existing_columns = $wpdb->get_col("DESC {$table}", 0);
    $sql = implode( ', ', $existing_columns );
    if ( 'all' === $fields ) {
      return $wpdb->get_results( "SELECT $sql FROM {$table}" );
    }
    // Build a query for specific fields if these are passed in to the function
  }
}

Note that the implode() function can accept parameters in any order. Probably best to use it as shown, to maintain consistency with explode().

Resources

Good description of various methods, with pros & cons

Atom Crashes on Startup When A Repo has Uncommitted Changes

Thu, 04 Feb 2016 00:00:00 +0000

Atom is a free open source text editor “A hackable text editor for the 21st Century”. I really like it. It is fast, reliable, and hackable.

I recently experienced an odd crash when attempting to open a particular directory. The circumstances were a bit unusual and probably won’t crop up again, but I’m documenting this anyway to avoid repeating myself if it does reoccur.

Context

Building a git based generator to speed up my WordPress plugin workflow.

The generator scaffolds out a namespaced WordPress plugin. The project is closely based on the WordPress Plugin Boilerplate, except the class structure conforms to PSR-4 autoloading standards, and uses a Composer autoloader. Plugins devloped this way are not intended for distribution - they are for our own projects, where we always control the production environment.

Namespacing & autoloading may not be the “WordPress way”, but it encourages small, tidy classes and I like it a lot.

The workflow will involve invoking a BASH script that git clones the starter project, copies it, and renames files and file references within the new project. The script also gives the option to create and push to a new GitHub repo for the project.

The idea is to run one command, enter the new plugin name, and be ready to go.

The Problem

During testing my BASH script successfully:

Cloned the starter project
Copied and renamed the project
Amended filenames and references to files within the project

To save time, I chose NOT to set up a new git repo.

When trying to open the new project in Atom, Atom crashed. Safe mode did not help. Atom would open other directories without issue.

The Clue

This issue reported a crash when trying to open a file that had been modified but not committed.

Verification

On temporarily renaming the .git directory, the project repo opened in Atom without any problem.

Solution

When files were modified by my script, they contained uncommitted changes. I think this was what triggered the crash.

In this case, when creating a new project, if the user does not choose to create a new repo the original .git directory should be removed.

Fail2Ban and WordPress

Wed, 27 Jan 2016 00:00:00 +0000

Fail2ban is generally used to prevent brute force break-in attempts against SSH servers. These attempts typically come from automated scripts attempting multiple username/password combinations. Fail2ban acts by automatically identifying and acting against possible break-in attempts.

For SSH, you could cut these kind of attacks off by disallowing password login, requiring server administrators to connect by means of a public/private key pair. You might also use a non-standard port for SSH connections - but don’t rely on this alone!

Fail2ban helps to secure SSH connections by dropping the connection after a defined number of unsuccessful login attempts.

Fail2ban is not an excuse to use weak authentication methods. You should protect important services with public/private authentication and/or very strong passwords.

How Fail2ban Works

Fail2ban is basically a sin bin for hosts. It scans log files (e.g. /var/log/auth.log) and bans IP addresses that cause multiple authentication errors.

When Fail2ban detects that the number of failed login attempts from an IP have exceeded the pre-defined threshold, it updates the system firewall rules to reject new connections from those IP addresses for a set length of time. When the ban time is up, the ban is lifted.

Fail2ban allows you to configure:

A white-list of IP addresses to avoid accidental lockouts
The threshold number of unsuccessful logins
The time period over which malicious attempts are considered
The length of stay in the sin-bin
The precise actions taken - e.g. you can set up an email detailing RIPE data for each ban

Get Fail2ban

On Ubuntu, this is very simple:

sudo apt-get update
sudo apt-get install fail2ban

Fail2ban Configuration

Fail2ban config files are located in /etc/fail2ban.

There is a default config file: /etc/fail2ban/jail.conf - but DO NOT edit this, since the file can be modified by upgrades.

Copy the file:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

You can now configure Fail2ban by editing jail.local:

sudo nano /etc/fail2ban/jail.local

Within this file you can:

Create an ignoreip whitelist
Set the bantime
Set the conditions to determine illegitimate users: findtime and maxretry i.e. how many retries are allowed over a set time period
Set the MTA and email address if you are adding email notifications to your Fail2ban action
Set the action that Fail2ban takes when an illegitimate login attempt is detected

WordPress & Fail2ban

Beyond SSH, “brute force” activity focusing on WordPress login pages (e.g. yoursite.com/wp-admin) is very common. You could lock down access to wp-login.php by white-listing IP addresses. However, this isn’t always possible:

The site might be member-focused, with lots of members logging in from variable IP addresses
The client/site administrator may not have a fixed IP address

The benefit of using Fail2ban is that you can protect multiple sites on a server. Fail2ban is actively developed and maintained, and you should probably be running it on your Linux box anyway.

Unsuccessful WordPress log in attempts are not normally logged to the server logfiles - so they are not picked up on by Fail2ban.

To set up Fail2ban with WordPress you need to:

Add malicious login attempts to the auth.log for all managed WordPress installations
Properly configure Fail2ban to act on unauthorized login attempts - add the IP of malicious users to a firewall “deny” directive

Add Failed Login Attempts to `auth.log`

Add WP fail2ban as a must-use plugin. This writes unsuccessful login attempts to /var/log/auth.log.

By defining the appropriate constant in wp-config.php, the plugin can also block WordPress user enumeration attempts.

User enumeration is when a quesry string like http://yoursite.com/?author=1 is entered. On an unprotected WordPress site, this will return an archive page for user with an ID of 1 (usually the first administrative user), displaying the Username. This gives a would-be hacker the benefit of knowing your username.

If using the WP fail2ban plugin, this is easily blocked. Just add this line to wp-config.php:

define( 'WP_FAIL2BAN_BLOCK_USER_ENUMERATION', true );

For academic interest, here’s how this is achieved within the plugin:

<?php
if (defined('WP_FAIL2BAN_BLOCK_USER_ENUMERATION') && true === WP_FAIL2BAN_BLOCK_USER_ENUMERATION) {
  add_filter( 'redirect_canonical',
  function($redirect_url, $requested_url)
  {
    if (intval(@$_GET['author'])) {

      openlog();
      \syslog(LOG_NOTICE,'Blocked user enumeration attempt from '.remote_addr());
      bail();

    }

    return $redirect_url;

  }, 10, 2);

}

Set Up a Fail2ban Filter

A Fail2ban filter contains a regular expression which is used to define a logged event as a break-in attempt.

Filters should be added to the /etc/fail2ban/filter.d directory. The regexes in the filter MUST match the log entries for failed login attempts. The WP fail2ban plugin has a sample filter - this can be added to /etc/fail2ban/filter.d/wordpress.conf:

cd /etc/fail2ban/filter.d
wget http://plugins.svn.wordpress.org/wp-fail2ban/trunk/wordpress.conf

Set Up a WordPress Jail

A WordPress jail can be created for Fail2ban. The jail is defined in a configuration file located in the directory /etc/fail2ban/jail.d. For example:

sudo nano /etc/fail2ban/jail.d/wordpress.conf

# Add the following to the jail config file:
[wordpress]
enabled = true
filter = wordpress
logpath = /var/log/auth.log
port = http,https

The jail configuration sets the filter that will be used, as well as the path to the log file.

Fail2Ban Action

When the failed login attempt has been logged, a Fail2ban action should be configured.

There are several default actions available in /etc/fail2ban/jail.local. The most basic action involves adding a new firewall rule, but you can configure email notifications as part of the Fail2ban action.

Unban an IP Address

To unban an IP address from the WordPress Fail2Ban jail:

sudo fail2ban-client set wordpress unbanip 11.111.11.11

Troubleshooting

Make sure that the regex in the WordPress filter matches the string that is written into the log file on authentication failure.

It can be useful to run tail -f /var/log/auth.log and run a few failed logins. You’ll see the log entries in real time, and can cross reference this with the regex in /etc/fail2ban/filter.d/wordpress.conf.

Automation

It is a good idea to apply these measures to all WordPress sites that are hosted on public-facing servers. Once Fail2ban has been configured for a server, the logging functionality just needs to be enabled for all new sites. If you have a site creation script, the WP Fail2ban mu-plugin could be added automatically whenever a new site is created.

Logging: Repeated Entries

The Rsyslog daemon should be configured to turn OFF the $RepeatedMsgReduction configuration parameter. To achieve this, amend /etc/rsyslog:

# Use sed to amend the config setting
sudo sed -i 's/RepeatedMsgReduction\ on/RepeatedMsgReduction\ off/' /etc/rsyslog.conf

…alternatively, edit /etc/rsyslog.conf manually.

By default in Ubuntu 14.04, repeat messages are suppressed in the rsyslog daemon. This can allow attackers to slip through - we need the consecutive failed authentication messages to appear in the log. The rsyslog project team actually recommend setting this parameter to “off”.

My testing has shown that with repeat message suppression turned on, Fail2Ban is less effective at banning offenders - repeat messages don’t seem to be counted if they come through straight after the previous attempt. I found this very confusing until I realised that the RepeatedMsgReduction should be turned off.

After this amendment, restart the rsyslog daemon:

sudo service rsyslog restart

Limit Login Attempts is a very useful plugin that basically limits login attempts (as you’d expect!). It provides a simple interface that allows you to control the lockout time, number of retries allowed etc.

Whilst extremely useful, the plugin is not actively maintained (it has more than 1 million downloads, but was last updated 3 years ago).

Resources

Fail2ban Manual
WordPress User Enumeration
Using Fail2ban with WordPress
Fail2ban on Ubuntu
Charles Lecklider, wp-fail2ban
Ubuntu UFW Firewall
Repeated Message Logging: bug report
Notes on rsyslog daemon config w.r.t repeat messages.

Open Graph Tags for Jekyll

Tue, 19 Jan 2016 00:00:00 +0000

The Jekyll Open Graph Include

Save this as an include - for example, as /_includes/og.html.

<meta content="{{ site.title }}" property="og:site_name">
{% if page.title %}
  <meta content="{{ page.title }}" property="og:title">
{% else %}
  <meta content="{{ site.title }}" property="og:title">
{% endif %}
{% if page.title %}
  <meta content="article" property="og:type">
{% else %}
  <meta content="website" property="og:type">
{% endif %}
{% if page.excerpt %}
  <meta content="{{ page.excerpt }}" property="og:description">
{% else %}
  <meta content="{{ site.description }}" property="og:description">
{% endif %}
{% if page.url %}
  <meta content="{{ site.url }}{{ page.url }}" property="og:url">
{% endif %}
{% if page.date %}
  <meta content="{{ page.date | date_to_xmlschema }}" property="article:published_time">
  <meta content="{{ site.url }}/about/" property="article:author">
{% endif %}
{% if page.featured-image %}
  <meta content="{{ site.url }}/{{ page.featured-image }}" property="og:image">
{% else %}
  <meta content="{{ site.url }}/assets/images/medium-images/main-site-image.jpg" property="og:image">
{% endif %}
{% if page.categories %}
  {% for category in page.categories limit:1 %}
  <meta content="{{ category }}" property="article:section">
  {% endfor %}
{% endif %}
{% if page.tags %}
  {% for tag in page.tags %}
  <meta content="{{ tag }}" property="article:tag">
  {% endfor %}
{% endif %}

Document Head

Call the include in the document head:

{% include general/og.html %}

Gotchas

Facebook generally needs URLs - which must include ‘http’ or ‘https’ (rather than a relative link) - hence the use of site.url throughout rather than site.baseurl.

Facebook Debugging

You can have Facebook scrape your page using the Developers debugging utility. This provides lots of useful information regarding how Facebook views your document.

Inserting Images in Markdown Jekyll Posts

Tue, 19 Jan 2016 00:00:00 +0000

It’s often useful to add images in the body of a markdown document. While the featured image of a post or page can be specified in the YAML frontmatter, you can also call images directly within the markdown document.

This article outlines how to achieve this within Jekyll.

Kramdown Image Syntax

Insert the following into your .md document:

![image-title-here](/path/to/image.jpg){:class="img-responsive"}

The methodology for adding a class to the image took a bit of hunting.

Extra attributes (like the image class attribute) are added via span and block inline-attribute-list (IAL). As well as adding a class, you could specify image width and height.

Because the image is an inline (span level) element, the IAL needs to be on the same line. If you add it on the following line, it will wrap the image in a <p> tag and apply the class to that element.

References

Recursively Find and Replace Strings in Files

Tue, 15 Dec 2015 00:00:00 +0000

It can be useful to amend strings within project files - perhaps when spinning up a new project from a generated starting point.

I use this when setting the namespace of a new project.

Combine `find` and `sed`

Gnu find allows you to find files within a directory tree.

Sed is a unix utility that parses and transforms text, and is available for most operating systems.

Replace String in All Project Files

The command shown below is for Ubuntu.

find /path/to/my-project -type f -exec sed -i 's/OldString/NewString/g' {} \;

find will execute sed substituting {} with the filename found - in this case, all files under /path/to/my-project.

The \; ending means that sed is passed one file at a time.

References

Exec: execute a command
sed, a stream editor. Parse & Transform text

Find Ruby Gems

Tue, 08 Dec 2015 00:00:00 +0000

To find the location of Ruby gems on your system enter:

gem environment

The installation directory is denoted by INSTALLATION DIRECTORY. For example:

INSTALLATION DIRECTORY: /home/david/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0

Retrieve and Display WordPress ACF Repeater Field Data

Thu, 03 Dec 2015 00:00:00 +0000

The Advanced Custom Fields Pro WordPress plugin has a repeater field function that allows content editors to add multiple field values.

For example, the site editor might need the ability to select files that will be available for download. Setting up an ACF repeater field allows as many files as necessary to be associated with the post being edited.

The ACF interface is excellent and very user friendly - but I prefer to use native WordPress functionality to retrieve and display data that has been stored by ACF. This ensures that the data will still be displayed even if the client disables or removes the ACF plugin.

This tutorial documents how to retrieve and display a repeater field called downloads with a single subfield called file.

The file attachment IDs are retrieved, and these are used to build an array that contains the title, permalink, file type and file size.

This array is used to construct a list of “click to download” links.

Class Methods

This example shows the data-retrieval function and a function that builds a ul element to display files for download.

I have shown them as static methods in a class, but if you are using procedural programming rather than an OOP approach you could drop the functions in your theme’s functions.php file.

If you do this, you should prefix your functions with a pseudo namespace, and change the reference to self::get_repeater_file_download() in the_attached_downloads().

Retrieve the Data

This method fetches data from the postmeta table and returns a multi-dimensional array.

<?php
/**
 * Build file data for downloadable files
 *
 * Iterates through database entries in the postmeta table that have been saved
 * in the Advanced Custom Fields repeater field format:
 * `fieldname_' . $i . '_subfieldname`.
 *
 * @param  string|int $post_ID The current post ID
 * @return array               Array of file data
 */
public static function get_repeater_file_download ( $post_ID ) {

  // The number of entries for this repeater field
  $files = get_post_meta( $post_ID, 'downloads', true );

  if ( !$files ) {
    return;
  }

  $file_data = [];

  for( $i = 0; $i < $files; $i++ ) {

    $file_ID  = get_post_meta( get_the_ID(), 'downloads_' . $i . '_file', true );
    $file     = get_attached_file( $file_ID );
    $file_data[]  = [
      'filesize'  => size_format( filesize( $file ) ),
      'filetype'  => wp_check_filetype( $file )['ext'],
      'url'       => wp_get_attachment_url( $file_ID ),
      'title'     => get_the_title( $file_ID )
    ];

  }

  return $file_data;

}

Display Data

This builds the HTML for a list of downloadable links using data from the method above.

The download attached files <ul> can be displayed within template files by calling <?php ClassName::the_attached_downloads(); ?>.

  
<?php
/**
 * Build a ul with attached file titles and links
 *
 * @return string HTML list of downloadable files
 */
public static function the_attached_downloads() {
  $downloads = self::get_repeater_file_download( get_the_ID() );
  if( !$downloads ) {
    return;
  }

  ob_start();
  ?>
  <h2>Downloads</h2>
  <p>Click the link to download.</p>
  <ul>
  <?php

  foreach( $downloads as $download ) {
    ?>
    <li>
      <a href="<?= $download['url']; ?>">
        <?= $download['title']; ?>
      </a> (<?= $download['filetype']; ?>, <?=$download['filesize']; ?>)
    </li>
    <?php
  }

  ?>
  </ul>
  <?php

  echo ob_get_clean();

}

Rebuild WordPress Site From Incremental Backup

Thu, 26 Nov 2015 00:00:00 +0000

This involves creating a new database, importing the backed up database, replacing site files and connecting site files to the new database.

Create Blank Database

# Replace "new_db_name", "new_user" and "12345"
# You will need to enter the MySQL password for the root user
mysql -u root -p -e "create database new_db_name; GRANT ALL PRIVILEGES ON new_db_name.* TO new_user@localhost IDENTIFIED BY '12345'"

Import Copy of Backup Database

After creating a new database, make sure you are logged out of the MySQL shell (ctrl + c), then enter:

mysql -u root -p new_db_name < database_to_import.sql

Set Up Files

Add backup site files to document root
Amend wp-config.php - enter the correct database name, user and password

If deploying to the same URL, you are done. If you are migrating the site, you need to edit database entries, since the site URL will be different.

Migration: Amend Database

Start a MySQL session for the new database:

mysql -u root -p new_db_name

Check you’re working in the right database:

mysql> SELECT DATABASE();
+---------------+
| DATABASE()    |
+---------------+
| new_db_name  |
+---------------+

Amend the following code block to reflect your old and new domain names:

UPDATE wp_options SET option_value = replace(option_value, 'http://old-domain.com', 'http://new-domain.com') WHERE option_name = 'home' OR option_name = 'siteurl';
UPDATE wp_posts SET guid = replace(guid, 'http://old-domain.com','http://new-domain.com');
UPDATE wp_posts SET post_content = replace(post_content, 'http://old-domain.com', 'http://new-domain.com');
UPDATE wp_postmeta SET meta_value = replace(meta_value,'http://old-domain.com','http://new-domain.com');

NB: if you use a custom database table prefix, replace the wp_ prefixes in the example code block

Migration: Flush Rewrite Rules

Jekyll Contact Forms Using Formspree

Wed, 25 Nov 2015 00:00:00 +0000

Forms need somewhere to be submitted to, and form data needs to be processed.

For WordPress/PHP sites, this is pretty trivial. The form posts data to a PHP script, which takes certain actions - adds data to the database, sends email, etc.

If you’re embedding a form on a Static site (in this case Jekyll), you could create a PHP form processor provided that the server is running PHP. A simpler alternative would be to use a third party email based service like Formspree - which allows 1000 emails per month on the free tier.

Formspree Form Markup

This page is testing a Formspree contact form.

The form markup is as follows:

<form class="form-horizontal" action="//formspree.io/youremail@yourweb.com" method="POST">
<fieldset>
  <div class="form-group">
    <input type="text" name="name" placeholder="Your Name">
  </div>
  <div class="form-group">
    <input type="email" name="_replyto" placeholder="Your Email">
  </div>
  <div class="form-group">
    <textarea class="form-control" id="textarea" name="message">Your Message</textarea>
  </div>
  <div class="form-group">
    <input type="submit" value="Send">
  </div>
    <input type="text" name="_gotcha" style="display:none" />
</fieldset>
</form>

Once you submit the form for the first time, you’ll get a confirmation email to prevent spam submissions from random sites.

This page is testing the Formspree form.

Contact Form

Get in touch!

JavaScript Variables from Markdown Includes in Jekyll

Sat, 21 Nov 2015 00:00:00 +0000

The problem: Adding JavaScript variables in the document head makes them available to scripts that execute on the page. Building JavaScript variables using Jekyll include tags results in a breaking error due to a sneaky trailing blank line that is rendered to the output HTML. This can be fixed by filtering the output with the strip_newlines liquid filter.

JavaScript Variables

Within Jekyll, we can add variables that have been set in _config.yml or in other data files. This allows us to easily define variables in a central location.

In this case, we are passing variables to a Google map script. These variables will be used to customise map styles, markers and infowindows. The variables could easily be held in a CSV format file to allow easy data entry by non-technical people. The description field for the main info window needs to be properly formatted HTML - so it can’t easily or conveniently be held in a data file field. It should be edited in a markdown file which is then rendered to the page. The reference to this markdown file is held in the data file /_data/map.yml:

latitude: 52.741865
longitude: -8.772812
zoom: 11
height: 548px
main-brand-colour: "#398A8D"
main-marker: assets/images/marker.png
secondary-marker: assets/images/marker-2.png
title: Cozy Cottage
# description refers to a markdown file that will be the source
# for a HTML string included in the JS variable
description: /content/contact/map-description.md

HTML Include

The following file, map.html, is included in the <head> tag conditionally on the page title being contained in an array of map-displaying pages:

<style>#map-canvas { width: 100%; height: {{ site.data.map.height }}; }</style>
{% capture map_description %}{% include {{site.data.map.description }}%}{% endcapture %}
<script>
var cwCentre = {
  latitude:{{ site.data.map.latitude }},
  longitude:{{ site.data.map.longitude }},
  zoom:{{ site.data.map.zoom }},
  mainMarker:"{{ site.baseurl}}/{{ site.data.map.main-marker }}",
  secondaryMarker:"{{ site.baseurl}}/{{ site.data.map.secondary-marker }}",
  mainBrandColour:"{{ site.data.map.main-brand-colour }}",
  title: "{{ site.data.map.title | escape }}",
  description:"{{ map_description | markdownify | strip_newlines }}",
};
console.log(cwCentre);
var markers = [
  {% for location in site.data.map-coords %}
    ['{{ location.name | escape }}',{{ location.latitude }}, {{ location.longitude }}, '{{ location.description | escape }}']{% unless forloop.last %},{% endunless %}
  {% endfor %}
];
</script>
<script src="https://maps.googleapis.com/maps/api/js"></script>

Strip Newlines

If you don’t add the strip_newlines filter to the markdownified output, the JavaScript variable will have an additional newline. It will look like this in the output HTML:

<script>
var cwCentre = {
  latitude:52.741865,
  longitude:-8.772812,
  zoom:11,
  mainMarker:"/cozy-cottage/_site/assets/images/marker.png",
  secondaryMarker:"/cozy-cottage/_site/assets/images/marker-2.png",
  mainBrandColour:"#398A8D",
  title: "Cozy Cottage",
  description:"<p>Test content</p>
",
};
console.log(cwCentre);
...
</script>

This will trigger an error, due to the bogus character at the end of the line.

You’ll see the error: ‘Uncaught SyntaxError: Unexpected token ILLEGAL’ in the browser console and the script will fail.

The strip_newlines filter sorts this out.

References

Using rsync to Copy Incremental Backups

Sat, 31 Oct 2015 00:00:00 +0000

Copy a directory comprised of incremental backup to a separate hard disk.

When an incremental backup makes use of hardlinks, these need to be preserved when copying the backup directory. Otherwise, all files are copied individually, taking up a lot more space on the destination directory.

File Format
If copying to an external drive, it should be formatted to an ext3 or ext4 filesystem. FAT(32) formatting won’t work - it doesn’t support permission & ownership data

Maintain Hardlinks

When rsync’ing with the -a option, hardlinks are NOT automatically preserved:

Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H.

rsync Man page

Hardlinks are generally a critical element of incremental backups. Hardlinks allow multiple directory/file entries to be associated with a single inode. If you do not include the rsync -H argument, when copying a set of incrementally backed-up directories rsync would copy files in their entirety. This would lose the space saving benefits of the hardlinked incremental backup.

rsync Command

rsync -az --exclude='*.zip' -H --progress /source-drive/name /destination-drive

Explanation:

-a Archive mode; recursive, copy symlinks as symlinks, preserve permissions, preserve modification times, preserve group, preserve owner, preserve special & device files.
-z Compress the file data being transferred
--exclude='*.zip' do not transfer files with a .zip extension
-H Preserve hardlinks
--progress print info about the transfer (for the terminally bored, but at least you can see it’s doing something)

Note that the lack of a trailing slash on the source directory will result in this directory being created in the destination directory (if it doesn’t already exist).

References

Relative File Paths in PHP

Tue, 06 Oct 2015 00:00:00 +0100

Relative file paths are used when including or requiring files.

In the context of a WordPress plugin or theme, you may be referencing a file that is in a ‘parallel’ directory.

In the example below, the plugin root directory is referenced by multiple use of the PHP dirname() function.

dirname() returns a parent directory’s path.

Target file: /includes/class-staff-area-access.php
Originating file: /admin/partials/access-check.php

// function is called in a subdirectory nested two deep from the project root
plugin_dir_path( dirname ( dirname( __FILE__ ) ) ) . 'includes/class-staff-area-access.php' ;

// This achieves the same result, using the __DIR__ magic constant:
echo plugin_dir_path( dirname( __DIR__ ) ) . 'includes/class-staff-area-access.php' ;

Notes on the PHP Magic constants:

__FILE__ - the full path and filename of the file (with symlinks resolved)
__DIR__ - the directory of the file

dirname() in PHP 7

In PHP 7, dirname() will accept an optional levels parameter. This means the above code could be replaced with:

// PHP 7:
echo dirname( __DIR__ , 2 ) . '/includes/class-staff-area-access.php' ;

WordPress Function: plugin_dir_path()

The WordPress function plugin_dir_path() gets the filesystem directory path for the file passed in. It includes a trailing slash.

The function is a wrapper for trailingslashit( dirname( $file ) )

Resources

Control File Access by IP in Apache 2.4

Thu, 01 Oct 2015 00:00:00 +0100

Denying access to wp-login.php for all but a set of whitelisted IP can be a good way of enhancing site security - provided that the client has a fixed IP address.

We typically add such access controls within a .htaccess file in the document root of a project, leaving login access for our own IP address and that of the site owner.

You might occasionally need to temporarily whitelist an additional IP address, but this is easy to do.

Restricting access by IP address is no substitute for a proper username/password policy - but it may be a useful additional layer, since would-be attackers don’t even get a chance to knock on the door.

Under Apache 2.2, you could use these directives within a .htaccess file:

# ==============================================================================
# Whitelisted IP access for wp-login.php
# ==============================================================================
<files wp-login.php>
order deny,allow
deny from all

# whitelist Your First IP address
allow from xxx.xxx.xxx.xxx
# whitelist Your Second IP Address
allow from xxx.xxx.xxx.xxx
# whitelist Your Third IP Address
allow from xxx.xxx.xxx.xxx

</files>

# ==============================================================================
# Protect specified files from direct access
# ==============================================================================
<FilesMatch "^(wp-config\.php|php\.ini|php5\.ini|install\.php|php\.info|readme\.html|bb-config\.php|\.htaccess|\.htpasswd|readme\.txt|timthumb\.php|error_log|error\.log|PHP_errors\.log|\.svn)">
Deny from all
</FilesMatch>

Whilst the Allow, Order, and Deny directives still work in Apache 2.4, they are deprecated:

The Allow, Deny, and Order directives, provided by mod_access_compat, are deprecated and will go away in a future version. You should avoid using them, and avoid outdated tutorials recommending their use.

-Apache 2.4 Documentation

Unfortunately, there is not a lot of literature on how to properly set up such restrictions on Apache 2.4 - without relying on mod_access_compat.

Deny Access Completely

In Apache 2.2:

Order deny,allow
Deny from all

In Apache 2.4 this becomes:

Require all denied

Restrict Access by IP address: Comparison of Apache 2.2 and 2.4

Allow from a particular IP in Apache 2.2:

Order Deny,Allow
Deny from all
Allow from xxx.xxx.xxx.xxx

Allow from a particular IP in Apache 2.4:

Require ip xxx.xxx.xxx.xxx

TL;DR Restrict Access Apache 2.4

# ==============================================================================
# Restrict access to WordPress login page by IP
# See: http://httpd.apache.org/docs/2.4/mod/core.html#files
# ==============================================================================
<Files "wp-login.php">
    Require ip 123.123.123.123
</Files>

If you have full access to Apache config on your server, you can enable these directives for all virtual hosts by adding them to the Apache config file:

sudo nano /etc/apache2/conf-enabled/security.conf

Resources

One of the rare useful guides on access control in Apache 2.2 vs 2.4
Apache 2.4. docs
Apache 2.4 Require directive - maybe it’s because it’s Friday night after a long week, but this page made my brain hurt…
A better resource on the Require directive
<Files> Directive
<FilesMatch> Directive
Options directive

Page Sections in Jekyll

Mon, 28 Sep 2015 00:00:00 +0100

The objective is to set up a services page that will have multiple sections. Each section will contain a block of HTML and an image.

It should be easy to edit section content in obvious markdown files, and we should avoid duplication as much as possible.

Data Driven Sections

Jekyll can read data files in either YAML, JSON or CSV format. Data is loaded from files with these formats located in the _data directory.

Data files allow us to specify files to be used as includes or images, and help avoid repetition in templates. You just set up the data file correctly, and edit the (content) source files without touching the template.

Data Setup

Though I have tinkered with using CSV format data files (to make it easier to harvest user generated content), I prefer the YAML format - basically because I prefer how the data looks in a text editor. If you do use CSV, you need to include a header row.

For this post, we’ll focus on the YAML format.

Create a new data file: /_data/service-sections.yml.

Enter data, being careful with whitespace and tabs:

- title: "Pleated Blinds"
  description: "domestic/pleated-blinds.md"
  image: "assets/images/medium-images/pleated-blinds.jpg"
  image-title:
  service-category: domestic

- title: "Roller Blinds"
  description: "commercial/roller-blinds.md"
  image: "assets/images/medium-images/Rollerblinds.jpg"
  image-title:
  service-category: commercial

- title: "Plantation Shutters"
  description: "domestic/plantation-shutters.md"
  image: "assets/images/medium-images/plantation-shutters.jpg"
  image-title:
  service-category: domestic

- title: "Vertical Blinds"
  description: "domestic/vertical-blinds.md"
  image: "assets/images/medium-images/vertical-blinds.jpg"
  image-title:
  service-category: domestic

- title: "Roman Blinds"
  description: "domestic/roman-blinds.md"
  image: "assets/images/medium-images/roman-blinds.jpg"
  image-title:
  service-category: domestic

In this case, the data file serves as a way of referencing the files that contain the content.

Under each YAML block, the description field is a path to a markdown include. I’ve tried to make this as logical as possible, by having an appropriate directory structure within the _includes directory: _includes/services/service-name/content-file.md… so if you want to add content to the section on “Roller Blinds” on the “Commercial” services page, you need to edit _includes/services/commercial/roller-blinds.md.

In a similar way, we can reference images, titles and other data for each section.

The HTML Include

The section data are looped through within a HTML include, which is in turn inserted in the service_page template. IN this case, the include is _includes/layouts/services-sections.html.

The include does the following:

Sets up a for loop that goes through each data block in the specified data file.
Makes section display conditional
Builds variable content into the correct HTML markup

Sections displayed must have a service-category that corresponds to the page service-category (which needs to be set in the page front-matter). This allows us to easily define section content for different service pages from within the data file.

For example, this is _includes/layouts/services-sections.html:

{% for section in site.data.service-sections %}
  {% if page.service-category == section.service-category %}
    <div class="row section">
      <div class="col-sm-6">
        <h2>{{ section.title }}</h2>
        <!--
        To properly render markdown includes as HTML, the include tag must be
        captured as a string, and this must be passed to the `markdownify` filter.
        -->
        {% capture section_description %}{% include services/{{ section.description }} %}{% endcapture %}
        {{ section_description | markdownify }}

        <!--
        The following will not parse the markdown include - it won't be properly
        rendered into HTML and will display as the original markdown.
        -->
        {% comment %}
        {% include services/{{ section.description }} %}      
        {% endcomment %}
      </div>
      <div class="col-sm-5 col-sm-offset-1">
        <div class="thumbnail">
          <!--
          Build the image markup for this section. If an image-caption has been
          defined, display it.
          -->
          <img src="{{ site.baseurl }}/{{ section.image }}" title="{{ section.title }}" alt="{{ section.title }}" class="img-responsive">
          {% if nil != service.img-caption %}
            <div class="img-caption">{{ service.img-caption }}</div>
          {% endif %}
        </div>
      </div>
    </div>
    <hr>
  {% endif %}
{% endfor %}

This is included within the index.html page of each service. For example, domestic/index.html.

index.html

The service page (service-name/index.html) front matter sets the page title, meta description etc. It also sets the service-category, which is used in the layouts/services-sections.html include to build the correct section content. For example, this is domestic/index.html:

---
layout: service-page
title: Domestic Blinds & Shutters
group: navigation
title: Domestic Blinds and Shutters
description: Quality domestic blinds & shutters in Limerick, Clare, Tipperary
service-category: domestic
intro-content: domestic/domestic.md
intro-image: assets/images/logo.png
intro-image-title: "Irish Blinds & Shutters"
extra-content: domestic/domestic-col-2.md
---
{% capture service_intro %}
  {% include services/{{ page.intro-content }} %}
{% endcapture %}
{% capture service_extra %}
  {% include services/{{ page.extra-content }} %}
{% endcapture %}
<div class="row">
  <div class="col-md-6">
    <header class="post-header">
      <h1>{{ page.title }}</h1>
    </header>
      {{ service_intro | markdownify }}
  </div>
  <div class="col-md-5 col-md-offset-1">
    <div class="thumbnail hidden-sm">
      <img src="{{ site.baseurl }}/{{ page.intro-image }}" title="{{ page.intro-image-title }}" alt="{{ page.intro-image-title }}" class="img-responsive">
    </div>
    {% if page.col-2-content != empty %}
      {{ service_extra | markdownify }}
    {% endif %}
  </div>
</div>
<hr>
<!-- Include the service sections here. -->
{% include layouts/services-sections.html %}
<!-- End service sections -->
<div class="row">
  <div class="col-md-6">
    <h2>Contact Us Now to Get Started</h2>
    {% include general/body-phone.html %}
  </div>
</div>

Conclusion

Did we achieve the objectives of data-driven page sections? Kind of. Overall it’s a pretty good solution - it means that we can manage site content in a fairly straightforward way.

There are still more edit points than I’d like, and I don’t like the way that the service sections are bunched up in a single data file.

I’d prefer to be able to create separate data files for each page category - but I had trouble creating a dynamic reference to the data file whilst setting up the for loop. I still have a lot to learn about Jekyll (and Ruby), so it could well be that I’m missing something obvious.

Resources

Jekyll Data Files
Interesting article on dynamic navigation in Jekyll
Reference to markdown includes

Find and Replace Filenames

Sun, 27 Sep 2015 00:00:00 +0100

This might be needed when starting a project by cloning a scaffold boilerplate from Github (e.g. the WordPress boilerplate plugin).

TODO Build a BASH script to scaffold projects with automatic renaming of files.

Rename filenames in WP Plugin Boilerplate

Rename files and directories recursively. Run from the project root:

# Test run
find . -iname "*" -exec rename -n 's/plugin-name/carawebs-codename/g' '{}' \;

# Apply the Rename, verbose output lists changes
find . -iname "*" -exec rename -v 's/plugin-name/carawebs-codename/g' '{}' \;

Options

The . runs the find command recursively in the current directory - in this case it finds all files & directories
The -exec argument executes rename for every matching file found - all files.
'{}' is replaced with the path name of the file.
\; marks the end of the exec expression.
The -n option on rename runs the command in “no act” mode.

The substitution takes the form s/target/replacement/[gi], where:

target is the string to be replaced
replacement is the replacing string

The g option is for ‘global’ action - all instances are replaced.

An i denotes a case insensitive rename.

Examples

Find .jpg files with the string ‘holiday’ in the filename, replacing this with ‘vacation’:

find . -type f -name '*.jpg' | rename 's/holiday/vacation/'

Resources

rsync Notes- Push a WP Plugin

Tue, 22 Sep 2015 00:00:00 +0100

Use rsync to push a plugin from a local development machine to a remote server.

Sync a Single WP Plugin

Trailing slash or not? That is the question…

rsync -raz -e 'ssh -p 1234' --progress /var/www/localsite/wp-content/plugins/plugin-name user@X.X.X.X:/var/www/sitename.com/public_html/wp-content/plugins

No trailing slash on source path: rsync will create the last directory of the path before copying content. In the case above, you’d end up with /var/www/sitename.com/public_html/wp-content/plugins/plugin-name/*files.*.
Append a trailing slash - rsync will only copy the contents (subdirectories and files) of the source directory directly to the target - Not the directory itself.

I’m always forgetting this - it seems counter-intuitive!

Resources

http://devblog.virtage.com/2013/01/to-trailing-slash-or-not-to-trailing-slash-to-rsync-path/ http://qdosmsq.dunbar-it.co.uk/blog/2013/02/rsync-to-slash-or-not-to-slash/

Finding Files in git History

Mon, 21 Sep 2015 00:00:00 +0100

You may need to find a file in your commit history - perhaps the file was deleted, and you need the functionality that it contained.

Show Commits where the File Changed

This will show commits where the file changed, even if the file has been deleted:

git log -- /path/to/target/file

This will provide a log of commits that involved the specified file changing.

View the Commit Associated with File Deletion

git log -1 -- /path/to/target/file

This shows the last commit that contained the file. You could specify a different number to see the last x commits.

View the Relevant Commit

Once you have determined the commit reference, you can view the file by selecting the relevant in an online repo.

You could also check out the commit:

git checkout d33ffc9

If you checkout the commit, you can view and edit files without affecting the current state of the project. You need to checkout your working branch (usually ‘master’) to go back to work.

It’s usually enough to grab the commit reference and view this in the online repo (Bitbucket or Github).

Resources

Comprehensive Git Tutorial by Lars Vogel

Bootstrap 4 Navbar

Mon, 21 Sep 2015 00:00:00 +0100

There are quite a few changes in Bootstrap 4 (it’s SASS rather than LESS for a start!).

One of these changes is the navbar. Here’s the markup for the basic navbar on this page:

<header id="top">
  <div class="container">
    <h1><a href="/">Web Development Notes</a></h1>
    <h4>Various Cheat Sheets and Resources by David Egan/Carawebs.</h4>
    <section id="main-nav">
      <nav class="navbar">
        <ul class="nav navbar-nav">
          <li class="nav-item"><a href="http://carawebs.com" class="btn nav-link">Carawebs Main Site</a></li>
          <li class="nav-item"><a href="/" class="btn nav-link">Dev-Notes Home</a></li>
          <li class="nav-item"><a href="/linux/" class="btn nav-link">Linux</a></li>          
          <li class="nav-item"><a href="/wordpress/" class="btn nav-link">WordPress</a></li>         
          <li class="nav-item"><a href="/tools/" class="btn nav-link">Tools</a></li>          
          <li class="nav-item"><a href="/contact" class="btn nav-link">Contact</a></li>          
        </ul>
      </nav>
    </section>
  </div>
</header>

New in Bootstrap 4

nav-item class on the li elements
nav-link class on the nav links themselves

I’m still digging in to Bootstrap - I intend to add more navbar fragments as I build them.

One thing to note is that the navbar doesn’t yet collapse at mobile views.

Liquid/Jekyll Markup

This is a Jekyll site, so I build the navbar shown above like this:

<header id="top">
  <div class="container">
    <h1><a href="{{site.baseurl}}/">{{ site.name }}</a></h1>
    <h4>{{ site.description }}</h4>
    <section id="main-nav">
      <nav class="navbar">
        <ul class="nav navbar-nav">
          <li class="nav-item">
            <a href="http://carawebs.com" class="btn nav-link">Carawebs Main Site</a>
          </li>
          {% for node in site.navigable-pages %}
          {% if page.url == node.url/ %}
          <li class="nav-item">
            <a href="{{site.baseurl}}{{ node.url }}" class="btn nav-link active">{{ node.title }}</a>
          </li>
          {% else %}
          <li class="nav-item">
            <a href="{{site.baseurl}}{{ node.url }}" class="btn nav-link">{{ node.title }}</a>
          </li>
          {% endif %}
          {% endfor %}
        </ul>
      </nav>
    </section>
  </div>
</header>

Jekyll will then render the final HTML as shown above.

Install a Dev Branch with Bower

Sun, 20 Sep 2015 00:00:00 +0100

I decided to try out the Bootstrap 4 CSS framework on this website. It’s still in Alpha, and not recommended for production sites.

I wanted to install Bootstrap by means of Bower. You normally install via Bower by moving to the project root directory and entering:

bower install <package> --save-dev

In the case of Bootstrap, you’d enter:

bower install bootstrap --save-dev

The Bootstrap assets would be downloaded and automatically added to a bower_components directory in your project.

package.json

The --save-dev flag adds the package as a project dependency in your bower.json file - so running bower install against your project will install the package you’ve just added.

If you don’t have a bower.json file, you can run bower init to generate one.

This method fetches the latest stable version, so it’s no good for fetching the Bootstrap 4 development branch.

Install endpoints

Fortunately, there are a range of endpoints for the install command. You can install a package version - a valid range, commit or branch by means of bower install <package>#<version>.

<package> refers to a package URL, physical location or Bower registry name.
<version> can be a:
- semver version (#1.2.3)
- version range
- git tag
- git commit SHA
- git branch

Install Bootstrap 4 Dev Branch

Determine the relevant branch (via GitHub) and add it as an endpoint:

bower install bootstrap#v4-dev --save-dev

Resources

Bower Install

WordPress Error Logging

Tue, 15 Sep 2015 00:00:00 +0100

In general, error logging should be enabled in the development environment and disabled in production environments.

To enable WordPress error reporting to the browser, and to enable error logging to file, add the following lines to wp-config.php:

// Enable error reporting output to browser. Default value is false.
define( 'WP_DEBUG', true );

// log errors to `/wp-content/debug.log`. Useful when debugging code that does not output to browser.
define('WP_DEBUG_LOG', true);

Enabling WP_DEBUG will cause all PHP errors, notices and warnings to be displayed. This is likely to modify the default behavior of PHP which only displays fatal errors and/or shows a white screen of death when errors are reached.

Showing all PHP notices and warnings often results in error messages for things that don’t seem broken, but do not follow proper data validation conventions inside PHP. These warnings are easy to fix once the relevant code has been identified, and the resulting code is almost always more bug-resistant and easier to maintain.

– WordPress Codex

Browser Debugging

Enabling WP_DEBUG provides useful error messages rather than the “white screen of death”.

Even when code is not broken, debugging output will highlight deprecated WordPress functions and other problems.

Logging

The WP_DEBUG_LOG option is very useful for situations that do not involve output to the browser - for example AJAX requests. It is also useful when debugging classes or functions that pass data to other scripts.

You can make code write to the debug log using the PHP error_log() function.

For example, in the following code fragment, taken from a class method, the $column_name variable is output to the error log - so that we can check we’re getting the right values at this point in the programme:

<?php

...

    // Headings
    // -------------------------------------------------------------------------
    foreach ( $wpdb->get_col( "DESC " . $table_name, 0 ) as $column_name ) {

      // output $column_name to the error log
      error_log( $column_name );
      $csv_output = $csv_output . $column_name . $this->delimiter;

    }

Monitoring the Log File

The log file must be owned by the server process - in the case of Ubuntu, this is ‘www-data’.

I often chown ownership back and forth for various reasons - so wrong ownership of debug.log often trips me up. When developing a theme or plugin, it’s probably best to create an alias giving ownership of just the theme or plugin directory to $USER - leaving the rest of the development site owned by www-data.

It can be useful to view the debug.log in realtime. The tail utility with the -f (follow) option is good for this.

Open a terminal and add the following:

tail -f /path/to/debug.log

This outputs the last 10 lines of debug.log, and the output data will be appended if the file grows - so you’ll have a realtime monitor of the debug log in your terminal. If you need more lines, append the -n flag to the tail command. To output the last 50 lines of the log file:

tail -f /path/to/debug.log -n 50

Alternatively, open debug.log in a text editor (e.g. gedit). After running your code, activate the editor window - under Ubuntu at least you’dd see a “changed on disk” notification - relaoding the file will show the new error output.

Resources

Debugging in WordPress: Codex

WordPress Filter Hooks

Wed, 02 Sep 2015 00:00:00 +0100

Filters can operate on content snippets or on arrays.

Filters are the hooks that WordPress launches to modify text of various types before adding it to the database or sending it to the browser screen. Your plugin can specify that one or more of its PHP functions is executed to modify specific types of text at these times, using the Filter API.

– WordPress Codex

To set up content for filtering, use the apply_filters() function:

<?php apply_filters( $tag, $value, $var ... ); ?>

Create the Hook

Apply filters attached to ‘custom_content_filter’ to $content. This would be typically added to a template file:

$content = apply_filters ( 'custom_content_filter', 'alert alert-info' );

Create Filter

Create the filter and attach to the filter hook. Typically in custom functions file.

function carawebs_wrap_content( $content, $div_class ) {

    if ( 'blockquote' == $wrap_tag ) {

        return "<blockquote>" . $content . "</blockquote>";

    } else {

        return "<div class='$div_class'>" . $content . "</div>";

    }

}

// Filter the content passed in from 'custom_content_filter' with the
// specified callback function - in this case, 'carawebs_wrap_content'.

add_filter ( 'custom_content_filter', 'carawebs_wrap_content' );

WordPress Filters

There are many built-in WordPress filters. For example, ‘the_content’ filters content after database retrieval and before printing to the screen. To use this filter, it’s simply a matter of passing the content. This can be very useful when fetching post content outside the loop using get_the_content() - which returns unfiltered content.

$raw_content = get_the_content( $current_ID );
$filtered_content = apply_filters( 'the_content', $raw_content );

Filter functions can be added using the add_filter() function. For example, this function prepends a post thumbnail to the $content passed in to the ‘the_content’ filter - which is usually the post content:

// From WP Codex.
// @see: https://codex.wordpress.org/Plugin_API/Filter_Reference/the_content

add_filter( 'the_content', 'featured_image_before_content' );

 function featured_image_before_content( $content ) {

    if ( is_singular('post') && has_post_thumbnail()) {

        $thumbnail = get_the_post_thumbnail();
        $content = $thumbnail . $content;

		}

    return $content;
}

From WP Codex.

Resources

Good decription of Filters

WordPress Action Hooks

Thu, 20 Aug 2015 00:00:00 +0100

Placing do_action( $tag, $arg ) within a theme or plugin allows all functions attached to the action hook $tag to be invoked at this specific point.

A callback function is registered to the action hook by means of the add_action() function, using the $tag defined in do_action().

This enables you to create an API for a theme or plugin - callbacks can be registered for a custom action.

To register a callback, use add_action(). This will typically be in the theme custom functions, where the related $function_to_add will also be defined:

<?php
add_action( $hook, $function_to_add, $priority, $accepted_args );

To execute the callback, use do_action(). This will typically be in the template or layout partial - anywhere where you’d like to give developers (or yourself) the ability to hook in additional content:

<?php
do_action( $tag, $arg_a, $arg_b, $etc );

Simplest Example

Simple example of an action hook - allows extra content to be added by the theme.

<?php
do_action( 'before_address' );
echo carawebs_custom_address();

// This could also be defined as part of a function or class method like this:
function carawebs_before_address() {

    do_action( 'before_address' );

}
// The function call `carawebs_before_address()` would then be added to the template.

We can now add HTML before the address block, by hooking to the ‘before_address’ action hook. This gives layout control to the theme, which is proper.

The following code would typically be located in the theme custom functions.

namespace Carawebs\Namespace;

function address_title() {

  echo "<h2>A Custom Title</h2>";

}

add_action (  'before_address', 'Carawebs\Namespace\address_title', 1 );

Arguments for the Action Hook Callback

The callback function that is registered by means of add_action() can accept arguments. The number of arguments must agree with those defined in the corresponding do_action().

This means that variables defined in the same scope as the do_action() call can be made available to the callback registered in add_action().

Within the template (or better within a function invoked from the template):

$i = 1;

foreach( $people as $person ){

  // Pass $i as an argument to do_action
  echo do_action( 'carawebs_before_person', $i );
  echo $person['name'] . "<br />";
  echo $person['title'] . "<br />";
  echo "<hr>";

  $i++;

}

Within the theme custom functions:

// $i is passed into the function by the do_action() setup.
function carawebs_person_intro( $i ) {

  // Echoes a numbered intro
  echo "This is data for person $i";

}

// add_action( $tag, $callback, $priority, $accepted_arguments )
add_action( 'carawebs_before_person', 'carawebs_person_intro', 1, 1 );

Resources

Add a .htaccess File to a Jekyll Site

Wed, 05 Aug 2015 00:00:00 +0100

Adding a .htaccess file to your document root allows fine control over access permissions.

Amongst other things, .htaccess rules can set:

In-browser caching
Access - you could allow/disallow access from certain IP addresses or ranges
Redirects
Rewrites

You can also prevent modification of code over 3G on some European providers (I’ve experienced UK providers in particular totally mangling site styles).

When setting up WordPress sites, we would typically lock down access to the entire admin area by IP address as a security measure. While this isn’t necessary for Jekyll sites (where there is no login), .htaccess rules can be a useful way of controlling how your site resources are cached.

A proper .htaccess thereby has the potential to speed up site loading times.

##.htaccess in Jekyll By default Jekyll excludes dotfiles - but you can easily override this behaviour:

In config.yml (or config_prod.yml if you have a production environment config file):

include: ['.htaccess']

Jekyll will now build the .htaccess file - which is much more convenient than editing the file on the server.

Add the .htaccess file to the root of your Jekyll project

Sample .htaccess for Cache Control

# Set browser caching
# ------------------------------------------------------------------------------
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType image/jpg "access 1 year"
ExpiresByType image/jpeg "access 1 year"
ExpiresByType image/gif "access 1 year"
ExpiresByType image/png "access 1 year"
ExpiresByType text/css "access 1 month"
ExpiresByType text/html "access 1 month"
ExpiresByType application/pdf "access 1 month"
ExpiresByType text/x-javascript "access 1 month"
ExpiresByType application/x-shockwave-flash "access 1 month"
ExpiresByType image/x-icon "access 1 year"
ExpiresDefault "access 1 month"
</IfModule>
# End caching block

WordPress File & Directory Permissions

Mon, 03 Aug 2015 00:00:00 +0100

Proper permissions are necessary to ensure that a site runs correctly and is secure.

Overly restrictive permissions can prevent important files from being accessed by users and/or server processes.

Permissions that are too permissive constitute a security vulnerability.

Typical WordPress Permissions

For files, 644. For Directories, 755.

If you’re viewing permissions by means of the ls utility on the command line:

Numerical Permissions	rwx format
644	-rw-r–r–
755	drwxr-xr-x

Change Permissions

Set Directory Permissions to 755:

# Set directory permissions using an absolute path:
find /var/www/domain.com/path-to-wp -type d -exec chmod 755 {} \;

# Set directory permissions recursively from the current working directory:
find . -type d -exec chmod 755 {} \;

Set File Permissions to 644:

# Set file permissions using an absolute path:
find /var/www/path-to-wp -type f -exec chmod 644 {} \;

# Set directory permissions recursively from the current working directory:
find . -type f -exec chmod 644 {} \;

wp-config.php

This file contains sensitive information and should be considered separately.

Access should be restricted to the owner and group only. Under Ubuntu/Apache, the following shows rational onwership/permission for wp-config.php:

# Output of ls -la in the document root
-rw-rw---- 1 youruser www-data 1835 Apr 11 10:07 wp-config.php

Bulk Amend wp-config.php

Check ownership & permissions on wp-config.php files, move into the server web root (e.g. /var/www/html) and run:

find . -name 'wp-config.php' -exec ls -la {} \;

To amend all wp-config.php files on a server, move into the server web root (e.g. /var/www/html) and run:

# Set permissions:
sudo find . -name 'wp-config.php' -exec chmod 660 {} \;

# Set ownership:
sudo find . -name 'wp-config.php' -exec chown youruser:www-data {} \;

TL;DR

Directories should have permissions set to 755
Files (except wp-config.php) should have permissions set to 644
Permissions for wp-config.php should be set to 660
wp-config.php should be owned by youruser:serveruser (e.g. sudo chown david:www-data wp-config.php)

Combined command to set file & directory permissions from the current working directory, which should be the WordPress root directory:

find . -type d -exec chmod 755 {} \;; find . -type f -exec chmod 644 {} \;

Set proper permissions/ownership on wp-config.php:

sudo find . -name 'wp-config.php' -exec chmod 660 {} \;
sudo find . -name 'wp-config.php' -exec chown youruser:www-data {} \;

Using git Submodules

Mon, 27 Jul 2015 00:00:00 +0100

Submodules are a way of including third-party git repos in a project. For example, the TCPDF library might be included as a submodule in the repo of a WordPress theme.

Adding a Submodule

Add an existing git repository as a submodule:

git submodule add git@github.com:WebDevStudios/CMB2.git

The submodule is added by default to a directory with the same name as the repo - in this case “CMB2”.

Clone With Submodules

To clone a project with submodules, use the recursive option:

git clone --recursive <url of repo>

Retrospectively Add Submodules

You can also clone submodules as a second step - this may be necessary if you have already cloned the repo:

git clone <repo>
cd <repo>
git submodule init
git submodule update

# To update submodule, add an extra step after running git pull
git pull origin master
git submodule update --recursive

You can create and switch to a new branch, which allows you to work without affecting the production code. Once happy wit changes, the branch can be merged back and deployed.

Deployment

If you’re using this to deploy code onto a remote server, remember that you may need an SSH key pair connection to the remote repo (e.g. GitHub).

Resources

Incremental Backup using rsync with Hardlinks

Sat, 27 Jun 2015 00:00:00 +0100

This method involves an incremental backup bash script that runs on a production server. The aim is to automatically build a local backup which can then be synchronised with a remote backup server.

Production Server: Backup User

Create a backup user - without sudo privileges:

adduser servernamebackup

Production Server: Source Directory

Create a directory called /home/backupuser/source. This will be used as the source for the local rsync, so it forms a target for:

a MySQL backup - mysqldump dumps database copies to /home/backupuser/source/sql
Symlinks to config files (in /home/backupuser/source/config)
Symlinks to site files (from the Apache root directory - /var/www/html)

Set up directories and add symlinks to /source:

sudo mkdir -p /home/backupuser/source/config # make the config directories
sudo mkdir /home/backupuser/source/sql # target for mysqldump
sudo ln -s /var/www/html /home/backupuser/source # Symlink site files
sudo ln -s /etc/apache2 /home/backupuser/source/config # Symlink Apache config files (including vhosts setup)
sudo ln -s /etc/mysql /home/backupuser/source/config # Symlink MySQL config files

Production Server: Incremental Backup

Incremental backup script:

#!/bin/bash
#
# Website Backup Script for SERVER
#
# Add this script as a cronjob to run daily. It creates an archive of incremental
# backups, with the current date set as the name of the backup directory.
# Assumes the script is run from /usr/local/sbin.
# ------------------------------------------------------------------------------

# Todays date in ISO-8601 format:
# ------------------------------------------------------------------------------
DAY0=$(date -I)
TIMESTAMP=$(date '+%Y-%m-%d at %H:%M:%S')

# Yesterdays date in ISO-8601 format:
# ------------------------------------------------------------------------------
DAY1=$(date -I -d "1 day ago")

# The source directory: this should contain a symlink to Apache doc root.
# ------------------------------------------------------------------------------
SRC="/home/servernamebackup/source/"

# The target directory
# ------------------------------------------------------------------------------
TRG="/home/servernamebackup/backup/$DAY0"

# The link destination directory
# ------------------------------------------------------------------------------
LNK="/home/servernamebackup/backup/$DAY1"

# Backup databases
# ------------------------------------------------------------------------------
USER="XXXXXXXXXXXXXXXXXXX"
PASSWORD="XXXXXXXXXXXXXXXXXXXXXXXXXX"
HOST="localhost"
DB_BACKUP_PATH="${SRC}sql"

# Get list of databases, but not 'Database' or 'information_schema'
# ------------------------------------------------------------------------------
DATABASES=$(mysql --user=$USER --password=$PASSWORD -e "SHOW DATABASES;" | grep -Ev "(Database|information_schema)")

DUMPFAIL=false

# Remove previous dumped databases
# ------------------------------------------------------------------------------
rm $DB_BACKUP_PATH/*

# Set up log
# ------------------------------------------------------------------------------
echo "Database backup report. ${TIMESTAMP}" > $DB_BACKUP_PATH/DB_LOG
echo "=======================================" >> $DB_BACKUP_PATH/DB_LOG

# Create dumps for each database
# ------------------------------------------------------------------------------
for DB in $DATABASES
do

  mysqldump -v --user=$USER --password=$PASSWORD --single-transaction --log-error=$DB_BACKUP_PATH/$DB.log --host=$HOST $DB > $DB_BACKUP_PATH/$DB.sql

  # Reportage - log result of mysqldump
  # ----------------------------------------------------------------------------
  if [[ $? -eq 0 ]]

  then

    echo -e "Mysqldump created ${DB}.sql\n" >> $DB_BACKUP_PATH/DB_LOG

  else

    echo "Mysqldump encountered a problem backing up ${DB}. Look in ${DB_BACKUP_PATH}/${DB}.log for information.\n" >> $DB_BACKUP_PATH/DB_LOG
    $DUMPFAIL=true

  fi

done

# The rsync options: follow the symlinks to make a hard backup
# ------------------------------------------------------------------------------
OPT=(-avL --progress --delete --link-dest=$LNK)

# Execute the backup
# ------------------------------------------------------------------------------
rsync "${OPT[@]}" $SRC $TRG

# Log Results
# ------------------------------------------------------------------------------
if [[ $? -gt 0 ]]
then

  # rsync Failure
  # ----------------------------------------------------------------------------
  echo "ERROR. rsync didn't complete the nightly backup: ${TIMESTAMP}" >> /var/log/server-backup.log
  echo "There was an error in the nightly backup for <servername>: ${TIMESTAMP}"| mail -s "Backup Error, <servername>" info@yourdomain.com

else

  # rsync Success
  # ----------------------------------------------------------------------------
  if [[ false == $DUMPFAIL ]]

    # rsync & mysqldump worked OK
    # --------------------------------------------------------------------------
    then

      echo "SUCCESS. Backup made on: ${TIMESTAMP}" >> /var/log/server-backup.log

      # email the report
      # ------------------------------------------------------------------------
      echo -e "${TIMESTAMP}: Server <servername> successfully ran a local backup.\nBoth rsync & mysqldump report success."| mail -s "Backup Success, <servername>" info@yourdomain.com

    # rsync worked but there was at least one mysqldump error
    # --------------------------------------------------------------------------
    else

      echo "PARTIAL SUCCESS. File backup (rsync) was successful, but mysqldump reports errors: ${TIMESTAMP}" >> /var/log/server-backup.log

      # email the report
      # ------------------------------------------------------------------------
      echo -e "${TIMESTAMP}: Server <servername> ran a local backup.\nFile backup reports success, however mysqldump reports at least one problem.\nCheck "| mail -s "Backup: Partial Success, <servername>" info@yourdomain.com

  fi

fi

Add the incremental backup script to /usr/local/sbin
Make executable
Set up cronjob

# Upload script
rsync --progress -a -v -rz -e "ssh -p 1234" ~/bash-projects/remote-backup-scripts/servername/backup-servername daviduser@123.456.789.0:~/

# Move into position
sudo mv ~/backup-servername /usr/local/sbin

# Make executable
sudo chmod u+x /usr/local/sbin/backup-servername

# Open crontable
sudo crontab -e

# Add the following to the crontab, save and exit
# Run backup script every day at 3 am
00 03 * * * /usr/local/sbin/backup-servername
# Send an email report from this cronjob
MAILTO=info@yourdomain.com

–delete

The delete option causes files that are not in source to be deleted from the target - and ONLY the target:

This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory … Files that are excluded from the transfer are also excluded from being deleted unless you use the –delete-excluded option or mark the rules as only matching on the sending side

rsync man-page

I originally used the --delete flag on this script - thinking that it would be necessary to keep the latest incremental backup synchronised with the current source filesystem.

However, --delete has no effect when syncing into an empty directory, which is basically what happens when using the --link-dest method of rsync. The directory specified by --link-dest is used as a reference point, and any files in source that are unchanged relative to this reference point are hardlinked to their exisiting inode in the reference directory.

Files that are not in the source directory (because they have been deleted) are therefore not hardlinked in the backup directory.

However there is a possibility that rsync is backing up to an incomplete directory that is left over from a previous failed run, and may contain things to delete.

--delete is probably superfluous when using --link-dest. The backup directory accurately reflects the source directory. However, we leave it in place in case of backing up to an incomplete directory.

Note: Be careful deleting old snapshots. The old directory in it’s entirety should be properly archived and a new full backup snapshot should be taken to kick off the next round of incremental backups.

This warning is not necessary as using the --link-dest option creates hard links. Thanks to aselinux for pointing this out in the comments.

Result

This will result in a backup directory on the production server, /home/backupuser/backup, that contains dated incremental backups.

The Production backup directory can then be targeted by a Backup server, again via rsync. Typically, this might involve:

Access to the backup directory by means of SSH public/private key pair
rrsync access for the backupuser - only rsync can run on the given SSH key, restricted to read-only

Next Steps

Setup Secure rsync link between servers
Run a cronjob on the backup server to read and rsync the backup directory on the Production server

Resources

Excellent readable article on rsync
The original article by Mike Rubel
Incrementally numbered backups, using rsync
Database backup
Example bash script using rsync for local and remote backups
rsync & cp for hardlinks
Snapshot backup - rsync with hardlinks - with detailed examples
Basic rsync examples
rsync man page - surprisingly helpful!
Time machine/rsync with good illustration of hard links - good sample “instant backup” script - backup every minute!
mysqldump status
Link Dest behaviour explation: answer in the rsync mailing list archive
Link Dest and deleting old snapshots

Secure rsync Between Servers

Wed, 17 Jun 2015 00:00:00 +0100

This method allows for automatic incremental backups between servers.

Backup Server Pulls from Production Server

The files to be synced have been built by a local automatic rsync-based incremental backup script that runs on the Production server. This is because:

There is enough space on the Production machine (approx 8G out of 40G is in use)
Having an incremental backup on the Production server allows for quick restores in the event of non-catastrophic failure
Having backups on the production server is “belt and braces” - backups are stored in multiple locations
The script has root access and can build a complete set of backup files: /var/www/html/, Apache config files, MySQL config files, Database files (built by mysqldump)

The Incremental Backup that is being synced is described here.

Set up a Backup User on the Production Server

Set up a user without sudo privileges. This user is only going to be used to store backups - which provides another level of security.

The backup user home directory will receive incremental backups, and hold the SSH public key to allow the remote backup server to connect - and that’s about it.

Set up an SSH Key Pair on the Backup Server

Because the backup script is designed to be triggered automatically by a cron job, the user that will be running the rsync command on the production server will be root.

It is therefore important to transfer the public SSH key for the root user to the backup target machine.

In addition, the SSH key should NOT have a passphrase, so that connection can be made automatically. We will also limit the actions that can be carried out by this SSH connection.

# Switch to root
su root

# Generate an SSH key pair, WITHOUT a passphrase
ssh-keygen -t rsa

# copy the public key to the backupuser account on the target Production machine
ssh-copy-id -p 1234 backupuser@123.456.789.0

Failure to Copy Public Key

If the public key doesn’t copy across, you’ll see an error message like this:

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Permission denied (publickey).

This means that the remote server has refused the connection.

In my case, this is because I usually set up servers such that access is via SSH key only - with password authentication disabled.

To copy a public SSH key to a remote server, password SSH login must be enabled on the remote server

To enable password login:

SSH into the remote server
Configure SSH: sudo nano /etc/ssh/sshd_config
Edit to allow password authentication: PasswordAuthentication yes
Save & exit, then reload the SSH server: sudo reload ssh

Once the key has been copied across (which will require password entry for the relevant user), test SSH access (as prompted). If it works, you can safely disable password login on the remote machine.

Limit Actions for this SSH Connection to Restricted rsync

Using an SSH key without a passphrase makes it possible to automate remote tasks - a user-entered passphrase is not required for key decryption.

A script called rrsync (which stands for restricted rsync) is provided with rsync specifically to ease the restricting keys to be used only for rsync via .ssh/authorized_keys.

On Ubuntu, the script is located here: /usr/share/doc/rsync/scripts/rrsync.gz. The script needs to be unzipped and installed under user/local/bin:

# Copy the archive rrsync script to /usr/local/bin
sudo cp /usr/share/doc/rsync/scripts/rrsync.gz  /usr/local/bin/

# Unzip the script
sudo gzip -d   /usr/local/bin/rrsync.gz

# Give it correct permissions
sudo chmod 755 /usr/local/bin/rrsync

Once rrsync is in place, we can lock down access for our SSH key by amending the /home/backupuser/.ssh/authorized_keys file.

Open the public key for the backup user on the Production Server. This will constitute a single line of text:

sudo nano /home/backupuser/.ssh/authorized_keys

Prepend the following:

command="/usr/local/bin/rrsync -ro /home/path-to/backup/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding

The command="/usr/local/bin/rrsync ..." restricts access of that particular public key - only the given command can be executed
The ...-ro /home/path-to/backup/" gives read-only access to the specified directory
The no-* options further restrict what actions can be carried out with the public key.
Note that the full path is need to reference rrsync

If you try and SSH into the Production machine from the Backup server you should now see a message like this:

PTY allocation request failed on channel 0
/usr/local/bin/rrsync: Not invoked via sshd
Use 'command="/usr/local/bin/rrsync [-ro] SUBDIR"' in front of lines in /home/backupuser/.ssh/authorized_keys
Connection to 123.45.67.89 closed.

Command rsync from Backup Server

Note: After the above amendment to the authorized_keys file, the final path from the Backup servers point of view is now “/”:

Sample rsync command from the Backup server:

rsync --log-file=$HOME/.rsyncd.log --progress -az -H -e "ssh -p 1234" backupuser@123.456.789.0:/ ~/backup-target-directory

This specifies:

-a: archive mode; equals -rlptgoD (no -H,-A,-X)
-z: Compress file data during the transfer
-H: Maintain hardlinks (important due to our incremental backup)
-e “ssh -p 1234”: connect via SSH on port 1234
See rsync man

Resources

Setup Logwatch

Tue, 16 Jun 2015 00:00:00 +0100

This article refers to Logwatch on Ubuntu 14.04 LTS.

Logwatch needs a mail transfer agent. Typically we use Exim4 in send-only mode.

To install Logwatch, run sudo apt-get update then install Logwatch with:

sudo apt-get install logwatch

Move config files - don’t edit the originals:

sudo cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/

The /etc/logwatch/conf directory is first searched for files with the same name and relative location as the /usr/share/logwatch/default.conf directory. Variables declared in these files override the defaults.

From: http://www.stellarcore.net/logwatch/tabs/docs/HOWTO-Customize-LogWatch.html

If using Apache server:

sudo cp /usr/share/logwatch/default.conf/logfiles/http.conf /etc/logwatch/conf/logfiles/

Then add *combined.log files to the list

Configure Logwatch

Edit config:

sudo nano /etc/logwatch/conf/logwatch.conf

Set email address, logs to be checked/ignored etc.

Cache

This may need to be manualy created, if Logwatch throws an error:

sudo mkdir /var/cache/logwatch

…then run logwatch as sudo.

Set up a Custom Cronjob for Logwatch

Amend /etc/cron.daily/00logwatch to include the admin email:

#!/bin/bash

#Check if removed-but-not-purged
test -x /usr/share/logwatch/scripts/logwatch.pl || exit 0

#execute
#/usr/sbin/logwatch --output mail
/usr/sbin/logwatch --mailto info@yourdomain.com

#Note: It's possible to force the recipient in above command
#Just pass --mailto address@a.com instead of --output mail

The run time is tied in to the time of the daily cron job. To change, remove /etc/cron.daily/00logwatch and add a new cronjob. Not tested:

# Run every day, @ midnight
sudo crontab -e

0 0 * * * /usr/sbin/logwatch

Resources

LAMP Stack Setup

Mon, 15 Jun 2015 00:00:00 +0100

Before starting, follow the guidelines on setting up Ubuntu 14.04 LTS.

Install Apache

sudo apt-get update
sudo apt-get install apache2

Check by visiting the IP address in the browser - you should see the default “Apache 2 Ubuntu” intro page. This shows some useful info on config files - if you’re not familiar with Apache2 config, it might be worth a look.

Install MySQL

We need some databases!

Install MySQL and some helper packages:

sudo apt-get install mysql-server php5-mysql

Open up keepass/KeepassX, set a strong password for MySQL root.

After installation, take extra measures to secure MySQL:

Install MySQL system tables - tell MySQL to create a database directory structure to store info:

sudo mysql_install_db

Run a MySQL security script:

sudo mysql_secure_installation

During execution of this script, you will first get the option to reset root password. You won’t need to do this, because you’ve already set a hardcore password, so answer “n”. Thereafter, hit return to accept the default (Y) answer for each question. This deletes some dummy DBs, disallows remote root login, removes anonymous users and reloads the privilege tables so that changes take effect immediately.

Install PHP

Use apt to install:

sudo apt-get install php5 libapache2-mod-php5 php5-mcrypt

Modify the way that Apache serves files, to give priority to PHP:

sudo nano /etc/apache2/mods-enabled/dir.conf

…bring the php index file forward in the hierarchy, so the directive looks like this:

<IfModule mod_dir.c>
    DirectoryIndex index.php index.html index.cgi index.pl index.xhtml index.htm
</IfModule>

Restart Apache:

sudo service apache2 restart

View available PHP modules:

apt-cache search php5-

If you need additional PHP modules (e.g. php5-curl):

sudo apt-get install php5-curl

Increase the max filesize for uploads, because … clients. You need to edit /etc/php5/apache2/php.ini, since this is the settings file used by Apache:

sudo nano /etc/php5/apache2/php.ini

# hit ctrl+w to open search
# enter upload_max_filesize
# Amend:
; Maximum allowed size for uploaded files.
; http://php.net/upload-max-filesize
upload_max_filesize = 40M

# After saving, restart Apache

Restart Apache:

sudo service apache2 restart

To install additional PHP modules, see the Justin Ellingwood article on Digital Ocean.

Test PHP Setup

Check PHP by outputting phpinfo(); in a php file and accessing via a browser:

sudo nano /var/www/html/info.php

Copy this into the file:

<?php phpinfo(); ?>

Check in a browser(e.g. 123.456.78.90/info.php), save the HTML file locally for your records, then remove:

sudo rm /var/www/html/info.php

PHPMyAdmin

Install - you will need to enter the root MySQL password:

sudo apt-get install phpmyadmin apache2-utils

Hit space then tab to select Apache2
Choose “yes” when asked “Configure database for phpmyadmin with dbconfig-common?”
Enter MySQL password when prompted
Enter password for the phpmyadmin user

Make sure your password manager is open!

Once installed, PHPMyAdmin needs to be added to the Apache config file:

sudo nano /etc/apache2/apache2.conf

Add the following line to the end of apache2.conf:

Include /etc/phpmyadmin/apache.conf

Restart apache:

sudo service apache2 restart

Secure PHPMyAdmin with a .htaccess File

Configure the phpmyadmin directory to allow .htaccess rules:

sudo nano /etc/phpmyadmin/apache.conf

Add AllowOverride All under the the first directory block, straight after “DirectoryIndex…”:

<Directory /usr/share/phpmyadmin>
        Options FollowSymLinks
        DirectoryIndex index.php
        AllowOverride All
        [...]

Set up .htaccess file:

sudo nano /usr/share/phpmyadmin/.htaccess

Enter:

AuthType Basic
AuthName "Restricted Files"
AuthUserFile /etc/apache2/.phpmyadmin.htpasswd
Require valid-user

Create a htpasswd file:

sudo htpasswd -c /etc/apache2/.phpmyadmin.htpasswd username

…where username is the username you want to give access to (doesn’t need to be an existing sytem user).

You’ll be prompted to enter a password for this user (twice). Restart Apache:

sudo service apache2 restart

The PHPMyAdmin directory is now password protected.

Note: If your system doesn’t recognise the htpasswd command - you receive the message htpasswd: command not found, you may need to install the apache-2 utils module:

sudo apt-get install apache2-utils

Once you’ve done this, you can re-try the htpasswd command and it should work. Don’t forget to restart Apache.

Access PHPMyAdmin

Use the MySQL root username & password.

See this Excellent & comprehensive article - most of the current instructions drawn from here.

Set Up Apache Virtual Hosts

Multiple sites on a single server/VPS.

Create Directory Structure

sudo mkdir -p /var/www/html/example.com/public_html
sudo mkdir -p /var/www/html/example2.com/public_html

The -p flag creates intermediate directories as required.

Give permissions:

sudo chown -R $USER:$USER /var/www/example.com/public_html
sudo chown -R $USER:$USER /var/www/test.com/public_html

Set Permissions to 755 for directories:

sudo chmod -R 755 /var/www

Add index.html demo pages if necessary.

Create config files:

Create a config file for each site:

sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/yoursite.com.conf

Use this as a template:

<VirtualHost *:80>
        ServerName yoursite.com
        ServerAlias www.yoursite.com
        ServerAdmin info@carawebs.com
        DocumentRoot /var/www/html/yoursite.com/public_html
        <Directory /var/www/html/yoursite.com/public_html>
                Options -Indexes +FollowSymLinks
                AllowOverride All
                Require all granted
        </Directory>
        ErrorLog ${APACHE_LOG_DIR}/yoursite.com.error.log
        CustomLog ${APACHE_LOG_DIR}/yoursite.com.access.log combined
</VirtualHost>

This config block sets the correct server name, alias and document root. Directory browsing is disallowed, and .htaccess files are allowed.

Site specific error reporting is added - log files are located here: /var/log/apache2/yoursite.com.error.log.

Enable the site using a2ensiteand restart Apache:

sudo a2ensite yoursite.com.conf
sudo service apache2 restart

Enable Apache Rewrites

Enabling the Apache rewrite module will be essential it you’re using pretty permalinks. Enable the module & restart Apache:

sudo a2enmod rewrite && sudo service apache2 restart

sudo a2enmod rewrite && sudo /etc/init.d/apache2 restart

Harden Apache

Prevent the Apache server signature that is printed as part of a web request - this is not needed and gives would-be hackers info about your server.

sudo nano /etc/apache2/conf-enabled/security.conf

# Set ServerSignature Off

Restart Apache for changes to take effect.

Prevent direct access of VCS files by adding this block to /etc/apache2/conf-enabled/security.conf

<DirectoryMatch "/\.git">
   Require all denied
</DirectoryMatch>

You could also prevent access to source control files by adding this to a .htaccess file in the site root:

# ==================================================================
# Prevent .git access
# ==================================================================

RedirectMatch 404 /\.git

Resources

Basic Setup

Guidelines & Tutorials to help set up a secure LAMP stack.

Setup new linode VPS

Post-setup actions:

Security

Good Digital Ocean security guide, including adding new user, setting up public key authentication, SSH configuration. Highly recommended:

LAMP Stack: Including PHPMyAdmin

Comprehensive instructions on how to set up a LAMP stack on a VPS running Ubuntu 14.04:

LAMP Stack, Ubuntu 14.04, Digital Ocean Guidelines Very good!

Set up PHPMyAdmin, including extra security measures (password protected directory with .htaccess):

PHPMyAdmin Setup, Ubuntu 12.04

Create SSL Certificate:

Create SSL Certificate for Apache on Ubuntu

Set Up Mailserver

Exim Cheatsheet:

http://bradthemad.org/tech/notes/exim_cheatsheet.php

Load Testing a Server

Testing with Siege

OS Upgrades/Patches: Ubuntu

The OS is patched very promptly when security issues arise, and it is essential that your server is updated/upgraded regularly. Usually, sudo apt-get update followed by sudo apt-get upgrade is sufficient. The Ubuntu “Message of the Day” generally advises when upgrading is necessary - but it pays to stay up to date with OS developments:

Harden Apache

http://websectips.com/articles/10-apache-security-hardening-tips

Ubuntu 14.04 LTS Server Setup

Sun, 14 Jun 2015 00:00:00 +0100

This article is a cheatsheet/guidelines for the secure setup of Ubuntu 14.04 LTS server.

The content has been largely drawn from this Digital Ocean article.

Once a new server is created, you should follow these setup guidelines as soon as possible. Check the auth.log file for malicious access attempts:

~~~
sudo cat /var/log/auth.log | grep 'sshd.*Invalid'
~~~

Malicious log in attempts begin almost immediately - presumably, would-be hackers are trying a range of IP addresses and knocking on doors, looking for low-hanging fruit. For this reason, the initial VPS setup should definitely include a strong root password.

Set FQDN/Hostname

To set the hostname to “archimedes”, edit /etc/hostname:

echo "archimedes" > /etc/hostname
hostname -F /etc/hostname

If it exists, edit the file /etc/default/dhcpcd to comment out the SET_HOSTNAME directive.

Edit /etc/hosts

Edit /etc/hosts - amend the localhost entry, and add a new line referring to the FQDN.

This should be in the format: IP-Address-of-system hostname.domainname.TLD hostname

127.0.0.1 localhost.localdomain localhost
123.345.567.78 archimedes.your-main-domain.com archimedes

Add ipV6 if necessary, e.g.

1b00:1b00::f0f0:11aa:aa98:bdf4 archimedes.your-main-domain.com archimedes

Restart the hostname service

/etc/init.d/hostname restart

Set Timezone

Activate wizard, follow instructions:

dpkg-reconfigure tzdata

Create New User

Create a new user:

adduser daviduser

Answer the questions, add password, hit enter.

Give new user sudo privileges:

gpasswd -a daviduser sudo

Generate SSH Key Pair on Local Machine

You can re-use and existing one if it already exists

ssh-keygen -t rsa

Add a passphrase.

Copy Public SSH Key to Server

The easiest way to do this:

ssh-copy-id daviduser@123.45.56.78

You will be prompted for the user’s password…and that’s it!

Reboot Server From Command Line

This is sometimes prompted for/required during the setup process:

sudo reboot

Harden SSH Access

Backup config:

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

Restrict root login so it is only via SSH key (NOT Password):

sudo nano /etc/ssh/sshd_config

Amend the PermitRootLogin entry:

PermitRootLogin without-password

Ctrl+o to save, Ctrl+x to exit. Then reload ssh:

sudo reload ssh

Try and connect in a new terminal BEFORE CLOSING THE OPEN TERMINAL!!!

Test logging in to root via SSH - it will allow password entry, but the response shoudl be: “Permission denied, please try again.”

Firewall

Set up ‘Uncomplicated Firewall’: ufw.

Allow on the custom SSH port:

sudo ufw allow 4444/tcp

Allow port 80, for internet:

sudo ufw allow 80/tcp

To allow https traffic, open port 443:

sudo ufw allow 443/tcp

Show the exceptions you have added:

sudo ufw show added

Enable the firewall:

sudo ufw enable

More info on configuring ufw.

Swapfile

Probably not necessary but may prevent server crashing in the event of high traffic. Accessing data stored on disk (rather than memory) is slow.

Generally recommended that the size of the swapfile should be 2 x RAM.

Determine RAM:

# return memory in MB
free -m

Use fallocate to allocate space to swap file:

sudo fallocate -l 4G /swapfile

fallocate is used to manipulate the allocated disk space for a file, either to deallocate or preallocate it. For filesystems which support the fallocate system call, preallocation is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeros. fallocate man page

Give correct permissions to swapfile - restrict access so other users/processors can’t see it:

sudo chmod 600 /swapfile

Format the file for swap:

sudo mkswap /swapfile

Make the file available for swap:

sudo swapon /swapfile

Make it available for swap at boot by amending /etc/fstab:

sudo sh -c 'echo "/swapfile none swap sw 0 0" >> /etc/fstab'

Resources

Set hostname & FQDN in Ubuntu

WordPress Body Classes

Thu, 04 Jun 2015 00:00:00 +0100

Adds extra body classes by hooking into the ‘body_class’ filter.

WP Codex on body_class()

<?php
/**
 * Add page slug to `body_class()` classes if it doesn't exist
 * & add user role to body class
 *
 * Hooks into the `body_class` filter.
 *
 * @package UserHelpers
 * @param  array $classes Passed in from the body_class filter
 * @return array $classes Extra classes added
 *
 */
function carawebs_body_class( $classes ) {

  // Add post/page slug
  // ---------------------------------------------------------------------------
  if ( is_single() || is_page() || is_archive() && !is_front_page() || is_bbpress() ) {

    if (!in_array(basename( get_permalink() ), $classes) ) {

      $classes[] = basename( get_permalink() );

    }

    // Check to see if the user is logged in, and if they are, add their role to the body class.
    // -------------------------------------------------------------------------
    if( is_user_logged_in() && !is_front_page() ){

      $current_user       = wp_get_current_user();  // Current user object
      $current_user_id    = $current_user->ID;      // Current user ID
      $user_role          = carawebs_get_user_role( $current_user_id );
      $user_role          .= '-role'; // append modifier to avoid confusion with pages having same name
      $classes[]          = $user_role;

    }

  }

  return $classes;

}
add_filter('body_class', 'carawebs_body_class');

Select Form Element Styling

Thu, 04 Jun 2015 00:00:00 +0100

Select tags are commonly used in forms:

<select>
  <option value="volvo">Volvo</option>
  <option value="saab">Saab</option>
  <option value="mercedes">Mercedes</option>
  <option value="audi">Audi</option>
</select>

This looks unstyled and brutal:

Impossible to Style

It’s not possible to apply style rules to select/option tags directly, unless I’, missing something.

The bootstrap-select Solution

Using Bootstrap-select, Bootstrap 3 dropdown styles can be added to the option dropdown.

Bootstrap-Select Install

Bootstrap select on Bower.

Install via Bower: bower install bootstrap-select
Enqueue JS files
Use bootstrap-select styles

Using with Sage

Refers to the Sage WordPress Starter Theme.

Amend bower.json (see below) and run bower install
Make the bootstrap-select style rules available - convert from CSS/LESS into SASS
One option: copy (via gulp if necessary) bootstrap-select.css into a new _bootstrap-select.scss file, and import this in assets/styles/main.scss

Amended bower.json will build the bootstrap-select javascript file into main.js by means of Asset Builder.

bower.json

Note that bootstrap-select has been added to the “overrides” block.

{
  "name": "sage",
  "homepage": "https://roots.io/sage/",
  "authors": [
    "Ben Word <ben@benword.com>"
  ],
  "license": "MIT",
  "private": true,
  "dependencies": {
    "modernizr": "2.8.2",
    "jquery": "1.11.2",
    "bootstrap-sass-official": "3.3.4",
    "bootstrap-select": "~1.7.2"
  },
  "overrides": {
    "modernizr": {
      "main": "./modernizr.js"
    },
    "bootstrap-select": {
      "main": "./dist/js/bootstrap-select.js"
    },
    "bootstrap-sass-official": {
      "main": [
        "./assets/stylesheets/_bootstrap.scss",
        "./assets/javascripts/bootstrap/transition.js",
        "./assets/javascripts/bootstrap/alert.js",
        "./assets/javascripts/bootstrap/button.js",
        "./assets/javascripts/bootstrap/carousel.js",
        "./assets/javascripts/bootstrap/collapse.js",
        "./assets/javascripts/bootstrap/dropdown.js",
        "./assets/javascripts/bootstrap/modal.js",
        "./assets/javascripts/bootstrap/tooltip.js",
        "./assets/javascripts/bootstrap/popover.js",
        "./assets/javascripts/bootstrap/scrollspy.js",
        "./assets/javascripts/bootstrap/tab.js",
        "./assets/javascripts/bootstrap/affix.js",
        "./assets/fonts/bootstrap/glyphicons-halflings-regular.eot",
        "./assets/fonts/bootstrap/glyphicons-halflings-regular.svg",
        "./assets/fonts/bootstrap/glyphicons-halflings-regular.ttf",
        "./assets/fonts/bootstrap/glyphicons-halflings-regular.woff",
        "./assets/fonts/bootstrap/glyphicons-halflings-regular.woff2"
      ]
    }
  }
}

Using the library

The easy bit - just create your <select> with the .selectpicker class.

For more options, see the docs.

Set Up SPF Records

Wed, 03 Jun 2015 00:00:00 +0100

Publishing a Sender Policy Framework (SPF) Record in your domain’s DNS specifies which server IP addresses are allowed to send emails from your domain.

Having a properly set up SPF record makes it less likely that outgoing mail will be tagged as spam.

Return as a TXT Record

Although it appears that best practice is to publish SPF records via DNS as both a SPF and TXT record, the DNS management utilities for some hosts only provide for the addition of txt records

For a good description of SPF as TXT records, see here.

SPF Generators

From the MS Generator: v=spf1 a mx ip4:123.45.678.90 mx:mail.yourdomain.com -all

Add text record with -all amended to ~all, which allows a ‘soft fail’:

v=spf1 a mx ip4:123.45.678.90 mx:mail.yourdomain.com ~all

This assumes that there is also an mx record for a mailserver for the domain in question.

Check SPF Record

Use this tool to check SPF records on your domain.

Resources

WordPress Class Autoloader

Wed, 29 Apr 2015 00:00:00 +0100

Assuming that classes have been saved in individual files, according to WordPress best practice guidelines: The file /classes/my-unique-class.php contains My_Unique_Class().

function my_class_autoloader( $classname ) {

  // Build a filename that matches the correct protocol
  ------------------------------------------------------------------------------
  $classfile = sprintf( '%s/classes/class-%s.php',
  get_template_directory(),
  str_replace( '_', '-', strtolower( $classname ) )
  );

  if ( file_exists( $classfile ) ) include_once( $classfile );

}

// This function "creates a queue of autoload functions, and runs through each of them in the order they are defined."
// -----------------------------------------------------------------------------
spl_autoload_register( 'my_class_autoloader' );

An Array of Post IDs from WP_Query

Tue, 21 Apr 2015 00:00:00 +0100

Return an array of Post IDs from a WordPress WP_Query object - which can then be used to set up data from the _postmeta table.

By having the query only return IDs, it is quicker and more efficient - but I constantly forget how to set it up.

Note: There’s no loop to set up, you get the array from the object’s posts property: $query_object->posts.

WP_Query

/**
 * Return an array of post IDs
 *
 * Build a more efficient query by only returning an array of post IDs.
 *
 * @author David Egan
 * @return array Array of post IDs
 * @see http://codex.wordpress.org/Class_Reference/WP_Query#Return_Fields_Parameter
 */
function carawebs_student_questionnaire_results(){

  $args = array(
    'post_type'     => 'questionnaire-result',
    'post_status'   => 'publish',
    'fields'        => 'ids',
    'meta_query'    => array(
      array(
        'key'        => 'user_type',
        'value'      => 'student',
        'compare'    => '=',
        'type'       => 'CHAR',
      ),
    ),
  );

  // The Query
  $result_query = new WP_Query( $args );

  $ID_array = $result_query->posts;

  // Restore original Post Data
  wp_reset_postdata();

  return $ID_array;

}

Resources

Incremental Timestamped Local Backup

Wed, 08 Apr 2015 00:00:00 +0100

This script creates a backup at regular intervals - you can backup every minute if necessary.

The backup is incremental, so very efficient in terms of space and system resources.

#!/bin/sh
#

date=$(date "+%Y-%m-%d-%H:%M:%S")
rsync -aP --link-dest=$HOME/Backups/current /home/david/Documents $HOME/Backups/back-$date
rm -f $HOME/Backups/current
ln -s back-$date $HOME/Backups/current

# After running:

david@david-desktop:~/Backups$ ls -la
total 28
drwxrwxr-x   5 david david  4096 Jun 26 17:39 .
drwxr-xr-x 149 david david 12288 Jun 27 09:56 ..
drwxr-xr-x   3 root  root   4096 Jun 26 17:39 back-2015-06-26-17_39_06
drwxr-xr-x   3 root  root   4096 Jun 26 17:39 back-2015-06-26-17_39_25
drwxr-xr-x   3 root  root   4096 Jun 26 17:39 back-2015-06-26-17_39_36
lrwxrwxrwx   1 root  root     24 Jun 26 17:39 current -> back-2015-06-26-17_39_36

Resources

This script is based on this article.

Node, NVM & NPM on Ubuntu 14.04

Wed, 04 Mar 2015 09:38:44 +0000

Guidelines on installing Node.js and npm using Node Version Manager(NVM) in Ubuntu 14.04.

This method will avoid EACCES errors when trying to install npm or npm packages. It will allow the use of npm install without sudo privileges on project directories.

What is Node.js?

Node.js provides a way of running JavaScript on the server-side.

What is NPM?

The package manager for Node.

An online repository for the publishing of open-source Node.js projects
A command-line utility for interacting with this repository that aids in package installation, version management, and dependency management.

Using npm, node packages can be installed from the repository using a single line of command. e.g.

npm install async

Another use for npm is dependency management. When you have a node project with a package.json file, you can run npm install from the project root and npm will install all the dependencies listed in the package.json.

Version Management: Installing NVM:

Before following the steps below to install node/npm, you should first make sure that there are no versions of node/npm already installed on your system. This can be done with the following lines of code:

which node
which npm

If there is no output after either of the above, then that respective package is not present on your system.

Once this is complete, enter:

sudo apt-get update
sudo apt-get install build-essential libssl-dev

Once these prerequisite packages are installed you can install NVM with with the command:

curl https://raw.githubusercontent.com/creationix/nvm/v0.16.1/install.sh | sh

You may now have to restart your terminal. Once you’ve done this, run the following command:

source ~/.profile

This allows the current BASH profile to be reloaded, without logging out and back in again.

Installing Node:

You can then view which versions of Node.js are available for installation with the command:

nvm ls-remote

#You can install the relevant version by entering:

nvm install 0.11.14

NPM should be installed by default along with Node.js. You should now be able to install packages without being required to use the sudo command.

You can set a default version of Node to be used by the setting the alias “default”. This is important, otherwise node isn’t recognised in a new shell:

nvm alias default 0.11.14

This allows you to revert to a different version of Node if the current version is incompatible with an particular application. Each version of Node will keep track of its own packages. You can also install a package globally, which will make it available to all projects using the same version of Node, by adding the -g tag. e.g.

npm install -g grunt

Installing globally will let you run the commands from the command line, but you’ll have to link the package into your local sphere to require it from within a program. eg.

npm link grunt

Resources

Update WordPress User Meta

Sun, 08 Feb 2015 00:00:00 +0000

#Build an Array The objective is to create a two-dimensional array that holds information about user progress - in this case, when a user finishes a workbook, information is stored in a ‘completed_workbooks’ field.

Update User Meta Function

/**
*
* Update usermeta when a student completes a workbook.
*
* @param int $student_ID Student user ID
* @param int $workbook_ID Workbook(CPT) post ID
* @param int $stage Student's current stage
* @param int $workbook_total_stages Number of stages
* @param string $supervisor_name
* @param string $coord_name
* @param string $company_name
*
*
*/
function carawebs_finished_workbook (
  $student_ID,
  $workbook_ID,
  $stage,
  $workbook_total_stages,
  $supervisor_name,
  $coord_name,
  $company_name
  ){

  if ( $stage === $workbook_total_stages + 1 ) {

    // The student has completed the current workbook
    // -------------------------------------------------------------------------
    $date = date("F j, Y, g:i a");

    $existing_completed_meta = get_user_meta( $student_ID, 'completed_workbooks', true );

    if ( $existing_completed_meta ){

      // Previous workbooks have been completed
      // -----------------------------------------------------------------------

      $new_data = array (
          'workbook_ID'       => $workbook_ID,
          'supervisor_name'   => $supervisor_name,
          'coordinator_name'  => $coord_name,
          'company_name'      => $company_name,
          'date_completed'    => $date
      );

      $existing_completed_meta["workbook_ID_$workbook_ID"] = $new_data;

      update_user_meta( $student_ID, 'completed_workbooks', $existing_completed_meta );


    } else {

      // First workbook completed - add 'completed_workbooks' array
      // -----------------------------------------------------------------------

      $completed_meta = array(
        "workbook_ID_$workbook_ID"  => array (
          'workbook_ID'       => $workbook_ID,
          'supervisor_name'   => $supervisor_name,
          'coordinator_name'  => $coord_name,
          'company_name'      => $company_name,
          'date_completed'    => $date
        )
      );

      update_user_meta( $student_ID, 'completed_workbooks', $completed_meta );

    }

  }

}

The Resulting Array

The resulting array looks like this:

array(2) {
  ["workbook_ID_14"]=>
  array(6) {
    ["workbook_ID"]=>
    string(2) "14"
    ["supervisor_name"]=>
    string(14) "Clark Griswold"
    ["coordinator_name"]=>
    string(10) "David Egan"
    ["company_name"]=>
    string(8) "ACME Ltd"
    ["date_completed"]=>
    string(25) "February 8, 2015, 3:50 pm"
  }
  ["workbook_ID_13"]=>
  array(6) {
    ["workbook_ID"]=>
    string(2) "13"
    ["supervisor_name"]=>
    string(14) "Clark Griswold"
    ["coordinator_name"]=>
    string(10) "David Egan"
    ["company_name"]=>
    string(8) "ACME Ltd"
    ["date_completed"]=>
    string(25) "February 8, 2015, 3:51 pm"
  }
}

Retrieve Data

To retrieve the data:

$completed_workbooks = get_user_meta( $student_ID, 'completed_workbooks', true );

// Access data for each workbook:

  $supervisor_name  = $completed_workbooks["workbook_ID_$workbook_ID"]['supervisor_name'];

SSH Notes

Wed, 31 Dec 2014 00:00:00 +0000

Paired SSH keys provide a secure way of linking two computers - e.g. a remote server and a local development machine.

Local

You need a locally generated public-private key pair. If you already have this on your local machine, skip this step - you can use the existing public key.

ssh-keygen -t rsa

For normal usage, you should add a passphrase to the SSH key. If the machine is a laptop, consider a strong passphrase, held in an encrypted password manager like keepass.

For automated processes (like backup scripts that need SSH connection), you can’t use a passphrase. You can however restrict the access for this kind of key - for example, you can restrict the commands that can be run on the connection by editing the ~/.ssh/authorized_keys file on the remote machine.

Copy the public key to the server. If using a service like GitHub or Bitbucket, you can paste the contents of the key in the appropriate window.

ssh-copy-id

To copy to a self-managed server, you can use ssh-copy-id:

ssh-copy-id is a script that uses ssh to log into a remote machine (presumably using a login password, so password authentication should be enabled, unless you’ve done some clever use of multiple identities)

It also changes the permissions of the remote user’s home, ~/.ssh, and ~/.ssh/authorized_keys to remove group writability (which would otherwise prevent you from logging in, if the remote sshd has StrictModes set in its configuration).

http://manpages.ubuntu.com/manpages/lucid/man1/ssh-copy-id.1.html

ssh-copy-id daviduser@123.45.56.78

If you’re accessing SSH over a non-standard port:

ssh-copy-id -p 1234 daviduser@123.45.56.78

Locked Down Password Connection

Passing the public key to the remote server requires a one-time password authenticated connection. If you have password authentication turned off, you’ll need to temporarily enable it to allow the key transfer. Once you’ve done this, check the connection works and then disable password authentication again.

Turning off password authentication forces all connections to be made via SSH key pairs. This is very secure, since the SSH key is to all intents and purposes unbreakable - unless someone gets hold of the machine holding your private key.

To temporarily enable password authentication:

# Open the SSH config file for editing
sudo nano /etc/ssh/sshd_config

# Amend password Authentication
PasswordAuthentication yes

Reload SSH:

sudo reload ssh

Change the setting back when you’re finished and don’t forget to reload SSH.

Emergency

If you lose a machine that is connected via SSH key-pair to a server, remove the relevant key in the ~/.ssh/authorized_keys file of the remote server.

Be prepared: keep a note of all servers connected via SSH key-pair, and keep a note of the local machine hostname to allow easy searching in the authorized_keys file in the case of multiple entries. e.g.:

Main Laptop: hostname: David-Gazelle
SSH Keys Connected:

Personal Github account
Company Bitbucket account
Server, plato
Server, archimedes

Set Up restricted SFTP Access Using OpenSSH

Wed, 31 Dec 2014 00:00:00 +0000

#Set Up restricted SFTP Access You can use OpenSSH to allow users to have restricted SFTP access to your filesystem.

This guide is for OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 running on Ubuntu Server 14.04.1 LTS.

Set Up a New User

# Create a new user
useradd guest-user

# Set a password
passwd guest-user

# Set the home directory for the new user to the target site directory
usermod -d /var/www/yoursite.com/public_html/subsite guest-user

Block SSH Access

Prevent the user from accessing the server’s shell - so they can’t access by SSH:

sudo usermod -s /bin/false guest-user

Configure SSH

First make a backup of sshd_config. Then edit the SSH config file:

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config_bak
sudo nano /etc/ssh/sshd_config

First set the sftp subsystem:

#Subsystem sftp /usr/lib/openssh/sftp-server
Subsystem sftp internal-sftp

Then append this stanza to the end of the file:

Match user guest-user
ChrootDirectory /var/www/yoursite.com/public_html/subsite
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Note that I had trouble making this work by matching Groups - which would obviously be a more efficient way of setting this up.

Ctrl + o to save, Ctrl + x to exit.

Restart SSH - use THIS:

sudo service ssh restart

DON’T use this:

sudo /etc/init.d/ssh restart

If you try to connect via SSH for the new user, you’ll see this:

david@david-desktop:~$ ssh -p 1234 guest-user@123.45.67.890
guest-user@123.45.67.890's password:
Could not chdir to home directory /var/www/yoursite.com/public_html/subsite: No such file or directory
This service allows sftp connections only.
Connection to 123.45.67.890 closed.

Remember to set the correct port for SSH access when accessing via SFTP.

External Resources

SFTP Chapter, Open SSH Cookbook - an excellent free online book that outlines how to configure SSH.
Ubuntu Guide to OpenSSH Server
http://serverfault.com/questions/392601/how-to-add-user-with-sftp-ftp-access-to-var-www-html-website-abc-folder-on-a
http://www.fullybaked.co.uk/articles/chroot-ssh-ftp-users-to-home-directory

Responsive Images in IE 8

Thu, 06 Nov 2014 00:00:00 +0000

Images can be made responsive, while retaining aspect ratio with:


.img-responsive {
  display: block;
  max-width: 100%;
  height: auto;
}

However IE8 doesn’t like max-width: 100%.

One way to deal with this is to put IE styles into a @media block, so that they are only applied to certain versions of IE.

This is hacky - most browsers don’t parse the code as they recognise the \ and characters.

This code block is only applied by IE 6, 7 & 8:

/* Add a different img width for IE 8 and lower */
@media \0screen\,screen\9 {
    img-responsive {
        width: auto;
    }
}

This doesn’t work in LESS, but we can hack the hack by declaring a LESS variable and using that:

@iehack: \0screen\,screen\9;

@media @iehack {
    .img-responsive {
        width: auto;
    }
}

It may be a bit hacky, but it gets the job done. Hopefully it won’t be needed for much longer as these old browsers drop off the face of the internet.

Resources

Quick Build WordPress & Roots

Sun, 26 Oct 2014 09:38:44 +0000

This BASH script was written to automate the deployment of WordPress & Roots on a local development server (Ubuntu 14.04). It could easily be adapted for use in a public-facing server.

Clone the project on GitHub - pull requests welcome.

Save the script as /usr/local/bin/roots-spin and make executable:

sudo chmod +x /usr/local/bin/roots-spin

#!/bin/bash
# This script:
# - Creates a new directory for a WordPress site in /var/www.
# - Creates a new WordPress test site, with a fresh database.
# - Downloads, renames and activates the Roots theme.
# - Sets up a new GitHub repo for the new theme (GitHub personal token needs to be in place)
# - Downloads & activates the "Soil" plugin.
# - Installs node & bower dependencies by running npm install.
# - Requires WP-CLI: http://wp-cli.org/
# - Ubuntu: save this script as /usr/local/bin/roots-spin
# - Make it executable, then you can invoke it by typing "roots-spin" in the terminal.
# NOTE: depending on your node setup, you may need to amend the setup_node () function in this script (see inline comments).
# ------------------------------------------------------------------------------
# Set up directory, download wp core

echo -e "Please enter the name of the new wp directory:"
read  wp_dir
cd /var/www
sudo mkdir $wp_dir
sudo chown -R $USER /var/www/$wp_dir
cd /var/www/$wp_dir
wp core download
echo "The new directory has been created at /var/www/$wp_dir"
echo "It is owned by \"$USER\""
echo "WordPress core has been downloaded"

# ------------------------------------------------------------------------------
# Build new database

echo -e "This script will create a local database with a new user. Username & password based on DB name, but this is OK because we're working LOCALLY."
echo -e "Enter the name of the new database:"
read  db_name

# Set up variables
# Username is just the "salted" db name. For publicly accessible databases, make this more secure.
newuser=$db_name
newuser+=_user
# Password is just the "salted" db name. Adjust this for better passwords - require user input.
newpassword=$db_name
newpassword+=_pass1234

echo -e "Enter the MySQL root password:"
read  rootpassword

db="create database $db_name;GRANT ALL PRIVILEGES ON $db_name.* TO $newuser@localhost IDENTIFIED BY '$newpassword';FLUSH PRIVILEGES;"
mysql -u root -p$rootpassword -e "$db"

if [ $? != "0" ]; then  # Note: "0" is the return from successful completion of the previous operation in BASH.
 echo "[Error]: Database creation failed"
 exit 1
else

# ---------------------------
# Set up wp-config.php. These are WP-CLI commands.
# ---------------------------

wp core config --dbname=$db_name --dbuser=$newuser --dbpass=$newpassword --dbhost=localhost

# ---------------------------
# Install WordPress. WP-CLI.
# ---------------------------

echo -e "Enter a strong password - your database may be exported to a publicly accessible server, so please set a strong password."
read strong_password

echo -e "Enter a username. NOT 'ADMIN'!"
read username

wp core install --url=http://localhost/$wp_dir --title="Basic Test Site" --admin_user=$username --admin_password=$strong_password --admin_email=you@yourweb.com


 echo "------------------------------------------"
 echo " WordPress has been installed successfully: http://localhost/$wp_dir "
 echo "------------------------------------------"
 echo " DB Info: "
 echo ""
 echo " DB Name: $dbname"
 echo " DB User: $newuser"
 echo " DB Pass: $newpassword"
 echo "------------------------------------------"
 echo " WordPress User details:"
 echo ""
 echo " WP username: $username"
 echo " WP password: $strong_password"
fi

# ---------------------------------
# Set Up Roots Theme

setup_theme () {
cd /var/www/$wp_dir/wp-content/themes
git clone https://github.com/roots/roots.git
echo -e "Choose a name for the new theme:"
read themename
cp -r /var/www/$wp_dir/wp-content/themes/roots /var/www/$wp_dir/wp-content/themes/$themename

# Change first line of style.css to avoid confusion
themeline="Theme Name: "
themeline+=$themename

awk -v stylename="$themeline" '{ if (NR == 1) print stylename; else print $0}' /var/www/$wp_dir/wp-content/themes/$themename/style.css > /var/www/$wp_dir/wp-content/themes/$themename/stylemod.css
sudo cp /var/www/$wp_dir/wp-content/themes/$themename/stylemod.css /var/www/$wp_dir/wp-content/themes/$themename/style.css
rm -rf /var/www/$wp_dir/wp-content/themes/$themename/stylemod.css

echo -e "-------------------------------------------------------------------------"
echo -e "The first line of style.css has been amended"
echo -e "to reflect your theme name - don\'t forget to amend the rest of this file."
echo -e "NB: this script has appended a line to wp-config.php to set up the Roots development environment."
echo -e "-------------------------------------------------------------------------"

# Backup original config file
cp /var/www/$wp_dir/wp-config.php /var/www/$wp_dir/wp-config.bak.php

newline="/* Define the Roots Environment */\ndefine('WP_ENV', 'development');\n"
awk -v insert="$newline" 'NR==3{print insert}1' /var/www/$wp_dir/wp-config.php > /var/www/$wp_dir/wp-config-temp.php

cp /var/www/$wp_dir/wp-config-temp.php /var/www/$wp_dir/wp-config.php

}

install_soil () {
cd /var/www/$wp_dir/wp-content/plugins
git clone https://github.com/roots/soil.git
wp plugin activate soil
}

setup_node () {
cd /var/www/$wp_dir/wp-content/themes/$themename

# Note: npm runs without sudo if node has been installed on your system by the nvm method:
# https://github.com/creationix/nvm (recommended!)
# Otherwise, the npm install command should run as sudo, followed by bower install.
# For this case comment out line 133 & enable lines 134 & 135)
# Or move to the theme directory after installation is complete and run sudo npm install followed by bower install
npm install
# sudo npm install
# bower install
}

run_grunt () {
cd /var/www/$wp_dir/wp-content/themes/$themename
grunt build
}

activate_theme () {
cd /var/www/$wp_dir/wp-content/themes/$themename
wp theme activate $themename
}

setup_repo () {
cd /var/www/$wp_dir/wp-content/themes/$themename

# remove existing git repo
sudo rm -rf /var/www/$wp_dir/wp-content/themes/$themename/.git
git init

repo_name=$1

  dir_name=`basename $(pwd)`
  invalid_credentials=0

  if [ "$repo_name" = "" ]; then
    echo "  Repo name (hit enter to use '$dir_name')?"
    read repo_name
  fi

  if [ "$repo_name" = "" ]; then
    repo_name=$dir_name
  fi

  username=`git config github.user`
  if [ "$username" = "" ]; then
    echo "  Could not find username, run 'git config --global github.user <username>'"
    invalid_credentials=1
  fi

  token=`git config github.token`
  if [ "$token" = "" ]; then
    echo "  Could not find token, run 'git config --global github.token <token>'"
    invalid_credentials=1
  fi

  if [ "$invalid_credentials" -eq "1" ]; then
    return 1
  fi

  echo -n "  Creating Github repository '$repo_name' ..."
  curl -u "$username:$token" https://api.github.com/user/repos -d '{"name":"'$repo_name'"}' > /dev/null 2>&1
  echo " done."

  echo -n "  Pushing local code to remote ..."
  git remote add origin git@github.com:$username/$repo_name.git > /dev/null 2>&1
  git push -u origin master > /dev/null 2>&1
  git add -A
  git commit -m "First Commit"
  git push origin master
  echo " done."

}

	read -p "Do you want to set up the Roots 7 Starter theme? [y/N]" yn
	case $yn in
	[Yy]* ) setup_theme;;
	[Nn]* ) exit ;;
	* ) echo "Please answer yes or no.";;
	esac

	read -p "Set up a new GitHub repo for $themename? [y/N]" yn
	case $yn in
	[Yy]* ) setup_repo;;
	[Nn]* ) ;;
	* ) echo "Please answer yes or no.";;
	esac

	read -p "Do you want to install and activate the Roots \"Soil\" plugin? [y/N]" yn
	case $yn in
	[Yy]* ) install_soil;;
	[Nn]* ) exit ;;
	* ) echo "Please answer yes or no.";;
	esac

	read -p "Do you want to go to the $themename theme and install node dependencies? [y/N]" yn
	case $yn in
	[Yy]* ) setup_node;;
	[Nn]* ) exit ;;
	* ) echo "Please answer yes or no.";;
	esac

	read -p "Do you want to activate the $themename theme? [y/N]" yn
	case $yn in
	[Yy]* ) activate_theme;;
	[Nn]* ) exit ;;
	* ) echo "Please answer yes or no.";;
	esac

	read -p "Move to the theme directory and run the grunt build task? [y/N]" yn
	case $yn in
	[Yy]* ) run_grunt;;
	[Nn]* ) exit ;;
	* ) echo "Please answer yes or no.";;
	esac

sitelocation="Access your new site: http://localhost/"
sitelocation+=$wp_dir
echo -e "------------------------------------------------"
echo -e "Finished!"
echo $sitelocation
echo -e "------------------------------------------------"

Testing Internet Explorer in Ubuntu Using Virtual Machines

Sat, 25 Oct 2014 09:38:44 +0000

Cross-browser testing of Internet Explorer in a Ubuntu 14.04 development environment - using VirtualBox and Microsoft VM.

VirtualBox

VirtualBox is a system emulator that can run a guest operating system in a window of the host operating system.

This allows us to test websites on a range of platforms without dual-booting or running a dedicated test-machine.

There are two versions of Virtualbox available for Ubuntu. Both open-source, one is packaged by Ubuntu, the other provided by VirtualBox. Either download Virtualbox for Linux and build it yourself, or access Virtualbox in the Ubuntu software manager:

Download VirtualBox for Linux hosts
Search for “virtualbox” in Ubuntu software centre & install as usual

Build a Windows VM

Go to modern.ie and create the build you need. You can select an OS and a version of Internet Explorer.

Select “VirtualBox for Linux” as the Platform, and select “Download your Virtual Machine”.

Click “Batch File Download” - this opens a txt file with a list of URLS. The URLS are the files required to build the Windows VM.

Create a new directory for the VM, and move into the directory:

cd
mkdir win7IE11
cd win7IE11

Run wget -i against the URL of the batch txt file to download all the necessary files:

wget -i https://az412801.vo.msecnd.net/vhd/VMBuild_20131127/VirtualBox/IE11_Win7/Linux/IE11.Win7.ForLinuxVirtualBox.txt

Go and have a cup of tea, cos it’s gonna take a while…

Make the file executable… sudo chmod +x IE11.Win7.ForLinuxVirtualBox.part1.sfx

To extract the files, you may need to install the following: apt-get install lib32stdc++6 See: http://stackoverflow.com/questions/11471722/libstdc-so-6-cannot-open-shared-object-file-no-such-file-or-directory.

Run the .sfx file to build the VM:

~/win7IE11/IE11.Win7.ForLinuxVirtualBox.part1.sfx

The VM files should be extracted to a .ova file.

Import & Use the VM

Open VirtualBox. Click “Import Appliance” in the menu bar, and navigate to the .ova file.

VirtualBox will build the VM. To use it, open VirtualBox and launch the relevant VM.

Localhost

The local server can be accessed on the VM at http://10.0.2.2/. However, my out-of-the box settings give me access to localhost files, but built in references to stylesheets etc are broken - since “localhost” is referenced.

This is fixed by amending C:\windows\system32\drivers\etc\hosts - just add the line 10.0.2.2 localhost. To open this as a Windows admin, open notepad as administrator (by right clicking) and then navigate to the hosts file.

Useful Resources

https://help.ubuntu.com/community/VirtualBox
VirtualBox installation in Ubuntu
VirtualBox home page
Windows/Internet Explorer VM Downloads
This article focuses on using VirtualBox/Windows VMs for testing on Macs - but is still relevant for Ubuntu users.

Bootstrap Carousel with Indicators Navigation for WordPress

Wed, 22 Oct 2014 09:38:44 +0000

A Bootstrap carousel with dual indicators/controls for WordPress. Populated by ACF “relationship” field.

Indicators

This carousel has two sets of indicators - a set of numbers, and an un-numbered “standard” set of indicators.

It needs a bit of work to have the active class attached properly to the second set of indicators.

This is achieved by:

Giving a unique class name to these indicators (e.g. class="no-numbers")
Testing the contained list items data-slide-to= against the currently displayed item

Building the Carousels - ACF

The carousel content is built from an ACF “relationship” field, which is used to create a custom WordPress loop. In this case, the relationship field is used to create a loop of “report” custom post types.

Build the Carousel

This PHP function builds the carousel. The code could be added to the theme custom.php file. I usually keep custom loops in a unique file for ease of editing, and require_once this file in functions.php.

/*==============================================================================
  Case Studies for Services Mobile
  ============================================================================*/
function carawebs_reports_mobile() {

    $relationship_field = 'related'; // This could be passed into the function

    $posts = get_field($relationship_field);

    if ($posts) {
      ?>
      <div id="mobile-reports-carousel" class="carousel slide" data-ride="carousel">
      <!-- Indicators -->
        <ol class="carousel-indicators numbered list-unstyled list-inline">
          <?php

          $item_count = 0; // Start up a count
          foreach ( $posts as $post) { // Loop through the array

            ?>
            <li data-target="#mobile-reports-carousel" data-slide-to="<?php echo $item_count; ?>"><?php echo $item_count + 1; ?></li>
            <?php
            $item_count++;

          } ?>
        </ol>
        <div id="carousel-count"></div><!-- output carousel index & total number of slides for debugging, set by jQuery function -->
        <ol class="carousel-indicators no-numbers">
          <?php

          $item_count = 0;
          foreach ( $posts as $post) {

            ?>
            <li data-target="#mobile-reports-carousel" data-slide-to="<?php echo $item_count; ?>"></li><?php
            $item_count++;

          } ?>
      </ol>
      <div class="carousel-inner">
        <?php
        $number = 0;
        foreach ( $posts as $post) { // variable must be called $post (IMPORTANT)

        global $post;
        setup_postdata($post);

        ?>
        <div class="item counter-<?php echo $number; ?>">
          <?php
          // custom function to bring in the 'medium' featured image
          carawebs_featured_image ('medium'); ?>
          <h4><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h4>
          <div class="post_content"><?php the_content(); ?></div>
          <?php
          // Another custom field - builds a ul of file downloads
          carawebs_download_link_repeater(); ?>
        </div>

      <?php
      $number++;
      } ?>
      </div>
    </div>
    <?php

    // Restore original Post Data
    wp_reset_postdata();

    }
}

This carousel needed to replace a more elaborate slider at small screen sizes.

<section>
    <div class="row report-downloads">
      <h2 class="text-center">Report Downloads</h2>

      <?php

        // This function returns 'true' for mobile phones
        if (carawebs_is_phone()=='true'){

          // If this is a mobile phone
          // --------------------------
          // #innovations--section-toggle & #innovation-content set up for collapse in /assets/js/_frontpage-toggles.js
          ?><div id="reports-mobile" class="row">
            <?php carawebs_reports_mobile(); ?>
          </div><?php

        } else {

          // The device is NOT a phone
          // Hide at small screens using CSS - target #reports-desktop
          // --------------------------
          ?>
          <div id="reports-desktop">

              <?php carawebs_reports_desktop(); ?>

          </div>
          <?php

          // Add the mobile view stuff here to allow a mobile-type view on contracted desktops
          // ---------------------------
          ?>
          <div id="reports-mobile-visible">
            <div class="col-xs-14 col-xs-offset-1">
              <?php carawebs_reports_mobile(); ?>
            </div>

          </div><?php

        }

      ?>

    </div>
</section><!-- /.report-downloads -->

The “check for phone” function:

/*==============================================================================
  Detect Phones
  ============================================================================*/
function carawebs_is_phone() {

	$useragent=$_SERVER['HTTP_USER_AGENT'];
	if(preg_match('/(android|bb\d+|meego).+mobile|avantgo|bada\/|blackberry|blazer|compal|elaine|fennec|hiptop|iemobile|ip(hone|od)|iris|kindle|lge |maemo|midp|mmp|netfront|opera m(ob|in)i|palm( os)?|phone|p(ixi|re)\/|plucker|pocket|psp|series(4|6)0|symbian|treo|up\.(browser|link)|vodafone|wap|windows (ce|phone)|xda|xiino/i',$useragent)||preg_match('/1207|6310|6590|3gso|4thp|50[1-6]i|770s|802s|a wa|abac|ac(er|oo|s\-)|ai(ko|rn)|al(av|ca|co)|amoi|an(ex|ny|yw)|aptu|ar(ch|go)|as(te|us)|attw|au(di|\-m|r |s )|avan|be(ck|ll|nq)|bi(lb|rd)|bl(ac|az)|br(e|v)w|bumb|bw\-(n|u)|c55\/|capi|ccwa|cdm\-|cell|chtm|cldc|cmd\-|co(mp|nd)|craw|da(it|ll|ng)|dbte|dc\-s|devi|dica|dmob|do(c|p)o|ds(12|\-d)|el(49|ai)|em(l2|ul)|er(ic|k0)|esl8|ez([4-7]0|os|wa|ze)|fetc|fly(\-|_)|g1 u|g560|gene|gf\-5|g\-mo|go(\.w|od)|gr(ad|un)|haie|hcit|hd\-(m|p|t)|hei\-|hi(pt|ta)|hp( i|ip)|hs\-c|ht(c(\-| |_|a|g|p|s|t)|tp)|hu(aw|tc)|i\-(20|go|ma)|i230|iac( |\-|\/)|ibro|idea|ig01|ikom|im1k|inno|ipaq|iris|ja(t|v)a|jbro|jemu|jigs|kddi|keji|kgt( |\/)|klon|kpt |kwc\-|kyo(c|k)|le(no|xi)|lg( g|\/(k|l|u)|50|54|\-[a-w])|libw|lynx|m1\-w|m3ga|m50\/|ma(te|ui|xo)|mc(01|21|ca)|m\-cr|me(rc|ri)|mi(o8|oa|ts)|mmef|mo(01|02|bi|de|do|t(\-| |o|v)|zz)|mt(50|p1|v )|mwbp|mywa|n10[0-2]|n20[2-3]|n30(0|2)|n50(0|2|5)|n7(0(0|1)|10)|ne((c|m)\-|on|tf|wf|wg|wt)|nok(6|i)|nzph|o2im|op(ti|wv)|oran|owg1|p800|pan(a|d|t)|pdxg|pg(13|\-([1-8]|c))|phil|pire|pl(ay|uc)|pn\-2|po(ck|rt|se)|prox|psio|pt\-g|qa\-a|qc(07|12|21|32|60|\-[2-7]|i\-)|qtek|r380|r600|raks|rim9|ro(ve|zo)|s55\/|sa(ge|ma|mm|ms|ny|va)|sc(01|h\-|oo|p\-)|sdk\/|se(c(\-|0|1)|47|mc|nd|ri)|sgh\-|shar|sie(\-|m)|sk\-0|sl(45|id)|sm(al|ar|b3|it|t5)|so(ft|ny)|sp(01|h\-|v\-|v )|sy(01|mb)|t2(18|50)|t6(00|10|18)|ta(gt|lk)|tcl\-|tdg\-|tel(i|m)|tim\-|t\-mo|to(pl|sh)|ts(70|m\-|m3|m5)|tx\-9|up(\.b|g1|si)|utst|v400|v750|veri|vi(rg|te)|vk(40|5[0-3]|\-v)|vm40|voda|vulc|vx(52|53|60|61|70|80|81|83|85|98)|w3c(\-| )|webc|whit|wi(g |nc|nw)|wmlb|wonu|x700|yas\-|your|zeto|zte\-/i',substr($useragent,0,4)))
	return true;
}

See here for lioghtweight phone detection strings: http://detectmobilebrowsers.com/. This might be a useful phone detecting class: https://github.com/serbanghita/Mobile-Detect/.

For the non-phone case, the relevant divs are shown/hidden by means of CSS.

jQuery

Controls are built into /js/_carousel.js:

/*==============================================================================
    Set up - give first items emphasis classes.
    Necessary because the slider is built dynamically
==============================================================================*/

   $("#mobile-reports-carousel .carousel-indicators li:first").addClass("active");
   $("#mobile-reports-carousel .carousel-indicators.no-numbers li:first").addClass("active");

   // Give active class to the first carousel item
   $("#mobile-reports-carousel .item:first").addClass("active");

   // Prevent auto play
   $('#mobile-reports-carousel').carousel({ interval: false });

/*==============================================================================
    Listen to the 'slid carousel' event to trigger code after each slide change
==============================================================================*/
$('#case-studies-carousel').on('slid.bs.carousel', function () {

  // For debugging...
  //alert("We have SLID!!! Thank God it's working...");

  // Useful variables
  var carouselData = $(this).data('bs.carousel');
  // get current index the Bootstrap way...
  var currentIndex = carouselData.getItemIndex(carouselData.$element.find('.item.active'));
  var total = carouselData.$items.length;

  // ...get the current index the jQuery way...
  var itemIndex = $(".item.active");
  var jCurrentIndex = itemIndex.index();

  // For debugging: needs a #carousel-count div in the markup
  var text = ( "Index: " + jCurrentIndex );
  $('#carousel-count').text(text);

  // Using bs method, getItemIndex - see: https://github.com/twbs/bootstrap/issues/14215
  //$('.carousel-indicators.no-numbers [data-slide-to=' + currentIndex + ']').addClass('active');//

  // Pure jQuery method - attach the active class to the indicator corresponding to the current item
  // -----------------------------------
  $('.carousel-indicators.no-numbers [data-slide-to=' + jCurrentIndex + ']').addClass('active');

});

Mobile Swipe Support

http://carawebs.com/dev-notes/bootstrap/wordpress/jquery/php/2014/07/05/bootstrap-carousel-wordpress/

Add Items to a WordPress Navigation ul

Wed, 08 Oct 2014 00:00:00 +0100

Add a button as the last item in a WordPress navigation:

  add_filter('wp_nav_menu_items','carawebs_payment_button', 10, 2);

function carawebs_payment_button($items, $args) {

  if (!is_admin() && $args->theme_location == 'primary_navigation') {

  $items .= '<li class="pay-button"><a href="http://#"><button>Click</button></a></li>';



  }

  return $items;
}

Add a search form as the last item in a WordPress navigation:

  add_filter('wp_nav_menu_items','carawebs_payment_button', 10, 2);

function carawebs_payment_button($items, $args) {

  if (!is_admin() && $args->theme_location == 'primary_navigation') {

  ob_start();
  get_search_form();
  $searchform = ob_get_contents();
  ob_end_clean();

  $items .= '<li class="search">Search ' . $searchform . '</li>';

  }

  return $items;
}

Securing a Ubuntu 14.04 Server

Thu, 25 Sep 2014 00:00:00 +0100

These guidelines assume a fresh install of Ubuntu 14.04.

Secure SSH Access

The most important step is probably to change the default port for SSH.

Change the default SSH port.

Many brute force hacking attempts target the username “root” on port 22.

Simply changing the port can reduce the number of malicious login attempts drastically.

Change the SSH port number to something between 1025 and 65536. The Server’s SSH connections will then look to the new port number.

In Ubuntu the SSH port is set in:

/etc/ssh/sshd_config.

Before going any further, backup the config file!

cp /etc/ssh/sshd_config /etc/ssh/sshd_config_backup

Open the config file and amend the port number (first uncommented line in the file):

sudo nano /etc/ssh/sshd_config

Restart SSH to enable the changes:

service ssh restart or

/etc/init.d/sshd restart

TEST THE NEW CONFIGURATION

Keep your terminal open, and try to log in with a new terminal. In this way, if you can’t login you still have server access.

Remember to specify the correct port number (see below).

Assuming the SSH port is 1234.

SSH login: ssh -p 1234 user@12.34.567.89

Using scp: scp -P 1234 username@12.34.567.89:/path/file.txt Note: uppercase P is necessary - lowercase p is an scp switch.

Using rsync: rsync --progress -a -v -rz --checksum --delete -e "ssh -p 1234" _site/ user@12.34.567.89:/var/www/path/directory

Once the SSH port has been reset, prevent root login with password. Root will still be able to gain access by means of public SSH keys - which can be set up on your local machine.

Amend the following block in /etc/ssh/sshd_config

# Authentication:
LoginGraceTime 120
# root can't log in via SSH & password - allows root login only with public key
PermitRootLogin without-password
StrictModes yes

Alternatively, prevent root access altogether: PermitRootLogin no

Make sure you’ve created a new user with sudo privileges before doing this.

After making changes to SSH configuration, restart SSH: /etc/init.d/sshd restart

Set up a Firewall

Enter sudo iptables -L, will return an empty ruleset:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Make a file for firewall rules:

sudo nano /etc/iptables.firewall.rules

…and add some rules:

*filter

#  Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT

#  Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

#  Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allow SSH connections
#
#  The -dport number should be the same port number you set in sshd_config
#
-A INPUT -p tcp -m state --state NEW --dport 1234 -j ACCEPT

#  Allow ping
-A INPUT -p icmp -j ACCEPT

#  Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

#  Drop all other inbound - default deny unless explicitly allowed policy
-A INPUT -j DROP
-A FORWARD -j DROP

COMMIT

Activate rules:

sudo iptables-restore < /etc/iptables.firewall.rules

Amend /etc/iptables.firewall.rules as necessary.

Check you Still Have SSH Access

The firewall rules set the port number for SSH access - this should correspond to the value set in /etc/ssh/sshd_config.

Make sure!

While still logged in on one terminal, open a new terminal and check SSH connection is still functional.

Apply Firewall Rules When System Reboots

sudo nano /etc/network/if-pre-up.d/firewall

Add:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.firewall.rules

Save, make executable:

sudo chmod +x /etc/network/if-pre-up.d/firewall

Fail2Ban

Fail2Ban Manual

sudo apt-get install fail2ban

View Fail2Ban configuration settings - read only:

nano /etc/fail2ban/jail.conf

This file can be modified by package upgrades - thereby losing customisations - so to make changes, create a jail.local file.

First copy the jail.conf:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Make all amendments to jail.local.

Default setting ignores traffic from the local machine: ignoreip = 127.0.0.1/8. Additional IP addresses could be added here, by appending to the existing parameter, separating IP addresses with a space.

You could add the local office IP address for example, to prevent accidental lockout.

Check it’s running: sudo fail2ban-client status

Check Rules

Enter sudo iptables -S

Typical Output:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N fail2ban-ssh
-A INPUT -p tcp -m multiport --dports 22 -j fail2ban-ssh
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 12345 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -j DROP
-A FORWARD -j DROP
-A OUTPUT -j ACCEPT
-A fail2ban-ssh -j RETURN

The –dport number on line 11 corresponds to the number set in firewall rules - SSH connections are accepted on that port.

This means fail2ban port should change. Amend /etc/fail2ban/jail.local:

[ssh]

enabled  = true
# change the port from ssh to the proper value
#port     = ssh
port   = 12345
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 6

Once this change is made, reload fail2ban configuration:

sudo fail2ban-client reload

Because Fail2Ban sets the port as “ssh” bu default, it searches in the /etc/services file and maps ssh to the named port.

The port is set to 22 in this file by default. Adjusting this value would be another way of pointing Fail2Ban to the right port.

See this Server Fault question.

Handy Fail2Ban commands

sudo fail2ban-client start - starts the server and the jails
sudo fail2ban-client reload - reloads the configuration
sudo fail2ban-client reload <JAIL> - reloads the jail <JAIL>
sudo fail2ban-client stop - stops all jails and terminate the server
sudo fail2ban-client status - gets the current status of the server
sudo fail2ban-client ping - tests if the server is alive
sudo fail2ban-client help - return this output

More Fail2Ban commands.

References

Bootstrap Modals

Thu, 25 Sep 2014 00:00:00 +0100

#Bootstrap Modals Modals enable user triggered hide and display of additional HTML content.

To set up a pure HTML Bootstrap 3 Modal triggered by a link, use the following markup

<a href="javascript:;" data-toggle="modal" data-target="#myModal">More Info...</a>
<div class="modal fade" id="myModal" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true">
 <div class="modal-dialog">
  <div class="modal-content">
   <div class="modal-header">
    <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span><span class="sr-only">Close</span></button>
     <h4 class="modal-title" id="myModalLabel"><?php echo $name; ?></h4>
    </div>
  <div class="modal-body">
   <p>Some HTML content</p>
  </div>
  <div class="modal-footer">
   <button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
   <button type="button" class="btn btn-primary">Save changes</button>
   </div>
  </div>
 </div>
</div>

WordPress

It’s pretty easy to wire up modals within WordPress.

This function takes custom field data and uses it to build a modal:

/*==============================================================================
  Biographical Repeater
  ============================================================================*/

function carawebs_biographical_repeater () {
  $repeater = "people";

  if(get_field($repeater)){

    $i = 1;

    while(has_sub_field( $repeater )):

      $name = get_sub_field('name');
      $text = get_sub_field('biographical_info');
      $modal_text = get_sub_field('modal_text');
      $link = get_sub_field('linked_in');
      $image = get_sub_field('person_image');

      // In ACF, set image to return an image object
      // Image variables
      //$url = $image['url'];
      $title = $image['title'];
      $alt = $image['alt'];
      $caption = $image['caption'];

      // thumbnail
      $size = 'medium';
      $thumb = $image['sizes'][ $size ]; // Returns the URL of the correct size
      $width = $image['sizes'][ $size . '-width' ];
      $height = $image['sizes'][ $size . '-height' ];

      // The Output
      ?>
      <div class="col-sm-4 col-sm-offset-1">
         <img class="img-responsive" <?php if ($title) echo 'title="' . $title . '"'; ?> src="<?php echo $thumb; ?>" alt="<?php echo $alt; ?>" width="<?php echo $width; ?>" height="<?php echo $height; ?>" /><br>
         <h4><?php echo $name; ?></h4><br>
         <div class="post-excerpt"><?php echo $text; ?></div>
          <!-- Modal trigger - the target is built using a counter, $i -->
          <a href="javascript:;" data-toggle="modal" data-target="#myModal-<?php echo $i; ?>">More Info about <?php echo $name; ?></a>
          <!-- Modal Code Begins -->
          <div class="modal fade" id="myModal-<?php echo $i; ?>" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true">
             <div class="modal-dialog">
              <div class="modal-content">
                 <div class="modal-header">
                  <button type="button" class="close" data-dismiss="modal"><span aria-hidden="true">&times;</span><span class="sr-only">Close</span></button>
                  <h4 class="modal-title" id="myModalLabel"><?php echo $name; ?></h4>
                 </div>
                 <div class="modal-body">
                  <?php echo $modal_text; ?>
                 </div>
                 <div class="modal-footer">
                  <button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
                  <button type="button" class="btn btn-primary">Save changes</button>
                 </div>
              </div>
             </div>
          </div>
          <!-- Modal end -->
          <a href="<?php echo $link; ?>" class="linked-in"><?php echo $name; ?>'s Linked In Profile</a><br>
      </div>
      <?php

    $i++;

    endwhile;

    ?></div><?php

		}

}

Body Jump in Chrome

I’m not sure if this occurs because of a fixed navbar, but the Bootstrap JS adds padding-right to the body element when the modal-open class is added:

<body class="page page-id-18 logged-in admin-bar about customize-support modal-open" style="padding-right: 15px;">

This can be corrected with:

// Prevent Modal Jumping
// -------------------------

body.modal-open {
  padding-right: 0!important;
}

Remove Unused CSS in WordPress

Fri, 19 Sep 2014 00:00:00 +0100

#Using Grunt Uncss with WordPress/Roots Building a site with the Bootstrap framework is a great way to take advantage of free open source styling.

CSS frameworks like Foundation and Bootstrap can’t be ignored - they are efficient collaborative efforts - and they only look “samey” when developers/designers get lazy.

When building a site with Bootstrap 3 under the Roots framework, you’ll typically have a minified CSS file of approximately 131kB. This is quite big, and the majority of styles won’t be used.

One option is to comment out the LESS modules that aren’t being used so that they are not compiled (no carousel? don’t compile carousel.less) but this is a blunt tool and still results in many CSS rules being loaded in the browser.

Enter UnCSS. This is a tool that removes unused CSS from stylesheets. it efficiently turns a sledgehammer into a precision instrument.

UnCSS had been turned into a Grunt module by Addy Osmani.

Though it has been designed to work with static sites, Liam Gladdy has developed a workaround for WordPress that works exceptionally well. There is a complete description of Liam’s method in this blog article. It involves using a WordPress plugin to build a JSON sitemap, and passing this to the UnCSS Grunt task. The UnCSS magic then checks each page of your dynamically generated site, and removes any unused style rules from the compiled CSS. This is an amazingly useful resource - and has the potential to give a significant performance boost.

We modified the Roots Gruntfile.js to manage all this - adding some minification for further savings.

Build the Sitemap

Create a mu-plugins directory in the wp-content directory of your development site, and add the JSON sitemap plugin.

You’ll need to autoload the mu plugin - create a file called load.php inside the mu-plugins dirtectory, and add the following line, referencing the correct path:

require WPMU_PLUGIN_DIR.'/JSON-Sitemap-Generator/grunt-plugin.php';

Make sure package.json includes the following lines before running sudo npm install:

"devDependencies": {
    "grunt": "~0.4.2",
    "grunt-contrib-clean": "~0.5.0",
    "grunt-contrib-cssmin": "^0.10.0",
    "grunt-contrib-jshint": "~0.6.4",
    "grunt-contrib-less": "~0.9.0",
    "grunt-contrib-uglify": "~0.2.4",
    "grunt-contrib-watch": "~0.5.3",
    "grunt-exec": "^0.4.6",
    "grunt-uncss": "^0.3.6",
    "grunt-wp-version": "~0.1.0"
  }

Add an exec task to Gruntfile.js that builds the sitemap:

exec: {
      get_grunt_sitemap: {
	// the output of the show_sitemap() function (from the plugin) used to build sitemap.json
        command: 'curl --silent --output sitemap.json http://localhost/useful/?show_sitemap'
      }
    },

Create an UnCSS Grunt Task

UnCSS task, followed by a cssmin task to minify the output CSS file.

uncss: {
      dist: {
        options: {
          ignore       : [
          // needed for Bootstrap's transitions
          '.bs.carousel',
          '.slid.bs.carousel',
          '.slide.bs.carousel',
          '.fade',
          '.fade.in',
          '.collapse',
          '.collapse.in',
          '.collapsing',
          '.alert-danger',
          '.logged-in .navbar-default',
          '.carousel-inner > .next',
          '.carousel-inner > .prev',
          '.carousel-inner > .next',
          '.carousel-inner > .prev',
          '.carousel-inner > .next.left',
          '.carousel-inner > .prev.right',
          '.carousel-inner > .active.left',
          '.carousel-inner > .active.right'

          ],
          stylesheets  : ['assets/css/main.css'],
          ignoreSheets : [/fonts.googleapis/],
          urls         : [], //Overwritten in load_sitemap_and_uncss task
        },
        files: {
          'assets/css/main.css': ['**/*.php']
        }
      }
    },
cssmin: {
        options: {
          // cssmin has a problem with relative urls. Ref: http://stackoverflow.com/questions/23916622/how-to-rewrite-relative-url-in-minified-css-with-cssmin
          // set 'noAdvanced: true' to circumvent this
                keepSpecialComments: 0,
                //target: 'commonDir',
                noAdvanced: true,
                banner: '/* ======= David Egan, carawebs: http://carawebs.com =======*/'
            },
        css:{

          files: {
            "assets/css/main.min.css": ["assets/css/main.css"]
            }

          }
      }

Register Tasks

grunt.registerTask('load_sitemap_json', function() {
    var sitemap_urls = grunt.file.readJSON('./sitemap.json');
    // pass the URLs to the uncss:dist task
    grunt.config.set('uncss.dist.options.urls', sitemap_urls);
  });

grunt.registerTask('deploy_build', [
  'clean',
  //'less',
  'less:deploy',
  'uglify',
  'exec:get_grunt_sitemap',
  'load_sitemap_json',
  'uncss:dist',
  'cssmin:css',
  'version'
  ]);

Entire Gruntfile, Roots 6.5

'use strict';
module.exports = function(grunt) {

  grunt.initConfig({
    jshint: {
      options: {
        jshintrc: '.jshintrc'
      },
      all: [
        'Gruntfile.js',
        'assets/js/*.js',
        '!assets/js/scripts.min.js'
      ]
    },
    less: {
      dist: {
        files: {
          'assets/css/main.min.css': [
            'assets/less/app.less'
          ]
        },
        options: {
          compress: true,
          // LESS source map
          // To enable, set sourceMap to true and update sourceMapRootpath based on your install
          sourceMap: true,
          sourceMapFilename: 'assets/css/main.min.css.map',
          sourceMapRootpath: '/useful/wp-content/themes/useful-studio/' // '/useful' necessary cos site not in root of domain
        }
      },

      deploy: {
        files: {
          'assets/css/main.css': [
            'assets/less/app.less'
          ]
        },

      }
    },
    uglify: {
      dist: {
        files: {
          'assets/js/scripts.min.js': [
            'assets/js/plugins/bootstrap/transition.js',
            'assets/js/plugins/bootstrap/alert.js',
            'assets/js/plugins/bootstrap/button.js',
            'assets/js/plugins/bootstrap/carousel.js',
            'assets/js/plugins/bootstrap/collapse.js',
            'assets/js/plugins/bootstrap/dropdown.js',
            'assets/js/plugins/bootstrap/modal.js',
            'assets/js/plugins/bootstrap/tooltip.js',
            'assets/js/plugins/bootstrap/popover.js',
            'assets/js/plugins/bootstrap/scrollspy.js',
            'assets/js/plugins/bootstrap/tab.js',
            'assets/js/plugins/bootstrap/affix.js',
            //'assets/js/plugins/bootstrap-touch-carousel.js',// swipe support
            'assets/js/plugins/*.js',
            'assets/js/_*.js'
          ]
        },
        options: {
          // JS source map: to enable, uncomment the lines below and update sourceMappingURL based on your install
          // sourceMap: 'assets/js/scripts.min.js.map',
          // sourceMappingURL: '/app/themes/roots/assets/js/scripts.min.js.map'
        }
      }
    },
    version: {
      options: {
        file: 'lib/scripts.php',
        css: 'assets/css/main.min.css',
        cssHandle: 'roots_main',
        js: 'assets/js/scripts.min.js',
        jsHandle: 'roots_scripts'
      }
    },
    watch: {
      less: {
        files: [
          'assets/less/*.less',
          'assets/less/bootstrap/*.less'
        ],
        tasks: ['less', 'version']
      },
      js: {
        files: [
          '<%= jshint.all %>'
        ],
        tasks: ['jshint', 'uglify', 'version']
      },
      livereload: {
        // Browser live reloading
        // https://github.com/gruntjs/grunt-contrib-watch#live-reloading
        options: {
          livereload: false
        },
        files: [
          'assets/css/main.min.css',
          'assets/js/scripts.min.js',
          'templates/*.php',
          '*.php'
        ]
      }
    },
    clean: {
      dist: [
        'assets/css/main.min.css',
        'assets/js/scripts.min.js'
      ]
    },

    exec: {
      get_grunt_sitemap: {
        command: 'curl --silent --output sitemap.json http://localhost/useful/?show_sitemap'
      }
    },

    // Using uncss with WordPress:
    // https://gladdy.uk/blog/2014/04/13/using-uncss-and-grunt-uncss-with-wordpress/

    uncss: {
      dist: {
        options: {
          ignore       : [
          // needed for Bootstrap's transitions
          '.bs.carousel',
          '.slid.bs.carousel',
          '.slide.bs.carousel',
          '.fade',
          '.fade.in',
          '.collapse',
          '.collapse.in',
          '.collapsing',
          '.alert-danger',
          '.logged-in .navbar-default',
          '.carousel-inner > .next',
          '.carousel-inner > .prev',
          '.carousel-inner > .next',
          '.carousel-inner > .prev',
          '.carousel-inner > .next.left',
          '.carousel-inner > .prev.right',
          '.carousel-inner > .active.left',
          '.carousel-inner > .active.right'

          ],
          stylesheets  : ['assets/css/main.css'],
          ignoreSheets : [/fonts.googleapis/],
          urls         : [], //Overwritten in load_sitemap_and_uncss task
        },
        files: {
          'assets/css/main.css': ['**/*.php']
        }
      }
    },

    cssmin: {
        options: {
          // cssmin has a problem with relative urls. Ref: http://stackoverflow.com/questions/23916622/how-to-rewrite-relative-url-in-minified-css-with-cssmin
          // set 'noAdvanced: true' to circumvent this
                keepSpecialComments: 0,
                //target: 'commonDir',
                noAdvanced: true,
                banner: '/* ======= David Egan, carawebs: http://carawebs.com =======*/'
            },
        css:{

          files: {
            "assets/css/main.min.css": ["assets/css/main.css"]
            }

          }
      }

  });

  // Load tasks
  grunt.loadNpmTasks('grunt-contrib-clean');
  grunt.loadNpmTasks('grunt-contrib-jshint');
  grunt.loadNpmTasks('grunt-contrib-uglify');
  grunt.loadNpmTasks('grunt-contrib-watch');
  grunt.loadNpmTasks('grunt-contrib-less');
  grunt.loadNpmTasks('grunt-wp-version');
  grunt.loadNpmTasks('grunt-exec');
  grunt.loadNpmTasks('grunt-uncss');
  grunt.loadNpmTasks('grunt-contrib-cssmin');

// Register tasks
grunt.registerTask('default', [
    'clean',
    'less',
    'uglify',
    'version'
  ]);


grunt.registerTask('load_sitemap_json', function() {
    var sitemap_urls = grunt.file.readJSON('./sitemap.json');
    grunt.config.set('uncss.dist.options.urls', sitemap_urls);
  });

grunt.registerTask('deploy_build', [
  'clean',
  //'less',
  'less:deploy',
  'uglify',
  'exec:get_grunt_sitemap',
  'load_sitemap_json',
  'uncss:dist',
  'cssmin:css',
  'version'
  ]);

grunt.registerTask('dev', [
    'watch'
  ]);

};

Glyphicons CSS for Bootstrap 3

Thu, 18 Sep 2014 00:00:00 +0100

To add Glyphicons in CSS, use pseudo classes. For example, this code substitutes a glyphicon left chevron for the usual bullet in the .entry-content ul:

.entry-content ul {margin-left: 0; padding-left: 0;}

.entry-content ul li {
  padding: 0 0 5px 20px;
  display: block;
  position: relative;
  margin-left: 0;
}
.entry-content ul li:before {
  font-family: 'Glyphicons Halflings';
  /* font-family: FontAwesome; */
  position: absolute;
  left: 0;
  top:0;
  font-size:80%
}

.entry-content ul li:before {
 content: '\e080';
 /* content: '\f105'; // Font awesome equivalent */
}

Go here: Glyphicons Cheatsheet for Bootstrap 3

nuff said!

Customise WordPress Menu Widget

Tue, 02 Sep 2014 00:00:00 +0100

WordPress has a handy menu widget that allows you to select and display a navigation menu in a registered widget area (like a sidebar or footer).

By copying the code from WP core, modifying it, and adding it to either a custom function or a plugin, the menu output can be customised.

In this case, the li elements are prepended by stars (delivered by font awesome).

PHP

/*==============================================================================
  Modified Navigation Menu widget class
*=============================================================================*/

class Carawebs_Nav_Menu_Widget extends WP_Widget {

	function __construct() {
		$widget_ops = array( 'description' => __('Add a custom menu to your sidebar.') );
		parent::__construct( 'nav_menu', __('Carawebs Custom Menu'), $widget_ops );
	}
	/*
  //constructor -- name this the same as the class above
    function Carawebs_Nav_Menu_Widget() {
        parent::WP_Widget(false, $name = 'Carawebs Custom Menu');
    }
  */

	function widget($args, $instance) {
		// Get menu
		$nav_menu = ! empty( $instance['nav_menu'] ) ? wp_get_nav_menu_object( $instance['nav_menu'] ) : false;

		if ( !$nav_menu )
			return;

		// This filter is documented in wp-includes/default-widgets.php
		$instance['title'] = apply_filters( 'widget_title', empty( $instance['title'] ) ? '' : $instance['title'], $instance, $this->id_base );

		echo $args['before_widget'];

		if ( !empty($instance['title']) )
			echo $args['before_title'] . $instance['title'] . $args['after_title'];

    // Add <i></i> or classes for customisation
		wp_nav_menu( array( 'fallback_cb' => '', 'menu' => $nav_menu, 'menu_class'=> 'i-ul', 'before'=>'<i class="icon-star-sharp i-li orange"></i>' ));

		echo $args['after_widget'];
	}

	function update( $new_instance, $old_instance ) {
		$instance['title'] = strip_tags( stripslashes($new_instance['title']) );
		$instance['nav_menu'] = (int) $new_instance['nav_menu'];
		return $instance;
	}

	function form( $instance ) {
		$title = isset( $instance['title'] ) ? $instance['title'] : '';
		$nav_menu = isset( $instance['nav_menu'] ) ? $instance['nav_menu'] : '';

		// Get menus
		$menus = wp_get_nav_menus( array( 'orderby' => 'name' ) );

		// If no menus exists, direct the user to go and create some.
		if ( !$menus ) {
			echo '<p>'. sprintf( __('No menus have been created yet. <a href="%s">Create some</a>.'), admin_url('nav-menus.php') ) .'</p>';
			return;
		}
		?>
		<p>
			<label for="<?php echo $this->get_field_id('title'); ?>"><?php _e('Title:') ?></label>
			<input type="text" class="widefat" id="<?php echo $this->get_field_id('title'); ?>" name="<?php echo $this->get_field_name('title'); ?>" value="<?php echo $title; ?>" />
		</p>
		<p>
			<label for="<?php echo $this->get_field_id('nav_menu'); ?>"><?php _e('Select Menu:'); ?></label>
			<select id="<?php echo $this->get_field_id('nav_menu'); ?>" name="<?php echo $this->get_field_name('nav_menu'); ?>">
		<?php
			foreach ( $menus as $menu ) {
				echo '<option value="' . $menu->term_id . '"'
					. selected( $nav_menu, $menu->term_id, false )
					. '>'. $menu->name . '</option>';
			}
		?>
			</select>
		</p>
		<?php
	}
}

add_action('widgets_init', create_function('', 'return register_widget("Carawebs_Nav_Menu_Widget");'));

// Unregister the default nav menu widget
function remove_WP_Nav_widget() {
	unregister_widget('WP_Nav_Menu_Widget');
}

add_action( 'widgets_init', 'remove_WP_Nav_widget' );

i-ul {
  display: inline-block;
  padding-left: 0;
  margin-left: 2.142857142857143em;
  list-style-type: none;
}
.i-ul > li {
  position: relative;
}
.i-li {
  display: inline-block;
  position: absolute;
  left: -2.142857142857143em;
  width: 2.142857142857143em;
  top: 0;
  text-align: center;
}

Row back to an Older Git Repo

Wed, 27 Aug 2014 00:00:00 +0100

Sometimes you might accidentally pull changes from a remote repo by running git pull origin master inadvertently overwriting local changes.

Fortunately there may be a way to go back to the pre-pulled state.

Run git reflog. This will provide an output like this:

60a22c5 HEAD@{0}: commit: Side commenting system
66e1412 HEAD@{1}: pull origin master: Fast-forward
deff5af HEAD@{2}: commit: grunt build
477fd3d HEAD@{3}: commit: test setup
497e4b7 HEAD@{4}: commit: sidebar dropdown menu
1e18e81 HEAD@{5}: commit: Live reload enabled, sourcemaps working
8a1a95f HEAD@{6}: commit (initial): First Commit

You can revert to one of these commits. E.g.:

git reset --hard 60a22c5

Git will output a message: HEAD is now at 60a22c5 Side commenting system and you’ll be back at this commit.

Set Up Chrome DevTools Workspace

Sat, 16 Aug 2014 00:00:00 +0100

Setting up Workspaces allows Chrome DevTools to write to disc. This is handy during local development - it speeds up workflow by allowing changes to LESS/CSS in the Chrome DevTools “Sources” window.

It requires compiled CSS (e.g. from LESS) to be source mapped - in our case this is usually taken care of by the Grunt grunt-contrib-less module.

Set up Workspaces

The sources tab is basically a file navigator in the Chrome Dev Tools sidebar that shows resources loaded by the web page. Set it up to write to disk:

Open local webpage NOTE - the page must be server output for LESS source mapping to work
Set up LESS source maps
Open Dev Tools
Right click in the navigation area of the “Sources” tab
Select “Add Folder to Workspace”
Give DevTools access - the folder will appear in the IDE-like navigation
Right click on a file within the folder
Select “Map to File System”
DevTools wil recognise if the file on the filesystem has the same name as a page resource (e.g. http://localhost:9000/less/main.less and user/testsite/css/main.less)

This only needs to be done once per project.

Using the DevTools Workspace

Right click and inspect element
Select the LESS or CSS file associated with the rule that you want to edit
The relevant file opens in the Sources tab
Edit the file and click ctrl + s to write changes to disk

Set Up LESS Sourcemap

Sample Gruntfile.js less task for local WordPress installation - requires grunt-contrib-less:

less: {
      dist: {
        files: {
          'assets/css/main.min.css': [
            'assets/less/app.less'
          ]
        },
        options: {
          compress: true,
          // LESS source map
          // To enable, set sourceMap to true and update sourceMapRootpath based on your install
          sourceMap: true,
          sourceMapFilename: 'assets/css/main.min.css.map',
          sourceMapRootpath: '/useful/wp-content/themes/useful-studio/' // '/useful' necessary cos site not in root of domain
        }
      }

Sample less task from a static HTML site served by means of grunt-contrib-connect module, serving on port 9000:

less: {
      development: {
        options: {
          compress: true,
          sourceMap: true,
          sourceMapRootpath: 'http://localhost:9000/', // adds a path onto the less files in the source map
          sourceMapFilename: 'css/agency.css.map'
        },
        files: {'less/main.min.css': [
            'less/main.less']
            }
        }
    }

The sourcemap is created, providing a reference to the relevant LESS files. The sourcemap is in turn referenced in the compiled CSS, in a comment at the end of the file.

Install Jekyll on Ubuntu

Sun, 03 Aug 2014 09:38:44 +0000

Jekyll is a static site generator that is “blog aware” - it looks like a dynamic website (and really it is) even though it is served up by static HTML and does not have a database. All the automatic connection and site building takes place locally, before the site is uploaded to production. With some Grunt configuration using rsync, Jekyll sites are easily deployed. Jekyll is not a CMS.

Jekyll requirements:

Ruby
RubyGems - package management framework for Ruby
NodeJS

Useful article by Chris Oliver on setting up Ruby on Rails on Ubuntu 14.04 Trusty Tahr. This includes an interactive script generator that allows different versions of Ruby to be installed.

Set Up Ruby

Ruby dependencies:

sudo apt-get update

sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties

Install Ruby using rbenv - will result in a more up to date version of Ruby compared with “sudo apt-get install ruby”.

First install rbenv. More info on rbenv here. This code based on the Chris Oliver article.

cd
git clone git://github.com/sstephenson/rbenv.git .rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
exec $SHELL

Install ruby-build:

git clone git://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc
exec $SHELL

rbenv install 2.1.2
rbenv global 2.1.2
ruby -v

Install Ruby gems: gem update --system # may need to be sudo

Tell Rubygems not to install local docs:

echo "gem: --no-ri --no-rdoc" > ~/.gemrc

NodeJS

Add this using a PPA repositiory.

sudo add-apt-repository ppa:chris-lea/node.js

sudo apt-get update

sudo apt-get install nodejs

Install Jekyll

sudo gem install jekyll

Build a Jekyll site

gem install jekyll
jekyll new mysite
cd mysite
jekyll serve
# => Now browse to http://localhost:4000

Install Atom Ubuntu

Sun, 03 Aug 2014 09:38:44 +0000

Atom is an excellent open source text-editor by GitHub, released under the MIT license. To install on Ubuntu, add the webupd8team ppa:

sudo add-apt-repository ppa:webupd8team/atom

sudo apt-get update

sudo apt-get install atom

Add the Emmet package through the Atom package manager.

Set up a LAMP stack

Fri, 01 Aug 2014 09:38:44 +0000

Install Apache

sudo apt-get update

sudo apt-get install apache2

If Apache has installed correctly, http://localhost will now display the Apache2 Ubuntu Default Page.

Install MySQL

We will also include “helper” packages:

sudo apt-get install mysql-server php5-mysql

It is not necessary to run sudo apt-get update before installing MySQL because it was run prior to installing Apache - so we can assume that everything is up to date.

Set passwords for root user when prompted (twice).

Set up MySQL database directory structure:

sudo mysql_install_db

Remove dangerous defaults - removes anonymous users, and prevents root login remotely, etc. Be careful with removal of remote root login - new user will be necessary. This script is probably not necessary on local environment install:

sudo mysql_secure_installation

If the MySQL install goes wrong and you need to start again:

sudo apt-get remove --purge mysql*

sudo apt-get autoremove

sudo apt-get autoclean

Install PHP

sudo apt-get install php5 libapache2-mod-php5 php5-mcrypt

Modify default for Apache serving files, give preference to PHP:

sudo nano /etc/apache2/mods-enabled/dir.conf

<IfModule mod_dir.c>
    DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm
</IfModule>

Bring index.php to head of the queue.

Restart Apache:

sudo service apache2 restart

PHP Modules

Get a list: apt-cache search php5-

Test PHP:

sudo nano /var/www/html/info.php

Enter:

<?php
phpinfo();
?>

Don’t leave the file in place - security vulnerability:

sudo rm /var/www/html/info.php

NOTE: Document root is currently /var/www/html

Change Document Root & Enable Rewrites

To change document root to /var/www, and enable rewrites.

Make sure mod_rewrite is installed:

sudo a2enmod rewrite

If mod_rewrite is not enabled, running this command will enable it. Restart Apache:

service apache2 restart

Find output of ls /etc/apache2/sites-enabled

In this case, 000-default.conf

This file is used for httpd configuration in /etc/apache2/sites-available/

Edit the config file: sudo nano /etc/apache2/sites-available/000-default.conf …by adding this code block, and commenting out the DocumentRoot /var/www/html:

DocumentRoot /var/www
<Directory /var/www/>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
</Directory>

Restart Apache: sudo service apache2 restart

Install PHPMyAdmin

sudo apt-get install phpmyadmin

sudo nano /etc/apache2/apache2.conf …and include this line:

Include /etc/phpmyadmin/apache.conf

“Since Ubuntu 13.10 (Saucy Salamander), Apache no longer loads configuration files from the /etc/apache2/conf.d directory. Instead, they are loaded from the /etc/apache2/conf-enabled directory. Therefore, if you need to manually include the phpMyAdmin-shipped Apache configuration file, you must run the following:”

sudo ln -s /etc/phpmyadmin/apache.conf /etc/apache2/conf-enabled/phpmyadmin.conf

sudo /etc/init.d/apache2 reload

Install PHP curl

sudo apt-get install php5-curl

sudo service apache2 restart

Install cURL

“Curl is a command line tool for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberosâ€¦), file transfer resume, and proxy tunneling.”

sudo apt-get install curl libcurl3 libcurl3-dev php5-curl

This allows WP-CLI to be installed.

Bootstrap Carousel & Thumbnail Navigation for WordPress

Sat, 05 Jul 2014 09:38:44 +0000

A Bootstrap carousel with thumbnail controls for WordPress.

Project files are available on GitHub - fork or clone your own version.

Quick Start

Paste contents of carousel-linked-thumbnails.php into the theme custom.php file
Create ACF repeater field called carousel_images
Create a sub field image - set this to return image ID
If using the Roots framework, add js/_carousel.js to the theme assets/js/vendor directory
If you use a different directory for the js files, remember to reference it properly in enqueue.php
Include jquery.touchSwipe.min.js and touchControl.js in assets/js/vendor for mobile swipe support.
Enqueue the javascript files by adding the contents of enqueue.php to your theme custom.php file, or /lib/scripts.php for the Roots framework
Include the contents of carousel-linked-thumbnails.css in the theme CSS/LESS

Intro ID

This is really two linked carousels:

A full size carousel, populated from an ACF repeater field
A thumbnail sized carousel, populated from the same field

The thumbnail carousel targets the main carousel by means of the Bootstrap data-target attribute. The following code block shows how the carousel with id “myCarousel” is targeted. $count works as a counter, and because both sliders are populated from the same repeater field cross-referencing is easy.

?>
<li data-target="#myCarousel" id="carousel-selector-<?php echo $count; ?>"data-slide-to="<?php echo $count; ?>">
    <img src="<?php echo $thumbimage[0]; ?>" />
</li>
<?php

The data attribute data-slide-to is used to pass a raw slide index to myCarousel - it does what it says on the tin. In this case, clicking the thumbnail triggers a slide to the $count slide on the main carousel.

Building the Carousels

The carousel is dynamic - it takes advantage of the Advanced Custom Fields WordPress plugin that allows the user to upload/select images.

An ACF field group is built, and associated with a Custom Post Type (in this case, the CPT “project”).
A repeater field is added - the repeater field is a premium add-on for ACF (it is extremely useful and ridiculously under-priced).
An “image” sub-field is added to the repeater field.
If necessary, extra fields can be added to allow fine user control over what is displayed on each slide.
The image data for each slide is held as a row in the ACF repeater field.

For these files, the repeater field is named carousel_images.

The image sub-field is named image.

The PHP function for building the carousels is in the file carousel-linked-thumbnails.php. The code in this file could be added to the theme custom.php file. View the code here:

/*==============================================================================
  Carousel from ACF Repeater Field.
==============================================================================*/
/*  This uses repeater field data, where the sub-field is the "image" type.
*   Set ACF to return image ID for the sub-field.
*   Repeater field name  = 'carousel_images'
*   Sub field 1          = 'image'
*/

function carawebs_carousel_images_slider() {

 $number = 0;
 $carousel_images = get_field('carousel_images');

  if(!empty($carousel_images) ){ // run if the repeater field has content
    ?>
    <div class="row">
        <!-- data-ride="carousel" causes auto play class="slide" causes slide-->
        <div id="myCarousel" class="carousel slide" data-ride="carousel">
          <!-- Carousel items -->
          <div class="carousel-inner">
            <?php
            $slidecount = 1; //Allows unique ids on items

            while( has_sub_field('carousel_images') ){

              $attachment_id = get_sub_field('image');
  		        $size = "full"; // (thumbnail, medium, large, full or custom size)
  		        $image = wp_get_attachment_image_src( $attachment_id, $size );
  		        // url = $image[0];
  		        // width = $image[1];
  		        // height = $image[2];

              ?>
                  <div id="slide-<?php echo $slidecount; ?>" class="item">
                      <img src="<?php echo $image[0]; ?>" />
                  </div><?php

              $slidecount++;
            } // End the while loop

          ?>
          </div>
          <!-- Carousel nav -->
          <a class="left carousel-control" href="#myCarousel" data-slide="prev">
          <span class="glyphicon glyphicon-chevron-left"></span></a>
          <a class="right carousel-control" href="#myCarousel" data-slide="next">
          <span class="glyphicon glyphicon-chevron-right"></span></a>

        </div>
<!--============================================================================
  The Thumbnail Navigation
=============================================================================-->
    <div class="container">
      <div class="carousel slide" id="thumb-carousel">
          <div class="carousel-inner">
            <?php

            $index = 0;
            $i = 1;
            //added before the while loop to ensure the div gets opened
            echo '<div id="item-' . $i . '" class="item"><ul class="carousel-thumbnails">';

            $rowcount = count(get_field('carousel_images')); // the total number of images - counts rows in the ACF repeater array

                while( has_sub_field('carousel_images') ){
                     $count = $index++;
                     /*===========================================================*/
                     $slidenumber = 4; // SET NUMBER OF SLIDES HERE
                     /*===========================================================*/
                     $thumb_id = get_sub_field('image');
                     $thumbsize = "thumbnail"; // (thumbnail, medium, large, full or custom size)
                     $thumbimage = wp_get_attachment_image_src( $thumb_id, $thumbsize );
                     // url = $thumbimage[0];
                     // width = $thumbimage[1];
                     // height = $thumbimage[2];

                     ?>
                      <li data-target="#myCarousel" id="carousel-selector-<?php echo $count; ?>"data-slide-to="<?php echo $count; ?>">
                         <img src="<?php echo $thumbimage[0]; ?>" />
                      </li><?php

                 // if multiple of 3 close div and open a new div
                 $itemNumber = ($i / $slidenumber) + 1;
                 if($i % $slidenumber == 0 && $i < $rowcount) {echo '</ul></div><div id="item-' . $itemNumber . '" class="item"><ul class="carousel-thumbnails">';}

                $i++; };
            //make sure open div is closed
            echo '</ul></div>';

            ?>
          </div><!-- /.carousel-inner -->
          <!-- Carousel nav -->
          <a class="left carousel-control" href="#thumb-carousel" data-slide="prev">
          <span class="glyphicon glyphicon-chevron-left"></span></a>
          <a class="right carousel-control" href="#thumb-carousel" data-slide="next">
          <span class="glyphicon glyphicon-chevron-right"></span></a>
          </div>
          <div id="carousel-index"></div><!-- data passed here by jQuery for debugging purposes -->
      </div><!-- /.container for the #thumb-carousel -->
  </div><!-- /.row this contains both carousels -->
    <?php
  } // The initial if statement

  else {
    return;
    }

} // End the function

When using the Roots framework, add the code to lib/custom.php.

The function can then be used in page templates. In this example, the function is added to the content-single-project.php template part:

<div class="row">
  <div class="col-md-12">
    <?php carawebs_carousel_images_slider(); ?>
  </div>
</div>

Controls

Controls are built into /js/_carousel.js:

/*==============================================================================
    Set up - give first items emphasis classes.
    Necessary because the slider is built dynamically
==============================================================================*/
   $(".carousel-inner .item:first").addClass("active");

   // Give active class to the first thumbnail image
   $("#thumb-carousel li:first").addClass("selected");

   // Give active class to the first thumbnail item (image-block)
   $("#thumb-carousel .item:first").addClass("active");

   // Prevent auto play, both carousels
   $('#myCarousel').carousel({ interval: false });
   $('#thumb-carousel').carousel({ interval: false });


/*==============================================================================
    Listen to the 'slid carousel' event to trigger code after each slide change
==============================================================================*/

  $('#myCarousel').on('slid.bs.carousel', function () {

    // Useful variables
    var carouselData = $(this).data('bs.carousel');
    var currentIndex = carouselData.getActiveIndex();
    var total = carouselData.$items.length;

    // To display a slide counter for debugging, enable this block.
    // Create a HTML element <div id="carousel-count"></div> to display the text.
    //var text = (currentIndex + 1) + " of " + total;
    //$('#carousel-count').text(text);

    // Highlight the thumbnail corresponding to the currently displayed slide in the main carousel
    $('[id^=carousel-selector-]').removeClass('selected');
    $('[id^=carousel-selector-'+(currentIndex)+']').addClass('selected');

/*==============================================================================
    Display the thumbnail slider item that contains the current myCarousel slide
    Enable this code block for a simple shift (no slide)
==============================================================================*/
    // Remove active class from #item-x
    //$('[id^=item-]').removeClass('active');
    // Which item contains the current thumbnail
    //$( "[id^=item-]" ).has("[id^=carousel-selector-"+(currentIndex)+"]").addClass( "active" );

/*==============================================================================
    SLIDE to the thumbnail slider item that contains the current myCarousel slide
    Enable this code block for a sliding shift
==============================================================================*/
    // Find the item in the thumb carousel that contains the currently displayed main image
    var targetItem = $( "[id^=item-]" ).has("[id^=carousel-selector-"+(currentIndex)+"]");

    // Find the index number of this item by getting the last character of the id string
    var id_selector = $(targetItem).attr("id");
    var id = id_selector.substr(id_selector.length -1);

    // Parse the string, return as an integer in base 10 format
    id = parseInt(id, 10);

    // Make a correction, because the first thumbnail carousel item has a count of zero
    id = (id -1);

    // Create a HTML element <div id="carousel-index"></div> to display the item index for debugging
    $('#carousel-index').text(id);

    // Force #thumb-carousel to slide to the correct item
    $('#thumb-carousel').carousel( id );

/*==============================================================================
    Hiding Prev & Next Links at extremes
==============================================================================*/
    checkitem = function() {
    var $this;
    $this = $("#myCarousel");
        if ($("#myCarousel .carousel-inner .item:first").hasClass("active")) {
          $this.children(".left").hide();
          $this.children(".right").show();
        } else if ($("#myCarousel .carousel-inner .item:last").hasClass("active")) {
          $this.children(".right").hide();
          $this.children(".left").show();
        } else {
          $this.children(".carousel-control").show();
        }
    };

  checkitem();

  $("#myCarousel").on("slid.bs.carousel", "", checkitem);


}); // Close the "slid" function

WordPress: Enqueue Scripts

When working within WordPress it’s important to enqueue scripts properly - this prevents inefficient double loading of libraries and ensures that scripts load rationally.

Within the Roots framework, the js files could be added to the assets/js folder and named with a leading underscore - they would then be concatenated into scripts.min.js. This isn’t a bad option - but it means that unecessary kilobytes will be added to pages that don’t contain the carousel.

Consider enqueueing the scripts only on the pages where they are required - in this case, the scripts are enqueued only on project CPT pages.

For better efficiency, the raw js files could be targeted by Grunt and uglified.

The enqueue script within the Roots framework are placed in lib/scripts.php in order to keep all script calls rational. In other setups the enqueue function could be added to the custom.php file.

function carawebs_carousel_control(){

	if ( is_singular( 'projects') ) { // only do this for "project" CPTs

    // Register the control script
    wp_register_script( 'carawebs_carousel', get_template_directory_uri() . '/assets/js/vendor/_carousel.js', array( 'jquery' ), null, true);
    wp_register_script('touchswipe', get_template_directory_uri() . '/assets/js/vendor/jquery.touchSwipe.min.js', array( 'jquery' ), null, true);
    wp_register_script('touchControl', get_template_directory_uri() . '/assets/js/vendor/touchControl.js', array( 'jquery' ), null, true);

	// Enqueue the carousel controls - they will be built into the footer
    wp_enqueue_script('carawebs_carousel');
    wp_enqueue_script('touchswipe');
    wp_enqueue_script('touchControl');
	}
}
// Add hooks for front-end
add_action('wp_enqueue_scripts', 'carawebs_carousel_control', 101);

Mobile Swipe Support

I plan to build in conditional logic so that touchSwipe is only loaded for mobile devices.

Useful Tutorials

http://www.sitepoint.com/creating-javascript-sliders-using-twitter-bootstrap-3/

Manipulating date format MySQL

Tue, 01 Jul 2014 09:38:44 +0000

Importing an “alien” date column into a WordPress format - necessary because the old date column contained data in the “VARCHAR” format. WordPress requires date to be in the “DATE” format.

Delete unnecessary columns.

Rename columns to suit WordPress schema.

Rename the old date column to “OLDDATE”.

Create a new column with the data-type “DATE”, named “post_date”:

Open phpMyAdmin
Select “Structure”
Add a new Column
Name appropriately (e.g. WP date = “post_date”)
Select Type “DATE”

Run the following SQL command:

UPDATE tablename SET post_date = str_to_date( OLDDATE, '%d/%m/%Y')

Where old dates are in string data-format, dd/mm/yyyy.

Manipulate Column Data

Copy data from ColumnA Data to ColumnB:

UPDATE tablename SET columnB = columnA

Delete table data: TRUNCATE tablename

Import Table Data, phpMyAdmin

Organise data on a test table, then rename before exporting:

Select table
Select operations
Select “rename table to”
Give the table the same name as the target table

Click “Export” in top level menu.

Open the target DB, select the target table. Click import and select the recently exported DB.

Responsive Images: Jekyll

Thu, 26 Jun 2014 12:54:44 +0000

How to add a responsive image to a Jekyll post:

Create a folder called assets in the root directory of your project, into which any images, downloads or any other assets are placed. They can then be linked to in the post.

Install jekyll-image-tag. This requires both Minimagick and Imagemagick to work.

You can install Minimagick with the command: sudo gem install jekyll-minimagick

In a plugin file within your projects _plugins directory add:

# _plugins/my-plugin.rb
require "jekyll-minimagick"

Define presets in your _config.yml file:

		image:
 		source: assets
 		 output: generated
  		presets:
   	 		users:
      		attr:
        	class: img-responsive
        	itemprop: image
      	width: 350
    	half:
    width: 500
    height: 500

Once you have the requirements installed, download jekyll-image-tag and copy image_tag.rb to your Jekyll _plugins folder.

Link the image from your assets folder to your post using the format:

{% image preset or WxH path/to/img.jpg attr="value" %}

My code looked like this:

{% image 300x300 Caribbean_reef_shark.jpg class="img-responsive" %}

Syncing with Bitbucket

Wed, 25 Jun 2014 00:00:00 +0100

Introduction

Bitbucket is a great resource - it provides free multi-user version control for groups of up to five developers.

It can also be a useful way of deploying to a development (or production) server.

The work flow is as follows:

Connect a local project directory with a remote Bitbucket repo
Clone the repo to a project directory on a remote server
Push changes from the local working directory to the Bitbucket remote repo
Pull changes from the remote Bitbucket repo to the remote development server

This allows all changes to be properly managed (for example, by means of git), and allows different developers to work on different branches. It has great tools for reviewing and merging changes.

It’s a great set-up for small teams, and is free.

Easy Pushing

This can be partially automated. Firstly, set up SSH keys to allow communication between:

Your local machine and Bitbucket
The remote server and Bitbucket

There are really good instructions for setting up SSH keys here.

Next, we’ll create an alias - “dir-push” that will manage the push to the repo.

We’ll put this alias in ~/.bashrc so that it is globally accessible.

Open ~/.bashrc and add the following lines:

#Push theme to Bitbucket
alias dir-push="cd /var/www/path/to/development/dir; add -A; git commit; git push origin master"

Close the terminal and reopen.

Now when you type “dir-push” on the command line, you’ll be prompted for a git commit message.

This is quite a painless way to manage the push to the repo - you just need to type a single command and your commit message.

I advise including a passphrase when you set up SSH keys - in case someone maliciously accesses your development machine - in which case you’ll also be prompted for the passphrase during the push.

Easy Pulling

The idea here is to connect to a remote server by means of SSH, and pull the latest version of the Bitbucket repo to the relevant directory on the development server. This is a bit more complicated.

It can be achieved easily with a GitHub repo, since you aren’t prompted for authentication when pulling. I’ve achieved this like so:

#!/bin/bash

# Pull the repo from GitHub to the development server
# The <<EOF part tells the shell that you're going to enter multi-lines until the ending EOF tag
ssh username@1.12.123.123 <<EOF
cd /var/www/path/to/development/directory
git pull
logout
EOF
echo "You should be logged out of the VPS"

This won’t work with Bitbucket - the required verification causes the script to stall.

The solution:

Create a shell script and add this to /usr/local/bin on the remote server
Create a shell script on the local machine (again, in /usr/local/bin) which starts an interactive SSH session and calls the remote script.

The Local Script

Create a suitably named file in the local /usr/local/bin:

sudo nano /usr/local/bin/pull-project

Add the following code:

#!/bin/bash
# Pull the Bitbucket repo to the development server
# Open an SSH session and run the shell script "remote-project-pull"
ssh username@1.12.123.123 -t "remote-project-pull"

The Remote Script

Create a suitably named file in the remote /usr/local/bin:

sudo nano /usr/local/bin/remote-project-pull

Add the following code:

#!/bin/bash
cd /var/www/path/to/development/directory
git pull
echo "End Script - logging out of VPS"
exit
echo "Logged out of VPS"

Now, when you’re working on your local development machine, you just enter the command “pull-project”. A SSH session will be opened, and the repo will be pulled from Bitbucket.

Again, if you have set up a passphrase for the SSH connection between the remote server and Bitbucket (recommended), you’ll be prompted for the passphrase.

Other Options

There are other options that automate the deployment from Bitbucket to the development server. This method looks interesting.

I can see how that would be pretty useful in some circumstances - but I think this method is simple, leaves you with lots of control, and it’s just a couple of extra commands in your workflow. You can commit to the Bitbucket repo without clients necessarily seeing your work-in-progress.

Custom Google Maps in Roots

Mon, 23 Jun 2014 00:00:00 +0100

Introduction

The Google maps JavaScript API v3 offers lots of opportunities to customise Google maps. The map can be styled to match the client website’s branding, retaining all the functionality of Google maps.

To display a custom Google map:

Define (style) a div to be used as a map canvas
Initialise Google Maps
Define the map – when working in WordPress consider saving the map as a *.js file and enqueueing this

It’s important that these three elements are added to the document head in the specified order.

The Google Styled Maps Wizard is a useful way of coding custom JSON data to be passed to your map, which is how the map gets it’s custom style. It’s not a particularly great way of creating an entire map but it’s useful for getting the right JSON data for particular element.

Map Script & File Locations

Save the custom map script:

theme/assets/js/vendor

Placing the map in this location means that it won’t be uglified by Grunt. This keeps the file out of the concatenation group, and allows us to control exactly when and where it is enqueued.

The Grunt task in Roots concatenates all js files into a single file: /assets/js/scripts.min.js. For the most part this works well, as it reduces HTTP requests and thereby gives a performance boost.

Sometimes we want to keep scripts out of the main js file - in this case, we only want to call the Google maps API and run our custom script on a single page. Therefore, the map script should be placed in the /assets/js/vendor directory.

Enqueuing Scripts in Roots

Within the Roots structure, enqueuing of scripts is controlled in the theme/lib/scripts.php file.

Scripts can be enqueued conditionally (for example only on certain pages, as in the example below). The order in which scripts are deployed can also be controlled.

In this file, define the map canvas:

function carawebs_map_style() {

	// Run on page 69 only
	if (is_page('69')) {
		echo '<style>#map_canvas { width: 100%; height: 548px; }</style>';
	}
}

Initialise the Google maps API on a specific page:

function carawebs_initialise_googlemaps(){

	if (is_page('69')) {
	    $googlescript = '<script src="https://maps.googleapis.com/maps/api/js?sensor=false"></script>';
	    echo $googlescript;
	}
}

Set up the map control script:

function carawebs_googlemaps_control(){

	if (is_page('69')) {

    // Register the control script - note script location
    wp_register_script( 'carawebs_googlemap', get_template_directory_uri() . '/assets/js/vendor/google-map.js');

		// Pass variables to the script - this avoids having to hardcode the map marker in javascript
		// This could be a user-selected field
		$imagelocation = get_template_directory_uri() . . '/assets/img/marker.png';
		$site_variables = array(
			'markerImage' => $imagelocation
			);

		wp_localize_script( 'carawebs_googlemap', 'carawebsMapVars', $site_variables );
		// This function makes variables available to carawebs_googlemap
		// Define the image in js like this: var image = (carawebsMapVars.markerImage);

	// Enqueue the map controls
    wp_enqueue_script('carawebs_googlemap');

	}
}

Insert scripts & style definitions in the correct order in wp_head:

// Add hooks for front-end
add_action('wp_head', 'carawebs_map_style', 1);
add_action('wp_head', 'carawebs_initialise_googlemaps', 2);
add_action('wp_enqueue_scripts', 'carawebs_googlemaps_control', 105);

The Entire Enqueue Script

/*================================================
* The Google Map Script
* ==============================================*/

/* Add Google Map Script to Specific Page */

function carawebs_map_style() {

	// Run on page 69 only
	if (is_page('69')) {
		echo '<style>#map_canvas { width: 100%; height: 548px; }</style>';
	}
}

function carawebs_initialise_googlemaps(){

	if (is_page('69')) {
	    $googlescript = '<script src="https://maps.googleapis.com/maps/api/js?sensor=false"></script>';
	    echo $googlescript;
	}
}

function carawebs_googlemaps_control(){

	if (is_page('69')) {

    // Register the control script
    wp_register_script( 'carawebs_googlemap', get_template_directory_uri() . '/assets/js/vendor/google-map.js');

		// Pass variables to the script - this avoids having to hardcode the map marker in javascript
		// This could be a user-selected field
		$imagelocation = get_template_directory_uri() . . '/assets/img/marker.png';
		$site_variables = array(
	    'markerImage' => $imagelocation
	    );

		wp_localize_script( 'carawebs_googlemap', 'carawebsMapVars', $site_variables );
		// This function makes variables available to carawebs_googlemap
		// Define the image in js like this: var image = (carawebsMapVars.markerImage);

		// Enqueue the map controls
    wp_enqueue_script('carawebs_googlemap');

	}
}


// Add hooks for front-end
add_action('wp_head', 'carawebs_map_style', 1);
add_action('wp_head', 'carawebs_initialise_googlemaps', 2);
add_action('wp_enqueue_scripts', 'carawebs_googlemaps_control', 105);

/*====================================================================*/

The Google Map Script

The following is a sample custom Google map, saved as /assets/js/vendor/google-map.js

var carawebsMap;
		function initialize() {
		var myOptions = {
			zoom: 9,
			center: new google.maps.LatLng(52.693032,-8.864808),
			mapTypeId: google.maps.MapTypeId.ROADMAP,
			scrollwheel: false,
			disableDefaultUI: false,
			styles:
        [
          {"featureType":"water",
					"stylers":[{"visibility":"on"},{"color":"#3bade2"}]},
          {"featureType":"landscape","stylers":[{"color":"#f2e5d4"}]},
          {"featureType":"road.highway","elementType":"geometry",
						"stylers":[{"color":"#777777"}]},
          {"featureType":"road.arterial","elementType":"geometry",
						"stylers":[{"color":"#c0c0c0"}]},//#e4d7c6
          {"featureType":"road.local","elementType":"geometry",
						"stylers":[{"color":"#fbfaf7"}]},
          {"featureType": "road.highway", "elementType": "labels",
						"stylers": [{ "visibility": "off" }]},
          {"featureType": "transit.line", "elementType": "labels.icon",
						"stylers": [{ "visibility": "off" }]},
          {"featureType":"poi.park","elementType":"geometry",
						"stylers":[{"color":"#c5dac6"}]},
          {"featureType":"administrative",
						"stylers":[{"visibility":"on"},{"lightness":33}]},{"featureType":"road"},
          {"featureType":"poi.park","elementType":"labels",
						"stylers":[{"visibility":"on"},{"lightness":20}]},{},
          {"featureType":"road","stylers":[{"lightness":20}]}]
        };

		carawebsMap = new google.maps.Map(document.getElementById("map_canvas"), myOptions);

		// Define an image to be used as a marker
		// var image set from wp_localize_script in the enqeueing function
		var image = (carawebsMapVars.markerImage);


		var centreLatLng = new google.maps.LatLng(52.693032,-8.864808);
        /* School Co-ordinates */
        var ennisLatLng = new google.maps.LatLng(52.847079,-8.987698); // Ennis Community College
        var ennistymon1LatLng = new google.maps.LatLng(52.941578,-9.289763); // Ennistymon CBS

    /*===============================
      Position the Markers
    ===============================*/
    var marker = new google.maps.Marker({
			position: ennisLatLng,
			icon: image,
			map: carawebsMap,
			title: 'Ennis Community College'
        });

    var marker2 = new google.maps.Marker({
			position: ennistymon1LatLng,
			icon: image,
			map: carawebsMap,
			title: 'Ennistymon CBS'
        });

      /*===============================
        Content Strings for Markers
      ===============================*/

      // Ennis Community College
        var ennisContentString =
        '<div id="content" style="color: #666;">'+
        '<h4 id="firstHeading" class="firstHeading">Ennis Community College</h4>'+
        '<div>'+
        '<p><b>Ennis Community College</b> is where we started<br>'+
        'our first School Cafe. We have been providing a service<br>' +
        'for the staff and students since 2009 -<br>'+
        'when we converted an old cupboard into a<br>'+
        'fully functioning school canteen.</p>'+
        '</div>'+
        '</div>';

      // Ennistymon CBS
			 	var ennistymon1ContentString = '<div id="content" style="color: #000;">'+
				 '<h4 id="firstHeading" class="firstHeading">Ennistymon CBS</h4>'+
				 '<div id="bodyContent">'+
				 '<p>Some information about Ennistymon CBS.</p>'+
				 '</div>'+
				 '</div>';

    /*===========================
      Set the InfoWindow
    ===========================*/

			var ennisInfowindow = new google.maps.InfoWindow({
                content: ennisContentString
            });

			var ennistymon1InfoWindow = new google.maps.InfoWindow({
                content: ennistymon1ContentString
            });


			/*===================================
        Click Events for Markers
      ===================================*/

            // Ennis Community College
            google.maps.event.addListener(marker, 'click', function() {
				ennisInfowindow.open(carawebsMap,marker); // Reference the info window and marker
            });

            // Ennistymon CBS
            google.maps.event.addListener(marker2, 'click', function() {
				ennistymon1InfoWindow.open(carawebsMap,marker2);
            });

		}

		google.maps.event.addDomListener(window, "load", initialize);
		google.maps.event.addDomListener(window, "resize", function() {
			 var center = carawebsMap.getCenter();
			 google.maps.event.trigger(carawebsMap, "resize");
			 carawebsMap.setCenter(center);
        });

Typical Responsive Breakpoints

Thu, 19 Jun 2014 00:00:00 +0100

Extra Small Screens/Phone

480px and below

Small Screen/Tablet

481px to 768px

Medium Screen/Desktop

769px to 992px

Large Screen/Wide Desktop

993px to 1200px

XL Desktop

wider than 1200px

Closely based on Twitter Bootstrap 3.x breakpoints.

These breakpoints need not be totally fixed - but they’re a good guide to basic responsive layouts. In particular, an extra breakpoint might be needed at 320px and below. Or alternatively, you could go fully responsive - use inline-block to define columns and make everything percentage based, changing proportions as and when needed based on your design!

WordPress CLI

Mon, 09 Jun 2014 00:00:00 +0100

The WordPress CLI tool allows you to deploy WordPress and add themes & plugins from the command line.

Installation

Change to user that will own the WP install:

root@server:~# su username

Give proper ownership to a suitable directory and move into it:

sudo chown username -R var/www/site.com/public_html
cd /var/www/site.com/public_html

Install WP-cli:

sudo curl -L https://raw.github.com/wp-cli/builds/gh-pages/phar/wp-cli.phar > wp-cli.phar

Make executable, move to make available across system:

chmod +x wp-cli.phar
sudo mv wp-cli.phar /usr/bin/wp

Get info about wp-cli to check the installation worked:

wp --info

Install WordPress Using WP-CLI

Create a new directory called “testsite”:

mkdir testsite

Give proper ownership:

sudo chown -R user:www-data /var/www/testsite

Download WP core:

wp core download

Create a MySQL Database:

mysql -u root -pROOTPASSWORD -e "create database db_name; GRANT ALL PRIVILEGES ON db_name.* TO new_user@localhost IDENTIFIED BY '12345'"

Configure WordPress to use it:

wp core config --dbname=db_name --dbuser=new_user --dbpass=1234567 --dbhost=localhost

Install WordPress:

wp core install --url=http://localhost/testsite --title="Basic Test Site" --admin_user=USERNAME --admin_password=STRONGPASSWORD --admin_email=USER@WEBSITE.COM

Using WP-CLI Deploy Plugin

Install WP-cli Deploy plugin in relevant WP directory:

git clone https://github.com/c10b10/wp-cli-deploy.git

Create a wp-cli.yml file in the root of you WordPress directory and add this line to the file:

require: "relative/path/to/deploy.php"

For example:

require: "wp-cli-deploy/deploy.php"

Enter wp help deploy to check it works.

To use WP-cli deploy, you need to define constants in wp-config.php.

For example, set up deploy for the “dev” environment:

    define( 'DEV_URL', 'testsite.com' );
    define( 'DEV_WP_PATH', '/var/www/testsite.com/public_html/' );
    define( 'DEV_HOST', '12.34.567.890' );
    define( 'DEV_USER', 'remote-server-username' );
    define( 'DEV_PORT', '22' );
    define( 'DEV_PATH', '/var/www/testsite.com/public_html' ); # Path to remote site
    define( 'DEV_UPLOADS_PATH', '/var/www/testsite.com/public_html/wp-content/uploads' );
    define( 'DEV_THEMES_PATH', '/var/www/testsite.com/public_html/wp-content/themes' );
    define( 'DEV_PLUGINS_PATH', '/var/www/testsite.com/public_html/wp-content/plugins' );
    define( 'DEV_DB_HOST', 'localhost' );
    define( 'DEV_DB_NAME', 'databaseName' );
    define( 'DEV_DB_USER', 'databaseUser' );
    define( 'DEV_DB_PASSWORD', '1234567goodpassword' );

Deploy Database and uploads to dev environment:

wp deploy push dev --what=db && wp deploy push dev --what=uploads

Necessary Local Ownership Local

Set permissions: allow rwx for user & group, r for others

Remote Ownership: Example drwxr-xr-x 5 USERNAME www-data 4096 May 5 17:20 testsite

Set Up WP Development Environment

Mon, 09 Jun 2014 00:00:00 +0100

This will create a test site here:

/var/www/testdomain.com/public_html/newsite

The site will be visible at testdomain.com/newsite

Create a Directory

Open a terminal on the VPS by means of SSH.

Navigate to the test domain document root:

cd /var/www/testdomain.com/public_html

Create a new directory:

sudo mkdir newsite

Set Ownership & Permissions

Grant ownership of the new directory to the proper user. Don’t leave the directory owned by root.

sudo chown -R $USER:www-data /var/www/testdomain.com/public_html/newsite

Set permissions so files can be viewed:

sudo chmod -R 755 /var/www/testdomain.com/public_html/newsite

WordPress Settings

Set Directory Permissions to 755:

find /var/www/domain.com/path-to-wp -type d -exec chmod 755 {} \;

Set File Permissions to 644:

find /var/www/path-to-wp -type f -exec chmod 644 {} \;

Note that permissions may need to be more permissive to allow uploading of files, depending on the server environment:

Any file that needs write access from WordPress should be owned or group-owned by the user account used by the WordPress (which may be different than the server account). For example, you may have a user account that lets you FTP files back and forth to your server, but your server itself may run using a separate user, in a separate usergroup, such as dhapache or nobody. If WordPress is running as the FTP account, that account needs to have write access, i.e., be the owner of the files, or belong to a group that has write access. In the latter case, that would mean permissions are set more permissively than default (for example, 775 rather than 755 for folders, and 664 instead of 644).

From WordPress Codex.

Create a MySQL Database with new user & password:

mysql -u root -pROOTPASSWORD -e "create database db_name; GRANT ALL PRIVILEGES ON db_name.* TO new_user@localhost IDENTIFIED BY 'NewUserPassword'"

Entering the root password is optional and may be inadvisable. Leave it out, and the system will prompt for Root password in normal way before carrying out the mysql command.

Install WordPress

Install WordPress using WP-CLI

Set Aliases

Open ~/.bashrc and set up aliases:

#####################################################
# Create New Project Site Deploy Aliases
#####################################################

#Push new project database
alias new-project-pushdb="cd /var/www/local-site; wp deploy push dev --what=db"

#Push new project plugins
alias new-project-pushplug="cd /var/www/local-site; wp deploy push dev --what=plugins"

#Push new project uploads
alias new-project-pushuploads="cd /var/www/local-site; wp deploy push dev --what=uploads"

#Push new project db and uploads
alias new-project-pushdbuploads="cd /var/www/local-site; wp deploy push dev --what=db && wp deploy push dev --what=uploads"

#Pull new project db and uploads folder
alias new-project-pullcontent="cd /var/www/local-site; wp deploy pull dev --what=db && wp deploy pull dev --what=uploads"

####### End new project ##############################

Linux Cheat Sheet

Mon, 09 Jun 2014 00:00:00 +0100

Introduction

This cheatsheet contains various terminal commands for Ubuntu, but they should work for any debian based Linux flavour. Maintained by @DavidCWebs - get in touch with me via Twitter.

General Commands

Change Directory	Move up one Directory	Move to Root
`cd /path/to/dir`	`cd..`	`cd /`

Move to Users Home	Show the directory you’re in	Find file whose name starts with “file”, starting from root directory
`cd ~`	`pwd`	`find / -name 'file*'`

Excellent tool for explaining commands:

Explainshell

Finding Stuff

Use grep to find a string recursively in a directory, output to a file:

grep -r "carawebs_download_link_repeater()" /var/www/thinkup/wp-content/themes/thinkup > thinkup-grep

Output File Info Recursively

Very handy for /wp-content/uploads to track down large files.

find /target-parent-dir -type f -follow -print|xargs ls -l > report

Get System Info

# Output IP address and other useful bits of info:
ifconfig | grep inet

# Output hostname:
hostnamectl

Managing Files & Directories

Copy an entire directory and it’s contents, including sub directories & files:

sudo cp -r /path/to/source /path/to/destination

Delete Directory	Delete directory and contents with no warnings	Move file to destination directory
`rm -r dirname`	`rm -rf dirname`	`mv file.txt destination-dir`

List all files in a directory	Write files & directories list to a file	Include hidden files and directories, preceded with “.”
`ls -l`	`ls -LR >> list.txt`	`ls -a`

List contents of parent directory	Print directory contents to print.txt	Remove file1 in current directory
`ls ../`	`ls > print.txt`	`rm file1`

Tree is a directory listing programme that displays files and folders recursively in a tree structure.

Install Tree	Write files/dirs in tree format	Tree format with file permissions
`sudo apt-get install tree`	`tree >> inventory.txt`	`tree -p >> inventory.txt`

Rename file extensions:

rename 's/\.foo$/\.bar/' * SPECIFIC EXAMPLE: rename 's/\.gddoc$/\.txt/' *

Managing Groups

List all groups on system	Create new group	Add user www-data to group
`cat /etc/group`	`sudo groupadd newgrp`	`sudo adduser www-data newgrp`

BASH Scripts

Make /file.sh executable:

sudo chmod +x /file.sh

Move to /usr/local/bin, execute in terminal with file.sh

sudo mv file.sh /usr/local/bin

Managing Users

Add User to Sudoers:

sudo adduser <username> sudo

Create user with password and home folder:

sudo useradd -d /home/testuser -m testuser sudo passwd testuser

Determine File/Directory Size

The du command summarises “Disk Usage”:

By default, the Single Unix Specification (SUS) specifies that du is to display the file space allocated to each file and directory contained in the current directory. Links will be displayed as the size of the link file, not what is being linked to; the size of the content of directories is displayed, as expected.

As du reports allocation space and not absolute file space, the amount of space on a file system shown by du may vary from that shown by df if files have been deleted but their blocks not yet freed. Also the minfree setting that allocates datablocks for the filesystem and the super user processes creates a discrepancy between total blocks and the sum of used and available blocks. The minfree setting is usually set to about 5% of the total filesystem size.

du wikipedia article.

To return directory size (not individually listed content size) in human-readable format (e.g. 3.9G):

du -hs /path/to/directory

du takes a single argument, specifying a pathname. If no pathname is specified, the current directory is used. du options:

-a In addition to the default output, include information for each non-directory entry
-c display a grand total of the disk usage found by the other arguments
-d #, the depth at which summing should occur. -d 0 sums at the current level, -d 1 sums at the subdirectory, -d 2 at sub-subdirectories, etc.
-H calculate disk usage for link references specified on the command line
-k show sizes as multiples of 1024 bytes, not 512-byte
-L calculate disk usage for link references anywhere
-s report only the sum of the usage in the current directory, not for each file
-x only traverse files and directories on the device on which the pathname argument is specified.

Other Unix and Unix-like operating systems may add extra options - BSD and GNU du specify a -h option, displaying disk usage in a format easier to read by the user (e.g. 10 MB).

WordPress Settings

Set Directory Permissions to 755:

find /var/www/domain.com/path-to-wp -type d -exec chmod 755 {} \;

Set File Permissions to 644:

find /var/www/path-to-wp -type f -exec chmod 644 {} \;

Note that permissions may need to be more permissive to allow uploading of files, depending on the server environment:

“Any file that needs write access from WordPress should be owned or group-owned by the user account used by the WordPress (which may be different than the server account). For example, you may have a user account that lets you FTP files back and forth to your server, but your server itself may run using a separate user, in a separate usergroup, such as dhapache or nobody. If WordPress is running as the FTP account, that account needs to have write access, i.e., be the owner of the files, or belong to a group that has write access. In the latter case, that would mean permissions are set more permissively than default (for example, 775 rather than 755 for folders, and 664 instead of 644).” From WordPress Codex.

Create a MySQL Database with new user & password:

mysql -u root -pROOTPASSWORD -e "create database db_name; GRANT ALL PRIVILEGES ON db_name.* TO new_user@localhost IDENTIFIED BY 'NewUserPassword'"

Entering root password is optional and may be inadvisable. Leave it out, and the system will prompt for Root password in normal way before carrying out the mysql command.

Sync to Remote Server

This is probably better than scp - since it only transfers the differences between two sets of files. Assumes that SSH has been set up between local and remote.

rsync -az /path/to/source username@host:/path/to/destination

Remote Server with Non-Standard SSH Port

If connecting on a non-standard SSH port (which you should be doing), set the port in the rsync command:

-e "ssh -p 1234" where “1234” is the SSH port number.

-e is the short form of rsh, which specifies the remote shell to use.

Sample rsync command for SSH port 1234:

cmd: 'rsync --progress -a -v -rz --checksum --delete -e "ssh -p 1234" _site/ username@12.23.567.89:/var/www/path/to/deploy'

Rsync Commands

May need to specify SSH:

rsync -az -e ssh /path/to/source username@host:/path/to/destination

Copy the contents of mu-plugins from a dev site to a local copy: rsync -raz -e "ssh -p 1234" username@1.123.123.1234:/var/www/remote.com/public_html/subdirectory/wp-content/mu-plugins/ /var/www/local-copy/wp-content/mu-plugins

HTML version of rsync man page: http://rsync.samba.org/ftp/rsync/rsync.html

Common options used with rsync commands:

-v : verbose
-r : copies data recursively (but don’t preserve timestamps and permission while transferring data
-a : archive mode - preserves almost everything - copies files recursively, preserves symbolic links, file permissions, user & group ownerships and timestamps
-z : compress file data
-h : human-readable, output numbers in a human-readable format
–delete : delete files on the destination that do not exist in source
–checksum : use checksum instead of timestamps to check whether the directory/site has changed

Exclude files from an rsync with --exclude. To exclude several different types of files, use --exclude more than once:

rsync -avP --exclude='*.zip' --exclude='*.gz'

Alternatively, use an exclude file: list each file/directory on a separate line.

Good description of rsync flags: http://lesterchan.net/blog/2011/07/15/rsync- to-dropbox-jungle-disk/

Good description of rsync & Jekyll: http://www.garron.me/en/blog/using-rsync-with-jekyll.html

Rsync from Remote Server

Copy directories recursively:

david@david-desktop:~$ rsync -raz username@12.123.1234.1234:/var/www/ ~/carawebs-server-backups

Combine this with a MYSQL backup:

mysqldump -u user -ppassword --all-databases > all_databases.sql

Restore from this:

mysql -u user -ppassword < all_databases.sql

Secure Copy to/from Remote Server

Copy example.txt from remote machine to local:

scp your_username@123.12.12.123:example.txt /local/directory

Note for the LOCAL machine!

SCP with Non-Standard Port Number

Copy example.html from local to remote, on a non-standard port:

scp -P 1234 ~/example.html username@123.123.12.123:/var/www/html/public_html

Note: to specify port number, SCP needs an uppercase “P” - in contrast to rsync.

Add Font Awesome to Roots

Mon, 09 Jun 2014 00:00:00 +0100

Font-awesome is a great way to add iconography to a website.

Copy Font Awesome font files to /assets/fonts.
Make a new directory called “font-awesome” in assets/less.
Add Font Awesome LESS files to the directory assets/less/font-awesome.

The file assets/less/font-awesome/font-awesome.less imports all other files.

Check assets/less/font-awesome/variables.less references the font files properly – relative to the compiled CSS file.

Open assets/less/app.less and add:

@import "font-awesome/font-awesome.less";

In this way, the font-awesome rules are brought into the main less file that will be compiled.

Open Gruntfile.js and add assets/less/font-awesome/*.less to the “watch” task.

If any watched less files change, the css will be recompiled.

Custom Sidebars in Roots

Mon, 09 Jun 2014 00:00:00 +0100

Custom Sidebars

<?php
/**
 * Register sidebars and widget Areas
 */
function roots_widgets_init() {
  // Primary Sidebar
  register_sidebar(array(
    'name'          => __('Primary', 'roots'),
    'id'            => 'sidebar-primary',
    'before_widget' => '<section class="widget %1$s %2$s">',
    'after_widget'  => '</section>',
    'before_title'  => '<h3>',
    'after_title'   => '</h3>',
  ));

  //Secondary Sidebar for courses CPTs
  register_sidebar(array(
    'name'          => __('Course Sidebar', 'roots'),
    'id'            => 'sidebar-courses',
    'before_widget' => '<section class="widget %1$s %2$s">',
    'after_widget'  => '</section>',
    'before_title'  => '<h3>',
    'after_title'   => '</h3>',
  ));

  //Footer
  register_sidebar(array(
    'name'          => __('Footer', 'roots'),
    'id'            => 'sidebar-footer',
    'before_widget' => '<section class="widget %1$s %2$s">',
    'after_widget'  => '</section>',
    'before_title'  => '<h3>',
    'after_title'   => '</h3>',
  ));

  // Widgets
  register_widget('Roots_Vcard_Widget');
}
add_action('widgets_init', 'roots_widgets_init');

Amend templates/sidebar.php so that the custom sidebar is delivered for the appropriate content types. In this example, the custom sidebar id “sidebar-courses” is associated with the sfwd-courses CPT:

<?php  
if ( is_singular('sfwd-courses') ) :
  dynamic_sidebar('sidebar-courses');
else:
  ?><div class="bottom-pad"><?php get_search_form(); ?></div><?php
  dynamic_sidebar('sidebar-primary');
endif;  
?>
<?php dynamic_sidebar('sidebar-secondary'); ?>

Download A Website Using wget

Sun, 08 Jun 2014 00:00:00 +0100

Use the wget command line utility to download an entire website.

Be careful with recursive retrieval - you might download the entire internet!

wget \
     --recursive \
     --no-clobber \
     --page-requisites \
     --html-extension \
     --convert-links \
     --domains testsite.com \
     --no-parent \
         http://testsite.com/

–recursive: downloads entire site
–no-clobber: doesn’t overwrite files, useful for interrupted downloads
–page-requisites: download all the files required to display the page (CSS, images etc)
–html-extension: save files with extension HTML
–convert-links: make links relative so they work off-line
–domains: Set domains to be followed
–no-parent: Don’t ascend to the parent directory when retrieving recursively - guarantees that only the files below a certain hierarchy will be downloaded

Alternative Method

This can sometimes works better:

wget --wait=20 --limit-rate=20K -r -p -U Mozilla http://www.testsite.com

Friendlier on the target website and avoids getting blocked.

Note: We use this primarily for downloading our own or our client’s CMS based sites in “flat” HTML - so we’re only hitting our own site resources, or we’re downloading with permission.

If you’re using this method to download other people’s websites, be responsible.

See wget man page for more details.

Second Dropbox on Ubuntu 14.04

Tue, 03 Jun 2014 09:38:44 +0000

Create two instances of Dropbox on one machine. This will allow a separate Dropbox to be run, which can help keep managed client backups rational.

Setup Directories

mkdir .name-dropbox: This will contain Dropbox configuration settings.

mkdir name-dropbox: This will contain the Dropbox folder.

Open a terminal and enter:

HOME=$HOME/.name-dropbox /usr/bin/dropbox start -i

This will start the Dropbox setup window. Select a customised location for Dropbox by ticking “I want to choose where to put my Dropbox folder”, which should be located in /name-dropbox.

Make a Control Script

To control the second Dropbox, create a BASH script.

sudo nano ~/DropboxAltStarter.sh

Paste:

#!/bin/bash
HOME=$HOME/.name-dropbox /usr/bin/dropbox start

Make this script executable:

chmod +x ~/DropboxAltStarter.sh

Start the Second Dropbox on Startup

Type “Start” into Ubuntu dash, access “Startup Applications”.

Create a new application with an appropriate name.

Paste the following as a command, replace “Username”:

/home/Username/DropboxAltStarter.sh

Jekyll Push to gh-pages Branch

Tue, 03 Jun 2014 00:00:00 +0100

Make a sub-directory that contains a Jekyll site available at the gh-pages branch of a GitHub repo.

The site will then be available at http://USERNAME.github.io/REPONAME/

To begin with, set up an empty Repo. Don’t add a README.md when prompted.

Initialise git in the local project directory, and connect to the new repo.

cd project-directory
git init
git remote add origin https://github.com/USERNAME/REPONAME.git

Initial Subtree Commit

The local Jekyll site is built to the _site directory.

We need to tell git about the sub-directory and make an initial commit:

git add _site && git commit -m "Initial _site subtree commit"

Push to gh-pages

Send _site to the gh-pages branch on GitHub

git subtree push --prefix _site origin gh-pages

Push Ongoing Changes

Push all changes to the master branch:

git add -A

git commit -m "commit message"

Push changes to the gh-pages branch:

git subtree push --prefix _site origin gh-pages

Using the Web Icons

Thu, 19 Jan 2012 00:00:00 +0000

This template uses Typicons web font, provided by Fontello font bundling service. It allows you to quickly add nice icons into your pages by using css tags.

To add an icon somewhere in the template simply do:

<i class="icon-home"></i>

This will insert a home icon, just as the one seen in the sidebar. The available class names you can use are as follows:

These should work in all the browsers, all the way down to and including IE7, but not IE6.