Many software projects now use TOR hidden services to bypass local NAT firewalls or to make their self-hosted, at-home services available remotely without a static, public IP address. There are also privacy benefits for both service operators and users.

This article assumes that you already have the TOR hidden service properly configured and everything is running smoothly on the service side. So how do you connect to a TOR hidden service remotely using nodejs? It's actually pretty simple. We will use TOR's socks proxy, a socks proxy agent module, and the built-in http/s modules of nodejs.

Install TOR

First you will need to install TOR. On debian/Ubuntu you can use the following command:

sudo apt-get install tor

This should automatically setup a service to run the TOR proxy on system boot.

For other systems have a look at the official website for installation guides.

Verify that the TOR proxy is working:

curl --verbose \
    --socks5 127.0.0.1:9050 \
    --socks5-hostname 127.0.0.1:9050 \
    -s https://check.torproject.org/ | cat | grep -m 1 Congratulations | xargs

If your TOR proxy is working then you should see "Congratulations" printed to your terminal.

Nodejs script to fetch web page from TOR hidden service

Below is an example script that can fetch a web page served via a TOR hidden service:

const url = require('url');
const http = require('http');
const https = require('https');
const SocksProxyAgent = require('socks-proxy-agent');

// Use the SOCKS_PROXY env var if using a custom bind address or port for your TOR proxy:
const proxy = process.env.SOCKS_PROXY || 'socks5h://127.0.0.1:9050';
console.log('Using proxy server %j', proxy);
// The default HTTP endpoint here is DuckDuckGo's v3 onion address:
const endpoint = process.argv[2] || 'https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion';
console.log('Attempting to GET %j', endpoint);
// Prepare options for the http/s module by parsing the endpoint URL:
let options = url.parse(endpoint);
const agent = new SocksProxyAgent(proxy);
// Here we pass the socks proxy agent to the http/s module:
options.agent = agent;
// Depending on the endpoint's protocol, we use http or https module:
const httpOrHttps = options.protocol === 'https:' ? https : http;
// Make an HTTP GET request:
httpOrHttps.get(options, res => {
    // Print headers on response:
    console.log('Response received', res.headers);
    // Pipe response body to output stream:
    res.pipe(process.stdout);
});

Install the socks-proxy-agent module via npm:

npm install socks-proxy-agent

Create a new file:

touch ./tor-http-fetch.js

Copy/paste the above script into the new file.

Run the script as follows:

node ./tor-http-fetch.js

By default it will fetch DuckDuckGo's home page via its v3 onion address. You can use the script to fetch another page instead:

node ./tor-http-fetch.js "http://your-tor-hidden-service.onion"

The script will work with both "http" and "https" URLs.

If you need to send custom headers or otherwise change the HTTP request to better suit your needs, have a look at the nodejs http module docs.

Are you still using a static bitcoin address to receive donations? You may want to stop that. Anyone who scans your donation QR code can see exactly how much money you've received and the full transaction history of your address. These days there is a better way to receive donations. With the Lightning Network and the lnurl-pay protocol it is possible for your supporters to send you bitcoin privately, instantly, and with low fees.

Example Street Art

In the example shown here, we can see a static QR code embedded in street art. This QR code contains a bitcoin address: "3Pbo...c4Grb". It is easy to look-up the transaction history for this address using a block explorer.

As of the time this article was written, the address has received over 100 transactions totaling about 0.17 BTC. About half of the total amount received is still sitting in the address waiting to be spent. It's also possible to follow the trail backwards to see where all these donations came from and forwards to where the funds are sent.

If you care about your own financial privacy or that of your supporters, you should learn how to use the Lightning Network to greatly improve the privacy of your bitcoin payments.

Static Donation QR Codes with Lightning Network

The Lightning Network allows instant, low-fee, and private bitcoin payments. This is possible because the bitcoin transactions that are exchanged between counter-parties on the Lightning Network are not broadcast or stored on the blockchain. Only in cases of final settlement is it necessary to pay the miner fee for a bitcoin transaction to be included in a block. However, there are down-sides to the Lightning Network.

One problem of the Lightning Network is that it doesn't use bitcoin addresses for receiving payments. Instead a new concept called invoices are used to request payments. These invoices are one-time use and are generated for an exact amount of bitcoin. So this means it's not possible to print an invoice as a QR code and use indefinitely.

This is where the lnurl-pay protocol comes in. It's a side-channel protocol to help facilitate a nicer end-user experience while using Lightning Network for everyday payments.

A wallet application that supports lnurl-pay will decode the bech32 text and make an HTTP request to the URL. The full lnurl-pay UX flow is as follows:

1. User opens a mobile wallet app that supports lnurl-pay
   • Find a list of wallet apps here
2. User uses the app to scan the QR code
3. App decodes the QR code to get the URL
4. App makes an HTTP request to the URL
5. Web service replies with the lnurl-pay response data, which includes:
   • Meta data about the lnurl-pay
   • Minimum and maximum payment amount in msats (millisatoshis)
   • Another URL to which the app will send a second HTTP request
6. App shows the information above to the user
7. User chooses the amount to pay and confirms
8. App sends the second HTTP request with the amount to pay
9. Web service replies with a Lightning invoice
10. App pays the invoice

It might look like a lot of steps, but to the user of the app it's only two steps:

1. Scan the QR code with their wallet app
2. Choose payment amount and confirm

The following services provide custodial Lightning accounts that support lnurl-pay:

• lnbits.com - Provides custodial Lightning wallet accounts with many extensions like "LNURLp" which allows the creation of re-usable lnurl-pay links.
• coinos.io - Another custodial Lightning wallet provider

Lightning Addresses Are Even Easier

The new Lightning Address protocol makes it even easier to receive Lightning payments. There are already several services that offer Lightning addresses as a service to their users:

• coinos - A custodial Lightning service provider
• lntxbot - A telegram bot that provides a custodial Lightning wallet to Telegram users
• LightningTipBot - Another telegram bot for Lightning

Once you've setup a Lightning wallet account with one of the above services, you can get your own Lightning address. Your address will follow the "you@service" pattern. So for lntxbot it would be "you@lntxbot.com" or for coinos it would be "you@coinos.io".

This style of address has a similar look and feel as an email address but instead of sending email, other users can send you bitcoin via the Lightning Network.

Under-the-hood, the Lightning Address protocol uses the lnurl-pay protocol. The real difference is in the aesthetics of the text that is shared. Compared to the long URLs that are typical of lnurl-pay, a Lightning Address is much simpler. If you'd like to read more about how this new protocol works you can find details in its GitHub repository.

How to Generate QR Codes

It's possible to generate QR codes from text in many ways. For example, you could do it using the qr command-line utility:

echo -n "hello" | qr

This will print a QR code in your terminal.

You can also save the output of the qr command to a file:

echo -n "hello" | qr > hello.png

If you prefer to use a web-based QR code generator, you can try the following:

• www.the-qrcode-generator.com

Are you running a firewall like ufw with docker? You might be surprised to learn that your firewall is probably not doing anything to block unwanted internet traffic from reaching your docker services. Docker modifies iptables rules to completely bypass or ignore the rules set by ufw. In this article, I will explain how to check if the services running on your server are exposed and how to protect them.

Check for exposed docker services

I usually begin articles, like this one, by explaining some history or back-story to provide context. But in this case, let's dive right into how to check if your services are exposed remotely.

In this section we will use netstat and nmap to check for local processes that are listening for TCP connections and to scan ports. To install them:

sudo apt-get install net-tools nmap

Use netstat to print a list of processes that are actively listening for TCP connections:

sudo netstat -tlpn

Example results:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:8332          0.0.0.0:*               LISTEN      17021/docker-proxy  
tcp        0      0 127.0.0.1:8333          0.0.0.0:*               LISTEN      17146/docker-proxy  
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      17330/docker-proxy  
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      651/sshd            

From the above results, we can see that we have 4 services listening for TCP connections. "Local Address" refers to the host (IP address and port number) on which the service is listening. For example, requests to "127.0.0.1:8332" will be handled by that service.

• "127.0.0.1" is the loopback address. Services bound to the loopback address are not accessible remotely.
• "0.0.0.0" means all interfaces. Services bound to this address are accessible remotely unless a firewall is blocking those requests.

Let's check this assumption by using nmap to scan for open ports:

nmap -p 0-65535 0.0.0.0

Example results:

Starting Nmap 7.60 ( https://nmap.org ) at 2021-08-16 16:00 UTC
Nmap scan report for 0.0.0.0
Host is up (0.010s latency).
Not shown: 65535 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
8332/tcp  open  unknown
8333/tcp  open  bitcoin
5432/tcp  open  postgresql

Nmap done: 1 IP address (1 host up) scanned in 3.15 seconds

Here we see that all 4 service ports are open on any interface. But this doesn't tell us what we really want to know - are these ports exposed remotely?

To answer that, we need the system's LAN IP address. You can get this by using ifconfig:

ifconfig | grep -Po "inet 192.168.[^ ]+" | grep -Po "192.168.[^ ]+"

If your system is a VPS, running in a cloud, then its LAN IP address might begin with "10." instead of "192.168.". Check the full output of ifconfig to view all of your system's networking interfaces.

Now let's repeat the scan with the LAN IP address:

nmap -p 0-65535 192.168.XXX.XXX

Example results:

Starting Nmap 7.60 ( https://nmap.org ) at 2021-08-16 16:00 UTC
Nmap scan report for 192.168.XXX.XXX
Host is up (0.010s latency).
Not shown: 65535 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
5432/tcp  open  postgresql

Nmap done: 1 IP address (1 host up) scanned in 3.31 seconds

From the above we can see that the system has two ports open for remote TCP traffic. The first is port 22 which is used for SSH access. If we need to access the machine remotely via SSH, then this port should stay open.

The second port is for postgreSQL, which very likely should not be exposed remotely. In this example server, we're running postgreSQL in a docker container.

So what gives? Why is docker exposing this service remotely? The answer is because you told it to. Now let's fix it.

The Fix: Don't expose docker services remotely

Sounds simple, right?

Most users of docker don't realize that they are exposing their services remotely when they publish ports. For example, this command creates and runs a docker container:

docker run -p 3000:3000 <image>

The -p argument tells docker to "publish" port 3000 - i.e. create a listener and forward requests to port 3000 to the new container. But this is insecure because the default host that docker binds to is "0.0.0.0"!

These kinds of examples are all over the internet in tutorials, how-to articles, GitHub issues, stackoverflow answers, and more. Users have been trained to use docker in an insecure way.

The fix is actually quite simple. When publishing ports, tell docker to bind to "127.0.0.1" instead:

docker run -p 127.0.0.1:3000:3000 <image>

Now the service will not be exposed remotely.

You can find more details about using the -p, --publish arguments in the official documentation.

The docker + ufw problem: Unintuitive defaults

Based on the many articles, bugs, and issues about this common problem, it's safe to say that docker's default behavior is far from intuitive. One could even call the default behavior dangerous.

From an old issue which remains un-fixed as of today:

Docker Network bypasses Firewall, no option to disable

Steps to reproduce the issue:

1. Setup the system with a locked down firewall
2. Create a set of docker containers with exposed ports
3. Check the firewall; docker will by use "anywhere" as the source, thereby all containers are exposed to the public.

And the problem has recently attracted the attention of hackernews:

Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

NewsBlur's founder here. I'll attempt to explain what's happening.

...

It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models).

...

Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn't work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world.

So what's going on here? Why is ufw ineffective at blocking traffic to services run inside docker containers?

Docker inserts its own iptables rules, which bypass ufw's own iptables rules. So those ufw rules that you think are protecting your docker services, are not actually doing that.

If you're curious, you can print your system's iptables rules with the following command:

sudo iptables -S

You will notice both ufw and docker have inserted their own rules.

I don't claim to understand the deep, dark magic of iptables. So I won't even begin to try to explain it here.

What about --iptables=false?

The most popular solution to the docker + ufw problem is to configure the docker daemon with --iptables=false. This is a bad idea because it makes docker unusable by blocking out-bound traffic as well as any networking between containers. So if you want docker to function properly, you will need to create and manage iptables rules manually. That doesn't sound like a long-term viable solution.

If you're really interested, you can have a look at the proposed solution that can be found in this stackoverflow answer. It looks like a lot of effort for not a lot of benefit. The much simpler solution is to just not expose your services. Bind to "127.0.0.1" when publishing your service ports.

Are non-docker services exposed as well?

The big question that might be on your mind. Are the services running outside of docker exposed too? Luckily, the answer appears to be no. You can verify this yourself. Create a TCP listener on port 12345 using netcat:

nc -l -k -p 12345

Leave this running and open a new, separate terminal window. If your ufw is enabled, then port 12345 should be blocked by default. Use nmap to perform a port scan on the individual port:

nmap -p 12345 192.168.XXX.XXX

Results:

Starting Nmap 7.70 ( https://nmap.org ) at 2021-08-16 16:00 UTC
Nmap scan report for 192.168.XXX.XXX
Host is up (0.00031s latency).

PORT      STATE SERVICE
12345/tcp open  netbus

Nmap done: 1 IP address (1 host up) scanned in 0.21 seconds

Whoops! Looks like ufw didn't block the port scan. Maybe it's because we're running the port scan locally? Let's try remotely. Run the following command from a different computer that's connected to the same LAN (router/wifi):

nmap -p 12345 192.168.XXX.XXX

Results:

Starting Nmap 7.60 ( https://nmap.org ) at 2021-08-16 16:00 UTC
Nmap scan report for 192.168.XXX.XXX
Host is up.

PORT      STATE    SERVICE
12345/tcp filtered netbus

Nmap done: 1 IP address (1 host up) scanned in 2.01 seconds

Phew! Looks like ufw is doing its job - at least for non-dockerized services.

Defense in-depth

Keep using ufw to protect your systems. But you shouldn't be relying on a single line of defense to protect your services. If ufw was the only thing standing between your services and the public internet, then that was a mistake.

Use the networking tools above to check for exposed services on your systems.

If your system is a VPS at a cloud provider, then you should look at what firewall options they have available. In the case of DigitalOcean, it's possible to configure a firewall that can protect your VPS ("droplet") from unwanted traffic. You can find this in Network > Firewalls. Create a new firewall, white-list the ports that you want, and then add your droplets. It's that simple.

If your cloud provider doesn't provide a network-level firewall, you could use CloudFlare's network firewall service. This will have negative privacy implications due to routing all traffic thru CloudFlare, but it could be a reasonable trade-off for your case.

If the system is on physical hardware to which you have physical access, then you might consider configuring a firewall on the router thru which your system connects to the internet. If your router doesn't allow you to configure such a firewall, then it might be time to invest in better hardware.

That's it for this one. Good luck and keep those docker services safe!

So you're operating your own Lightning Network node and you've opened a few channels - nice work! But opening channels is just the first step. To make your node more useful for yourself as well as to the rest of the network - i.e. to route payments - it's necessary to balance your channels. In this article, you will learn how to use the lncli command-line tool to manually rebalance your lnd node's channels.

Why Balanced Channels Are Better

Un-balanced channels are a problem because only the counter-parties in a channel know about the internal state of a channel (local vs. remote balances). So when a node constructs a route thru which to send a payment, they are hoping that each channel along the route will have sufficient local balance to route the payment onward. If all of your node's channels are equally balanced, then there's a better chance that another node will succeed in routing a payment thru your node. This is better for the health of the network and also it's better for you, because you can collect routing fees.

Another reason to balance your channels is to give your node more options when sending its own payments. For example, if many of your channels have zero or very low local balance, your node won't be able to use those channels to send payments.

Manual Channel Rebalancing

So first thing to do is to figure out which of your node's channels are in need of rebalancing. Use the following command to get a list of your node's active channels:

lncli listchannels --active_only

Example results:

{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "REMOTE_NODE_PUBKEY_01",
            "channel_point": "XXX:0",
            "chan_id": "SHORT_CHANNEL_ID_01",
            "capacity": "500000",
            "local_balance": "499530",
            "remote_balance": "0",
            "commit_fee": "729",
            "commit_weight": "724",
            "fee_per_kw": "1006",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [
            ],
            "csv_delay": 144,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "5000",
            "remote_chan_reserve_sat": "5000",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "7",
            "uptime": "7",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "5000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "495000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "5000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "495000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        },
        {
            "active": true,
            "remote_pubkey": "REMOTE_NODE_PUBKEY_02",
            "channel_point": "XXX:0",
            "chan_id": "SHORT_CHANNEL_ID_02",
            "capacity": "500000",
            "local_balance": "0",
            "remote_balance": "499302",
            "commit_fee": "728",
            "commit_weight": "724",
            "fee_per_kw": "1006",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [
            ],
            "csv_delay": 144,
            "private": false,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "5000",
            "remote_chan_reserve_sat": "5000",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "7",
            "uptime": "7",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "5000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "495000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "5000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "495000000",
                "min_htlc_msat": "1000",
                "max_accepted_htlcs": 483
            }
        }
    ]
}

In the above example output, we can see that we have two active channels. Both have a channel capacity of 500000 sats. Each of these channels has their balance completely on one side or the other. With such an imbalance, it's unlikely that either channel will be used to route payments for other nodes.

Not Enough Remote Balance?

Triangular channel swaps can help you to use your capital more effectively

If all of your channels have only a local balance and no remote balance, then you will not be able to rebalance them. Before you can continue with the rest of this guide, you will need to first find a counter-party who is willing to open a channel to your node.

Channel Swapping and Liquidity Triangles

I recommend to use one of the following collaborative channel swapping services or groups:

• lightningnetwork.plus
• r/thelightningnetwork channel thread
• r/lightningnetwork channel thread
• t.me/theRingsOfFire
• t.me/Plebnet

Manual rebalance via self payment

To balance the example channels above, you will pick two of your channels that need to be re-balanced. The first channel should have excess local balance. The second channel should have excess remote balance. You will use these two channels to pay yourself thru what is sometimes referred to as a "channel cycle".

Generate a new payment request so that you can pay yourself the amount that you wish to rebalance.

lncli addinvoice 250000

The invoice amount here is 250000 sats because that's half the channel capacity in the example channels above. Change this amount to whatever makes sense in your case.

Use the following command to pay the invoice:

lncli payinvoice \
    --allow_self_payment \
    --fee_limit 30 \
    --outgoing_chan_id SHORT_CHANNEL_ID_01 \
    --last_hop REMOTE_NODE_PUBKEY_02 \
    SELF_PAYREQ

• --allow_self_payment - This flag is required to pay your own invoice
• --fee_limit - The maximum fee to pay (sats).
• --outgoing_chan_id - The short channel id ("chan_id") of the out-bound channel. This is the channel that has excess local balance.
• --last_hop - The public key of the remote node for the in-bound channel. This is the channel that has excess remote balance.

Replace SHORT_CHANNEL_ID_01 with the "chan_id" of the out-bound channel.

Replace REMOTE_NODE_PUBKEY_02 with the "remote_pubkey" of the in-bound channel.

Replace SELF_PAYREQ with the invoice that you generated in the previous command.

You may need to increase the fee limit in case you are unable to route any payments. Alternatively, you can try to generate and pay an invoice for a smaller amount.

If the payment was successful, then congratulations! You've rebalanced your first channels.

Use the command from earlier to inspect the updated balances of your active channels:

lncli listchannels --active_only

You should see that both of the channels are now balanced.

Rebalancing with other tools

Here's a list of other tools that you could use to help you re-balance your channels:

• RTL (Ride the Lightning) - A web interface to help manage a Lightning Network node
• balanceofsatoshis ("bos") - CLI tool that helps with liquidity and channel management
• rebalance-lnd - A script to help rebalance lnd node channels

So you want to setup a Lightning Network node and you want to do it somewhat safely. More than likely the node software that you've chosen is lnd by Lightning Labs. It's a reasonable choice. It provides many tools and interfaces including CLI, JSON-RPC, REST, several backup and restore options, watchtowers, and more.

There are other node implementations for the Lightning Network:

• Eclair
• C-Lightning
• Rust-Lightning
• Electrum - Yes, it supports Lightning Network!

You can investigate those other implementations to decide which one is best for your use-case. The practical guidance in this article will assume you're using lnd.

The Risks Are Real

Lightning Network software is still very much a work-in-progress. And in case you had ideas about putting a significant amount of money into Lightning Network channels, you should read this comment thread on GitHub: "Still cannot force close inactive channels". The comments contain such gems as:

Aah... looks like you restored from an outdated backup? If you're unlucky, you force-closed the channels (which is a breach when your backup is outdated), and the remote took the funds. If you're lucky, the remote force closed and has data-loss protection enabled. If that's the case, you can try to connect to the peer and see what happens. Please post debug logs!

And:

.. you lost data. Therefore your node is now waiting on the remote peer to connect and give you data you need to recover the channel state. Depending on which version they're running (or even implementation), they may never give you this special data.

And:

I have the same problem with more than 3 BTC missing and "If you restored from an old backup (which you should never do!)" sounds like the most irrational thing...

Ouch! But lucky for the last commenter they were able to recover their funds in the end - after a long and difficult debugging process with the help of one of the lnd project contributors.

Still interested in being your own bank?

Setup

The first step towards being your own sovereign individual (at least with regards to finance) is to run your own full bitcoin node. Since lnd can interface with bitcoind and btcd, I recommend to use one of those as your full node software. Some notes regarding full node setup:

For both bitcoind and btcd:

• Set the txindex flag to build the transaction index
• Configure RPC access for your lnd

From lnd's installation documentation:

We don't require --txindex when running with bitcoind or btcd but activating the txindex will generally make lnd run faster.

Additional notes for bitcoind:

• Do not enable pruning

It is also possible to run lnd with neutrino:

Neutrino is an experimental Bitcoin light client written in Go and designed with mobile Lightning Network clients in mind. It uses a new proposal for compact block filters to minimize bandwidth and storage use on the client side, while attempting to preserve privacy and minimize processor load on full nodes serving light clients.

I've found neutrino to be pretty unreliable. But maybe the situation has improved since the last time I used it. Give it a try if you want to avoid setting up a full bitcoin node.

Node Software Installation

There already exist detailed installation instructions provided by the lnd project itself. So if you don't already have lnd installed, follow those instructions and then return here to continue.

Wallet Setup

Every Lightning Network node needs a bitcoin wallet. The lnd node software provides an easy-to-follow setup process via its CLI:

lncli create

Follow the instructions to generate a new seed, set a wallet password, and set your cipher seed passphrase.

Before it can be used, you need to unlock the wallet as follows:

lncli unlock

Provide the wallet password when prompted. The node will begin to download block headers and other information needed to build its internal database.

Watchtower Configuration

Watchtowers act as a second line of defense in responding to malicious or accidental breach scenarios in the event that the client’s node is offline or unable to respond at the time of a breach, offering greater degree of safety to channel funds.

At present there are only "altruistic" watchtowers, meaning node operators are running these watchtower nodes not for profit but to help the network. You might be able to find an altruistic watchtower to watch over your node's channels here:

• Watchtower list
• What are Watchtowers

You can configure a watchtower via the CLI with the lncli wtclient add command. And it is important to note that your lnd node can use multiple watchtowers in case one of them disappears or doesn't fulfill its function.

Or you can configure a watchtower in your lnd.conf:

wtclient.active=true
wtclient.private-tower-uris=PUBKEY@127.0.0.1:9911

Setup Your Own Watchtower

You don't need to use someone else's watchtower. You can run your own. But you should note that it is highly recommended to run a watchtower on its own physical machine with its own infrastructure (electricity, internet access, etc) or in a separate data center from your primary lnd node. You could even run multiple watchtower nodes all in separate data centers around the world.

A watchtower node is an lnd node that has been configured to act as a watchtower. You will need to go thru the usual wallet setup process as if you were preparing a normal lnd node - e.g lncli create and so on.

Here's an example lnd.conf for a watchtower node:

bitcoin.active=1
bitcoin.mainnet=1
bitcoin.node=bitcoind
bitcoind.rpchost=localhost:8332
bitcoind.rpcuser=XXX
bitcoind.rpcpass=XXX
bitcoind.zmqpubrawblock=tcp://localhost:28332
bitcoind.zmqpubrawtx=tcp://localhost:28333
watchtower.active=true
watchtower.listen=localhost:9911

This sample configuration uses bitcoind. The important lines are the last two which are prefixed watchtower. These tell the lnd node to enable its watchtower and to listen at localhost:9911. This works for my setup because I use port-forwarding via SSH reverse proxies to connect my servers.

For more details:

• Private Altruist Watchtowers
• Watchtower Example Demo

Important Tips

After you've finished your node setup, there are a few things you can do to manage your new Lightning Network node and to help save yourself some pain later.

Use a Mobile App

These days there are several solid mobile apps that can help you manage your node:

• Eclair Mobile - Android only
• Zap - Desktop, iOS, and Android
• Zeus - iOS and Android

Using a mobile wallet to access your lnd node remotely will allow you to:

• Use bitcoin as a currency in the real-world
• Check your remote/local channel balances
• Monitor for offline channels

Follow each project's configuration instructions to connect them to your lnd. In general you need three things: Your node's "hostname" (IP Address plus port number), TLS certificate, and macaroon (with admin privileges). The TLS certificate allows your mobile app to establish encrypted communications with your node directly. The macaroon is an authorization token that has some privileges associated with it. In this case you will need the admin macaroon.

Prevent Zombie Channels

This is not an official term, but "Zombie Channel" could be used to refer to an open channel where both nodes are offline. Since neither node involved in the channel is online anymore, the channel will stick around, holding the funds hostage inside.

This could happen in the event that you have an open channel with an offline peer and then your node also goes offline due to some kind of failure. This will make recovering those funds difficult. To prevent this scenario, you can force-close a channel if your peer has been offline for some time.

Backup

To protect your funds, you should ensure that you are backing up important files to help with a possible recovery process in the event of node failure. The most important file to backup is channel.backup - the so called "Static Channel Backups" file.

After version v0.6-beta of lnd, the daemon now ships with a new feature called Static Channel Backups (SCBs). We call these static as they only need to be obtained once: when the channel is created. From there on, a backup is good until the channel is closed. The backup contains all the information we need to initiate the Data Loss Protection (DLP) feature in the protocol, which ultimately leads to us recovering the funds from the channel on-chain. This is a foolproof safe backup mechanism.

On Linux systems, the default location of this file is ~/.lnd/data/chain/bitcoin/mainnet/channel.backup. Have a look at LND backup script for channel.backup using inotify for an example how to backup your channel.backup file.

The channel.backup file is encrypted using your wallet's seed phrase, so it is safe to copy it to your cloud file storage service provider. If you are already using an S3-compatible cloud storage provider then you might find this article useful for configuring your automated remote backup.

Recovery

If you're here that means something has gone wrong. But, there is hope! You will very likely be able to recover most if not all of the bitcoin from your crashed lnd node. Stop for a moment. Breathe. Feeling ok? Good. Now is the time to go slow and make a plan before doing anything.

What should you not do? The absolute worst thing you can do is copy an old channel.db file into a new instance of lnd and run it. That will almost certainly result in permanent loss of funds.

Here is a nice flow chart that can help you figure out your next steps in the recovery process:

The above flow chart was taken from the very helpful chantools project. It is an open-source set of tools developed by one of the contributors to lnd. The chantools program will (hopefully) help you recover your funds from an old copy of your lnd's channel.db. But before we get to that, let's first recover any on-chain funds and attempt a recovery using your static channel backup file.

Recovering On-Chain Funds

Before you do anything with an old channel.db file, let's follow steps 2 and 3 from the above flow chart. With a fresh lnd/lncli instance, let's create a new wallet using your old seed:

lncli create

When prompted for your seed and cipher seed phrase, be sure to provide the same exact values that were used for the crashed lnd node which you are recovering. This will allow the wallet to check for and recover its existing on-chain balance. Wait for your lnd to finish syncing to the chain backend (bitcoind or btcd). You can check its progress as follows:

lncli getinfo

Recovering Off-Chain Funds

Once your lnd has finished syncing, you can move on to the next step: Restoring from channel backup. Check the integrity of your static channel backup file (SCB):

lncli verifychanbackup --multi_file=./channel.backup

Expected output if your backup is OK:

{

}

Initiate the recovery process with the following command:

lncli restorechanbackup --multi_file=./channel.backup

You should see a prompt similar to the following:

WARNING: You are attempting to restore from a static channel backup (SCB) file. This action will CLOSE all currently open channels, and you will pay on-chain fees.

Are you sure you want to recover funds from a static channel backup? (Enter y/n):

Enter "y" to initiate the restore process.

This command can take a long time to execute. Do not stop it prematurely. For more information about these steps, see recovering using SCBs.

And now... you wait. It may take a week or more to recover your funds from channel force-closures.

Further Recovery Steps

After following all the previous recovery steps, if you still have open or pending channels, you will need to use chantools. Carefully follow the steps described in the project's readme file to recover your remaining funds.

If you've never had to migrate a project from one build system to another, I envy you. How sweet it is to have not experienced the psychological torture that is unwinding years of hacks and work-arounds layered on-top of one another during a legacy project's lifetime. But nothing lasts forever. You too will know this feeling eventually. Or maybe not. Let's talk about how you can avoid this fate by using a new (very old) build system called Make.

The Case Against Modern Build Systems

Modern build systems use an insane number of dependencies:

• gulp - 296 nodes, 513 links
• grunt - 170 nodes, 277 links
• webpack - 82 nodes, 119 links

gulp (left), grunt (center), webpack (right)

Of the build systems mentioned above, only one is still under active development. That one is webpack, which is used by the popular web framework React. If you are unfamiliar with webpack, good for you. It is a gigantic piece of software that does too many things. From my experience, webpack can be nice to bootstrap a simple proof-of-concept project, but eventually you will hit a wall where you need to do some extremely hacky work-arounds to do something that it doesn't easily support.

Why am I talking about dependencies? Well...

Earlier this week, many npm users suffered a disruption when a package that many projects depend on — directly or indirectly — was unpublished by its author, as part of a dispute over a package name. The event generated a lot of attention and raised many concerns, because of the scale of disruption, the circumstances that led to this dispute, and the actions npm, Inc. took in response.

This was the "leftpad incident" as it has become to be known.

The package author mentioned in the above quote unpublished more than 250 packages from the npm registry in a very short time. This broke thousands of projects and caused a lot of headaches for maintainers and developers throughout the ecosystem.

This alone should be a strong reason to try to limit your project's exposure to huge dependency graphs.

But in case you are not yet convinced, here are a few more reasons:

• Less time wasted fixing the build process after upgrading dependencies.
• Reduce the attack surface that could allow malicious/rogue dependencies to:
  • Exfiltrate sensitive data such as keys or secrets via the file system or environment variables.
  • Utilize (abuse) system resources to mine cryptocurrencies.
  • Use system's network capacity to spam, run proxy servers, or DOS attack other services.

The Case For Make

Make. Is. Everywhere. You very likely already have it installed on your system. Or if not, it will be available to install via your system's package repository.

Make is ideal for running builds or as a general purpose task runner. It allows you to easily incorporate bash commands and tools that already exist on your system. All of these tools have been around forever, are well tested, and they are stable.

Make In Practice

Here is an example Makefile that includes comments that explain each section:

## Usage
#
#   $ make build        # compile files that need compiling
#   $ make clean        # remove build files
#   $ make clean build  # remove build files and recompile build files from scratch
#

## Variables
BUILD=build
ALL_CSS=$(BUILD)/css/all.css
SRC=src

# Targets
#
# The format goes:
#
#   target: list of dependencies
#     commands to build target
#
# If something isn't re-compiling double-check the changed file is in the
# target's dependencies list.

# Phony targets - these are for when the target-side of a definition
# (such as "build" below) isn't a file but instead a just label. Declaring
# it as phony ensures that it always run, even if a file by the same name
# exists.
.PHONY: build\
clean\
fonts\
images

build: fonts images $(ALL_CSS)

clean:
  # Delete build files:
  rm -rf $(BUILD)/*

# Define a list of CSS source files.
CSS_FILES=$(SRC)/css/fonts.css\
$(SRC)/css/reset.css\
$(SRC)/css/styles.css\
$(SRC)/css/responsive.css

$(ALL_CSS): $(SRC)/css/
  mkdir -p $$(dirname $@)
  rm -f $(ALL_CSS)
  for file in $(CSS_FILES); do \
    echo "/* $$file */" >> $(ALL_CSS); \
    cat $$file >> $(ALL_CSS); \
    echo "" >> $(ALL_CSS); \
  done

fonts:
  # Copy fonts to build directory.
  mkdir -p $(BUILD)/fonts/OpenSans
  cp -r node_modules/open-sans-fontface/fonts/**/* $(BUILD)/fonts/OpenSans/

images:
  # Copy images to build directory.
  mkdir -p $(BUILD)/images/
  cp -r $(SRC)/* $(BUILD)/images/

It is basically bash with the added syntax for build targets. Make will only build files whose inputs have been modified. So in this example, if you change one of your CSS source files then the all.css build file will be recompiled.

The example above is quite simple. It only includes copying and concatenating files. You can add dependencies as you need them to perform minification of JavaScript files, syntax highlighting, templating, and more.

For more advanced build processes, it's a good idea to execute bash (or node.js) scripts from within the Makefile. This gives you the structure and functionality of Make with the flexibility of whichever scripting language you prefer.

During the last few years, I've migrated several projects to Make and it has turned out to be a great move for the long-term maintainability of those projects. You can have a look at some of these projects for more complex, real-world examples using Make:

• PayNoWay - Bitcoin double-spending app for Android
• Bleskomat - Next generation Bitcoin ATM hardware and software project

Choose Make as your next project's build system. Your future self will thank you!

A lot of progress is being made to improve the user experience for those who would like to buy a cup of coffee with their bitcoin. For the longest time most new developments were happening in infrastructure projects such as nodes, exchanges, trading, hardware and software wallets. Now the Lightning Network has brought the focus of many developers back to the end-user, including a resurgence in mobile wallet development with several new Lightning-first wallets.

In the early days a high-level of knowledge was required to use the Lightning Network. But this is no longer the case. There are already some solid, newbie-friendly wallet apps available:

• Phoenix by ACINQ - easy, non-custodial
• WalletOfSatoshi - very easy, but custodial!
• BLW (Bitcoin Lightning Wallet) - non-custodial, medium complexity
• Breez - non-custodial

One of the latest developments that is helping to improve the usability of the Lightning Network is LNURL - a side channel communication protocol to smooth over some of the remaining UX problems. LNURL is a set of subprotocols each with a specific UX flow designed to be simple and easy to implement.

Withdrawals

One pain point with the Lightning Network has been the need to generate a new, unique invoice for every payment that you would like to receive and then somehow give this invoice to the service that you are using so that it can be paid. One real-world example is a web-based paywall service that accumulates satoshis via Lightning Network on your behalf. Once in a while you might want to manually withdraw your satoshis to your own wallet. To do this you would need to first generate an invoice for the exact number of satoshis and then copy/paste that invoice into your computer's browser. But if your wallet is on your phone, this can be an annoying experience.

Bleskomat: The next gen. Bitcoin Lightning ATM

Enter the lnurl-withdraw subprotocol. With a service that supports it, the end user simply scans a single QR code and the service communicates with an LNURL HTTP server to facilitate the payment process via the Lightning Network. The user doesn't need to manually generate invoices or count satoshis - their wallet and the web service can do all of that in the background.

Smooth, New Real-World Applications

This improved user experience is a perfect fit for many applications such as an offline Lightning Network ATM, simplified withdraw processes from exchange accounts, withdrawing change from a PoS terminal, and more.

Static Payment QR Codes

Another popular subprotocol is lnurl-pay. This one enables static QR codes that can be printed and re-used many times. A common example would be a printed sign that contains a QR code for donations. The lnurl-pay subprotocol allows the receiver to specify a range (min/max) for the accepted payment amounts. When the end-user (payer) scans the QR code, they are presented with a dialog that asks how much they want to pay. They choose the amount, confirm the payment in their app, and then their app communicates with an LNURL server to complete the payment via the Lightning Network.

Street Artists Beware!

Those QR-encoded donation addresses are destroying your privacy. Anyone who scans the QR code can see your address and its complete transaction history. One huge advantage to lnurl-pay QR codes is privacy. The payer's app only sees a decoded URL and min/max parameters. They won't be able to see your transaction history or how much money you've received.

Authentication

Growing in popularity with bitcoin and Lightning Network developers, lnurl-auth allows service operators to provide a new method for authentication, authorization, and login.

A special linkingKey can be used to login user to a service or authorise sensitive actions. This preferrably should be done without compromising user identity so plain LN node key can not be used here. Instead of asking for user credentials a service could display a "login" QR code which contains a specialized LNURL.

Though part of the LNURL specification, lnurl-auth doesn't actually touch the Lightning Network. It uses your bitcoin wallet's existing seed to generate a unique signing key for each website with which you authenticate. The protocol can be used as a 2FA option or as a replacement for email/username + password login.

The protocol dictates that the user's app should deterministically derive a "hashing key" and use that to derive a unique linking key for each service with which the user attempts to authenticate. Privacy of the user is protected because the user's app provides a unique public key to each service. App developers are encouraged to follow the derivation scheme defined in the specification to allow for portability between apps.

Open Channel Request

Probably the least popular of the subprotocols, lnurl-channel allows an end-user to request that a channel be opened to their node.

Suppose user has a balance on a certain service which he wishes to turn into an incoming channel and service supports such functionality. This would require many parameters so the resulting QR may be overly dense and cause scanning issues. Additionally, the user has to make sure that a connection to target LN node is established before an incoming channel is requested.

Resources and Tools

If you'd like to add support for LNURL to your own app or service, have a look at this comprehensive list for tools, libraries, and other services that already support it.

During the lockdown earlier this year, I created lnurl-toolbox - a browser-based tool to test mobile and browser-based implementations of the LNURL protocol. If you'd like to try it out for yourself, I recommend the BLW wallet app (linked above) because it has the most comprehensive LNURL support.

The internet is a very different place than the one that I experienced growing up. Back then there were no ad networks, browser fingerprinting, drive-by exploits, obfuscated code running cryptomining software while you read, and all other manner of shady money-driven tactics you see today. It was a simpler time. Now without ad/tracker-blocking browser extensions, the web is almost unusable.

So what can be done by an individual still operating their own website today? Well we can stop using third-party analytics solutions. There is a reason those services are free for you to use. You are paying with your visitors' privacy and eventual erosion of trust in you. And since you are including third-party code on the client, you are risking your visitors' security as well.

Let's get to it. I've been using GoAccess as a replacement for web traffic analytics:

GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

GoAccess analyzes web server log files directly to generate analytics reports. It is compatible with Apache, Nginx, Amazon S3, Elastic Load Balancing, CloudFront, and others. It's open-source, so if you're using some obscure web server with unusual log formats, you can contribute to the project via their GitHub.

Simple Web Server Configuration

You will need to install GoAccess wherever your web server's log files are stored. For the sake of simplicity, we will assume it's on your web server. Here's how to install goaccess:

wget https://tar.goaccess.io/goaccess-1.4.tar.gz
tar -xzvf goaccess-1.4.tar.gz
cd goaccess-1.4/
./configure --enable-utf8 --enable-geoip=legacy
make
make install

For more information see the download page on the project's website.

If you're running your web server on a Linux machine, its log files are likely generated and pruned by a utility named "logrotate". The default max age for log files is quite short (14 days). So you will want to increase this to something like 6 months or a year. Configuration files for logrotate are located in /etc/logrotate.d. Here is an example for nginx (/etc/logrotate.d/nginx):

/var/log/nginx/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        notifempty
        create 640 nginx adm
        sharedscripts
        postrotate
                if [ -f /var/run/nginx.pid ]; then
                        kill -USR1 7
                fi
        endscript
}

The line that you will want to change is rotate 14. This is the maximum age (in days) for log files. Files older than this setting will be pruned/deleted.

To generate a report from your web server's current log files:

goaccess \
        -f /var/log/nginx/access.log* \
        --log-format=COMBINED \
        --ignore-crawlers \
        --output html

Using Docker

If you're using Docker to run your web server, have a look at running GoAccess in Docker - detailed guide to running goaccess in its own container. Not mentioned in that guide is that you will need to use a mounted volume to store your web server's logs.

A short and quick guide to setting up a reverse proxy from your local machine thru a remote virtual private server (VPS). This setup is useful for manual testing a service that's running on your local machine temporarily or if you're running permanent services behind a NAT firewall.

The first thing you will need to do is to reconfigure the SSH service on your VPS. You will need to add the following options to the SSH service's configuration file:

GatewayPorts yes
AllowTcpForwarding yes
ClientAliveInterval 60
ClientAliveCountMax 10

• GatewayPorts - When set to "yes", remote hosts are allowed to connect to ports forwarded for the client.
• AllowTcpForwarding - When set to "yes", TCP forwarding is permitted.
• ClientAliveInterval - Number of seconds that the server will wait before sending a null packet to the client (to keep the connection alive).
• ClientAliveCountMax - This is the limit of how long (increments of ClientAliveInterval) a client is allowed to stay unresponsive before being disconnected.

You can simply append the above configuration options to the end of your server's /etc/ssh/sshd_config, but the options will be applied to all SSH connections - not immediately insecure but also not a good habit to leave such options available system-wide.

A more secure setup is to grant these options to a single user which will be created for the sole purpose of reverse proxying.

To create the reverse proxy user:

useradd \
    --shell /bin/rbash \
    --home-dir /home/reverseproxy \
    --create-home \
    reverseproxy

• --shell /bin/rbash - Sets the login shell for the user to a restricted version of bash.

It is necessary to set a password for the new user even if logging in via pubkey:

passwd reverseproxy

Generate the .ssh directory with authorized_keys file for the new user:

mkdir -p /home/reverseproxy/.ssh; \
    touch /home/reverseproxy/.ssh/authorized_keys

Don't forget to append your pubkey to the authorized_keys file.

If you need further help with this step, see my previous tutorial about how to configure passwordless SSH.

Append the configuration options to your server's SSH configuration file:

cat >> /etc/ssh/sshd_config << EOL
Match User reverseproxy
    GatewayPorts yes
    AllowTcpForwarding yes
    ClientAliveInterval 60
    ClientAliveCountMax 10
    EOL

Then restart the server's SSH service:

service ssh restart

And finally run the following command on your local machine to establish the reverse proxy tunnel:

ssh -v -N -T -R 8080:localhost:8080 reverseproxy@IP_ADDRESS_OF_VPS

• -v - Print verbose log messages.
• -N - Do not execute a remote command.
• -T - Disable pseudo-terminal allocation.
• -R - Establish a reverse tunnel with a remote entry point.

That's it! You should now be able to access the service running at port 8080 (in this example) on your local machine via the virtual private server's IP address.

If you'd like to keep the tunnel open long-term, I suggest to use autossh:

autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic

And if you followed this tutorial and you're still not able to get it working, you can try ngrok instead:

ngrok exposes local servers behind NATs and firewalls to the public internet over secure tunnels

When designing a backup scheme, it's important to start with your high-level requirements. In my case, they are as follows:

• Is fully automated and non-interactive
• Does not expose decryption key by writing it to disk in an unprotected state
• Uses well-supported, high-availability, open-source tools

So with that, let's get started. This article is split into a few different sections, which I've linked here:

• Data Encryption
  • Create a New Key Pair
  • Export and Import Public Key
  • Test the Encryption/Decryption Setup
• Uploading Encrypted Backups to Remote Storage
• Task Scheduling
  • Potential Gotcha's of crontab
  • Testing and Debugging crontab
• Complete Example Script

Data Encryption

One of my goals with this setup is to reduce exposure of the decryption key as much as possible. So writing it to disk in plaintext by storing it in a file on a filesystem should be avoided. With symmetric encryption, the encryption and decryption steps both use the same key. And since we are trying to design an automated (non-interactive) backup scheme, the encryption step requires access to the unprotected encryption key. For these reasons, symmetric encryption is not a good choice.

Asymmetric encryption is the way. So which open source tool has been around forever, is widely available, and is relatively simple to use? Why gpg of course!

“GPG” stands for “Pretty Good Privacy”; “GPG” stands for “Gnu Privacy Guard.” It was the original freeware copyrighted program; GPG is the re-write of GPG. The GPG uses the RSA algorithm and the IDEA encryption algorithm. GPG uses the NIST AES, Advanced Encryption Standard

We will need gpg to be installed on both our local system as well as on the system where we want to run our backup script. For debian and ubuntu, it's as simple as:

sudo apt-get install gpg

For other operating systems it is likely just as simple. Once you've got gpg installed, you can move on to the next steps.

Create a New Key Pair

On your local system, generate a new private/public key pair using gpg:

gpg --full-generate-key

This will guide you through the key generation process. For this backup scheme, I chose the following:

• "RSA and RSA" for kind of key
• 4096 bits key size
• Does not expire

The name and other identifying information is up to you.

Don't forget to backup your keys! If you don't already have a backup scheme setup for your password databases, SSH/PGP keys, and other critical data then now is a good time to do it.

Use the following command to list the keys for which you have both the public and private key:

gpg --list-secret-keys --keyid-format LONG

The output will look something like this:

/path/to/user/.gnupg/pubring.kbx
------------------------------
sec   rsa4096/XXXXXXXXXXXXXXXX 2020-001-01 [SC]
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uid                 [ultimate] xxx (xxx) <xxx@xxx>
ssb   rsa4096/XXXXXXXXXXXXXXXX 2020-001-01 [E]

Export and Import Public Key

The next step is to export the public key from your local machine so that we can import it to the system that will run the backup script.

Use the following command to print the GPG key ID for the last key that you generated:

gpg --list-secret-keys --keyid-format LONG | sed -r -n 's/sec +[a-z0-9]+\/([A-Z0-9]+) .*/\1/p' | tac | head -n 1

And then use this command to export your public key:

gpg --armor --export YOUR_GPG_KEY_ID > key.asc

Upload the "key.asc" file to the system that will run the backup.

Finally, on the backup system run the following to import the key:

gpg --import key.asc

Whew!

Test the Encryption/Decryption Setup

Now let's test the whole encryption/decryption process. Run the following command on the system where you will be running the backup script:

echo 'it works!' | gpg --encrypt -r "YOUR_GPG_KEY_ID" --output ./test.gpg

To demonstrate that the system is not able to decrypt:

gpg -r "YOUR_GPG_KEY_ID" --decrypt ./test.gpg

You should see the following error message:

gpg: encrypted with 4096-bit RSA key, ID XXX, created YYYY-MM-DD
      "xxx"
gpg: decryption failed: No secret key

For a successful decryption test, run the command again but on the system that has your GPG private key. You should see an

gpg -r "YOUR_GPG_KEY_ID" --decrypt ./test.gpg

If you see "it works!" printed in your terminal, then congratulations! If not, you will need to review the above steps to see what you might've missed.

Uploading Encrypted Backups to Remote Storage

You may have already heard the expression, "two is one and one is none", but it's worth considering when designing a data backup scheme. Though it is a bit simplistic, you can interpret it as a reminder that more redundancy is usually better. In our case, adding a remote storage backup can be very beneficial to the overall durability of our data.

You are probably already familiar or at least aware of Amazon's AWS S3 cloud storage, but I prefer to use DigitalOcean's Spaces which offers a compatible API. Both options work with s3cmd - an open source tool to manage your S3-compatible cloud storage.

Like before, installation on debian and ubuntu is quite simple:

sudo apt-get install s3cmd

If using another system, you can follow the installation instructions on the official website linked above.

Now it's time to configure your s3cmd tool. Be sure that your current shell is logged in as the user that will be running the backup script later:

s3cmd --configure

Complete the steps in the configuration. You will need an access key, bucket location and domain, etc. For help with this specific command have a look at the s3cmd howto page or, if using DigitalOcean, have a look at this article about s3cmd usage for "Spaces".

Once you've got the s3cmd configured, you can test it by trying to upload and read a file from your remote storage:

export S3_LOCATION="YOUR_BUCKETNAME" \
    && echo "it works!" | s3cmd -c ~/.s3cfg put - s3://$S3_LOCATION/test \
    && s3cmd -c ~/.s3cfg --no-progress get s3://$S3_LOCATION/test - | cat

Do you see "it works!" printed in your console? Super!

Task Scheduling

Crontab is the obvious choice here because it's available pretty much universally across all unix-like systems.

A crontab file contains instructions for the cron daemon in the following simplified manner: "run this command at this time on this date."

Basically what crontab allows us to do is configure our system to run scheduled tasks. The format for defining which comands to run and when is fairly simple:

m h  dom mon dow   command

• m = minutes
• h = hours
• dom = day of the month
• mon = month
• dow = day of the week
• command = the command to be executed

So for the following example:

0 */3 * * * /path/to/script.sh

The script will be run once at 0 minutes past every 3 hours.

To schedule cron tasks, you must add them to your user's crontab file. Run the following to open the terminal-based editor:

crontab -e

Your tasks should go at the bottom of the file, each on their own new line.

For an interactive, human-friendly tool to practice or test cron scheduling, have a look at crontab.guru. It also includes a few tips for working with crontab.

Potential Gotcha's of crontab

Scripts run via cron do not source the user's profile or bashrc files, so environment variables like PATH will not be set and you will not be able to use all the programs that you expect. The solution is to set the PATH variable at the top of your scripts:

PATH=$PATH:/usr/bin:/usr/sbin:/usr/local/bin

Testing and Debugging crontab

It's a good idea to test your cron setup to verify that everything is working as you expect. So let's start by creating a simple test script:

echo '#!/bin/bash' > ~/cron-test.sh \
    && echo 'PATH=$PATH:/usr/bin:/usr/sbin:/usr/local/bin' >> ~/cron-test.sh \
    && echo 'set -e' >> ~/cron-test.sh \
    && echo 'VERSION="$(s3cmd --version)"' >> ~/cron-test.sh \
    && echo 'echo "[$(date -u +%FT%T)] $VERSION"' >> ~/cron-test.sh \
    && chmod a+x ~/cron-test.sh

Let's set the test script to run once every 15 seconds. This is a little tricky because crontab's granularity is down to the minute. So this requires a bit of a creative kludge:

* * * * * ~/cron-test.sh
* * * * * ( sleep 15 ; ~/cron-test.sh )
* * * * * ( sleep 30 ; ~/cron-test.sh )
* * * * * ( sleep 45 ; ~/cron-test.sh )

What's happening here is that all the cron tasks run at the same time (once every minute). But each subsequent task is delayed by +15 seconds (sleep X).

And to help with debugging the tasks as they run, I like to write the script's output to a log file:

~/cron-test.sh 2>&1 >> ~/cron-test.log;

• 2>&1 = redirect stderr (error output) to wherever stdout is being redirected
• >> ~/cron-test.log = redirect stdout (and append) to crontab.log file

Combining both of the above will look like this:

* * * * * ~/cron-test.sh 2>&1 >> ~/cron-test.log
* * * * * ( sleep 15 ; ~/cron-test.sh 2>&1 >> ~/cron-test.log )
* * * * * ( sleep 30 ; ~/cron-test.sh 2>&1 >> ~/cron-test.log )
* * * * * ( sleep 45 ; ~/cron-test.sh 2>&1 >> ~/cron-test.log )

And finally to watch the log output as it is written:

touch ~/cron-test.log \
    && tail -n 20 -f ~/cron-test.log

If all is well, you should see something like the following:

[2020-01-01T00:00:01] s3cmd version 2.0.2
[2020-01-01T00:00:16] s3cmd version 2.0.2
[2020-01-01T00:00:31] s3cmd version 2.0.2

A new line should appear once every 15 seconds.

Complete Example Script

So after all of that, here's a complete example script for you to copy/paste to your heart's content. It uses pg_dump to dump an entire Postgres database, gzip to compress the data, gpg to encrypt the data, and finally s3cmd to upload the encrypted backup file to remote storage.

#!/bin/bash

#   MIT License
#   
#   Copyright (c) 2020 Charles Hill
#   
#   Permission is hereby granted, free of charge, to any person obtaining a copy
#   of this software and associated documentation files (the "Software"), to deal
#   in the Software without restriction, including without limitation the rights
#   to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#   copies of the Software, and to permit persons to whom the Software is
#   furnished to do so, subject to the following conditions:
#   
#   The above copyright notice and this permission notice shall be included in all
#   copies or substantial portions of the Software.
#   
#   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#   IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#   FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#   AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#   LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#   OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
#   SOFTWARE.
#   
#   For a human-friendly explanation of this license:
#   https://tldrlegal.com/license/mit-license
#   
#   If you have questions technical or otherwise, find my contact details here:
#   https://degreesofzero.com/
#   

# This is required because scripts run via cron do not source the user's profile or bashrc files.
PATH=$PATH:/usr/bin:/usr/sbin:/usr/local/bin

# Date/time will be used to keep our backup files properly organized.
DATE=$(date -u +%F)
DATETIME="$(date -u +%FT%T)"

# https://github.com/s3tools/s3cmd
# https://www.digitalocean.com/docs/spaces/resources/s3cmd/
S3_LOCATION="s3/bucket/path/$DATE"
S3_CONFIG_FILE="/path/to/user/.s3cfg"

# The GPG key ID that was previously imported to the server's keyring.
# NOTE: Don't forget to mark the imported key as trusted.
GPG_KEY_ID="XXX"

# Directory where backup files are stored locally.
BACKUPS="/path/to/local/backups"
mkdir -p $BACKUPS

# Dump SQL, compress and encrypt.
DBHOST="localhost"
DBNAME="XXX"
DBUSER="XXX"
DBPASS="XXX"
FILE="$BACKUPS/backup-file-name-$DATETIME.sql.gz.gpg"
if [ ! -f "$FILE" ]; then
    echo "Creating encrypted SQL dump..."
    pg_dump \
        --host="$DBHOST" \
        --dbname="$DBNAME" \
        --username="$DBUSER" \
        --password="$DBPASS" \
            | gzip --best - \
            | gpg --encrypt -r "$GPG_KEY_ID" --output $FILE
else
    echo "Encrypted backup already exists"
fi

echo "$FILE"

# Upload to remote location.
echo "Uploading to remote..."
s3cmd -c $S3_CONFIG_FILE put $FILE s3://$S3_LOCATION/

echo "Done!"
exit 0

Double-spending is no longer a theoretical possibility but a practical reality. Most of the end-user applications used widely today leave their users vulnerable to being defrauded via double-spend attacks.

So what is double-spending and what can be done to protect users? First you need to understand two things:

• Transactions in Bitcoin are made of inputs and outputs
• A transaction is valid if its inputs have not been previously spent

A double-spend is a transaction that has successfully invalidated a previously accepted payment transaction. Over the years, several different types of double-spend attacks have been theorized and some even put to practice:

• Race Attack - Two transactions that spend the same input are in a race against each other. The first to be confirmed is the winner.
• Finney Attack - An attacker mines a block in secret. This secretly mined block contains a double-spend transaction. The attacker then broadcasts a payment transaction that an unsuspecting merchant accepts. The attacker then broadcasts the mined block.
• Alternative History Attack - A.k.a "51% Attack"
• Unconfirmed Transaction Replacement - A.k.a "Replace-by-fee (RBF) Attack"

Race Attack (left) and Finney Attack (right)

Alternative History Attack

This type of double-spend attack involves rewriting blocks to effectively erase a confirmed transaction from history. Once this is done, the inputs used in the original transaction can be double-spent however the attacker sees fit. The more confirmations a transaction has, the more difficult it becomes to change it. This is why it is recommended to wait at least a few confirmations before considering high-value transactions as final.

Note that it is not necessary, but highly unlikely, to be able to achieve a change-of-history double-spend attack without 51% of the network hashrate. But due to the size of the Bitcoin network, this is a very expensive and risky attack to manage successfully. There are many instances of this attack in recent history, but on much smaller networks:

• "Double Spend Attack On Bitcoin Gold (BTG) Causes Chaos For Exchanges" - May 2018
• "51% attack on monacoin" - May 2018
• "Litecoin Cash Allegedly the Latest Small-Cap Altcoin to Suffer 51 Percent Attack" - June 2018
• "So EMC2 was Hacked by a 'Double Spend Attack.'" - July 2018
• "219,500 Ethereum Classic (ETC) Worth $1.1 Million Double Spent During 51 Percent Attack; Coinbase May Delist" - January 2019
• "Coinbase Says Ethereum Classic Attack Included $500,000 in Double Spends" - January 2019

As a network grows it becomes more and more expensive to launch a successful 51% attack. That's not to say that it is impossible or that Bitcoin itself is immune to double-spending. There are historical examples of successful double-spend attacks against Bitcoin. However these attacks take a different approach - they use unconfirmed transaction replacement. There are even historical examples of this happening in the wild:

• "Bitcoin ATM Double-Spenders: Police Need Help Identifying Four Criminals" - March 2019

Unconfirmed Transaction Replacement

Don't have access to 51% of the hash rate? Not to worry. There's another type of double-spend attack that is Much more accessible to the average Bitcoin user: Replace-by-fee (RBF) or Unconfirmed Transaction Replacement.

A transaction that has not yet been included in a block is considered "unconfirmed" - also referred to as a zero-confirmation ("0-conf") transaction. Once confirmed in a block, it is not possible to replace it in the manner described here.

To successfully double-spend by replacing an unconfirmed transaction, you will need to be able to:

• Build raw transactions that use the replace-by-fee (RBF) feature
• Broadcast raw transactions reliably
• Fetch unspent transaction outputs (UTXO) for a given address
• Get network fee estimate

There's An App For That!

All this can be done for you with the help of a script or, even more conveniently, as an app. To illustrate the point, I created PayNoWay to demonstrate to business owners what a practical double-spend attack can look like.

The app looks and acts like a "normal" mobile wallet application: Scan a QR code generated by a Point-of-Sale app; Send the payment (blue button); Send a double-spend transaction to recover the funds (red button); Wait for one of the transactions to be confirmed. A successful double-spend is not guaranteed, but the success rate is quite high (~90%) on mainnet Bitcoin.

Screenshots from PayNoWay

Almost every person to whom I have demonstrated the double-spend attack was shocked at just how easy it was. This attack is within the capabilities of many cryptocurrency users. Anyone who is accepting transactions as payment for goods or services without waiting for at least one confirmation is vulnerable and should be aware of the risks.

As a proof-of-concept the app also supports Litecoin mainnet, but the success rate is low due to replacement transactions being rejected by almost every relay node. However this does not mean that Litecoin is not susceptible. It would be possible to greatly improve the success rates by connecting directly to mining nodes - which are far less likely to reject any transactions that pay a higher fee.

Wallet Apps vs. Double-Spending

Let's have a look at how some of the most popular mobile and desktop apps perform against double-spending. The following testing criteria were used:

• RBF Flag - Does the app display a warning to the user if a received transaction is RBF?
• Double-Spend Alert - Does the app display a warning when a transaction was double-spent?
• Child-Pays-For-Parent - Does the app allow the user to attempt to boost a received transaction's fee by using CPFP?
• Correct Balance - Does the app show the correct balance before and after a double-spend?

Child pays for parent means, as the name implies, that spending an unconfirmed transaction will cause miners to consider confirming the parent transaction in order to get the fees from the child transaction included in the same block.

Only wallet applications that allow receiving on-chain Bitcoin payments were tested.

Block Explorers vs. Double-Spending

Some Bitcoin users rely on websites called "Block Explorers" to show them their account (address) balances and transaction details. This is a bad practice, but users will use what is the easiest for them. So it's up to the maintainers and operators of these websites to do their best to educate and protect their users.

The following testing criteria were used:

• RBF Flag - Does the explorer display a warning to the user if a transaction is RBF?
• Double-Spend Alert - Does the explorer display a warning when a transaction was double-spent?
• Both txs visible - Are both the payment and double-spend transactions visible while unconfirmed?
• Original tx preserved - Does the explorer keep the original payment transaction even after the double-spend transaction was confirmed?

Solutions

There is no magical solution to this problem. Nodes and transaction relays can't know if a transaction is actually a malicious double-spend. A transaction might look suspicious because it's consuming the same exact inputs but has changed its outputs. But that doesn't necessarily mean that it is a malicious double-spend. Even worse, there's no guarantee that your wallet app will receive both the payment and double-spend transactions.

All that said, there are solutions!

• Don't accept unconfirmed RBF transactions
• Only accept confirmed transactions
• Accept unconfirmed RBF transactions but monitor the memory pool for double-spends
• Only accept payments via Lightning Network (LN) - Off-chain transactions that do not require confirmations for the recipient to know that they have been paid.

Which solution is best depends on the context. The best solution for in-person payments (cafes, bars, restaurants, etc) is to use Lightning Network only. On-chain payments have high fees and require more time to be safe. Using LN for this context is great because it reduces fees and is not vulnerable to double-spending.

The purpose of this article is to bring awareness to the community. Please do not double-spend against merchants who are accepting cryptocurrencies as payment. If you see a business accepting Bitcoin payments in a naive way, do try to bring it to their attention but in a responsible way. As a still small community we should do our best to help each other and encourage the businesses that do already accept Bitcoin payments.

I'd love to hear about other solutions to this problem, so please feel free to reach out to me with your ideas or suggestions. See my contact details below.

Additional Resources

• Double-Spending Made Easy - Slides I used during my presentation at the HCPP19.

• Introduction
  • What is the Lightning Network?
  • Our Own Private Bitcoin Network
  • Sharing Information During the Workshop
• For participants:
  • Install and Configure Your Lightning Network Node
  • Create and Fund Your Wallet
  • Sending and Receiving Payments
• For the organizer:
  • Setup Private Bitcoin Network
• More Resources

Introduction

In this workshop, we are going to setup a shared, private lightning network. Each of you will have your own Lightning Network node with which you will participate in the network. You will:

• Create and fund your own bitcoin wallet
• Connect to and open a channel with "Bob" (see the diagram below)
• Send a payment to one of the other participants
• Receive a payment

Here is a diagram to illustrate:

   LND                        LND                        LND
+ ----- +                   + --- +                   + ----- +
| Alice | <--- channel ---> | Bob | <--- channel ---> | Carol |
+ ----- +                   + --- +                   + ----- +
    |                          |                          |
    |                          |                          |
    +- - - - - - - - - - - - - +- - - - - - - - - - - - - +
                               |
                         + ----------- +
                         | BTC network | <--- BTCD
                         + ----------- +

This setup will allow each of the connected parties to send and receive payments with all the others - even if they are not directly connected by a channel. For example, Alice can send payments to Bob because they have a channel between each other but she can also send payments to Carol via Bob's node because he has a channel with Carol.

What is the Lightning Network?

First let's start with a brief, high-level explanation of what the Lightning Network is and how it works. From the community documentation:

The Lightning Network scales blockchains and enables trustless instant payments by keeping most transactions off-chain and leveraging the security of the underlying blockchain as an arbitration layer.

This is accomplished primarily through "payment channels", wherein two parties commit funds and pay each other by updating the balance redeemable by either party in the channel. This process is instant and saves users from having to wait for block confirmations before they can render goods or services.

And the key word to remember is network. Payment channels by themselves are not so interesting. But when a transaction can be passed along from one peer to the next until it reaches its recipient - without the need for a direct connection between the payer and payee - now that is interesting.

Our Own Private Bitcoin Network

Lightning Network nodes need to communicate with a bitcoin node to be able to create on-chain transactions, watch the blockchain for updates, and to open/close channels.

Your LN node will communicate with a remote bitcoin node that we are running specifically for this workshop. This bitcoin node will be running in "simnet" mode, which gives us full control over the blockchain. This allows us to instantly mine blocks so that we don't have to wait 10 minutes for each block - the average time per block on the mainnet or testnet networks.

Sharing Information During the Workshop

To make it easy for us to share information with each other during this workshop, we will use an open IRC channel:

• In your browser go to webchat.freenode.net
• In the "Channels" field enter #ln-workshop
• You can send private messages or just post in the general chat

Install and Configure Your Lightning Network Node

Download and install GoLang:

wget https://storage.googleapis.com/golang/go1.10.4.linux-amd64.tar.gz
tar -xvf go1.10.4.linux-amd64.tar.gz
mv go ~/.go

Add GoLang-related environment variables to your user's .profile:

cat >> ~/.profile << EOL
export GOROOT="\$HOME/.go"
export GOPATH="\$GOROOT/packages"
PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin
EOL

Reload your user's .profile to load the new environment variables into the current terminal session:

source ~/.profile

Test GoLang setup:

go version

The previous command should output something like this:

go version go1.10.4.linux/amd64

Install lnd (Lightning Network Daemon):

go get -v -d github.com/lightningnetwork/lnd
cd $GOPATH/src/github.com/lightningnetwork/lnd
make && make install

Create an lnd configuration file:

mkdir -p ~/.lnd;
cat > ~/.lnd/lnd.conf << EOL
bitcoin.active=1
bitcoin.simnet=true
bitcoin.node=btcd
btcd.rpchost=PROVIDED_BY_ORGANIZER
btcd.rpcuser=PROVIDED_BY_ORGANIZER
btcd.rpcpass=PROVIDED_BY_ORGANIZER
btcd.rpccert=$HOME/.btcd/rpc.cert
debuglevel=debug
EOL

Create aliases for lnd and lncli:

cat >> ~/.profile << EOL
alias lnd-ws="lnd --lnddir=$HOME/.lnd"
alias lncli-ws="lncli --lnddir=$HOME/.lnd --network=simnet"
EOL

Make the new aliases usable in the current terminal window:

source ~/.profile

One last step to be able to communicate with the btcd node via a TLS-encrypted connection. Save the remote btcd node's TLS certificate to a file locally:

mkdir -p ~/.btcd;
cat > ~/.btcd/rpc.cert << EOL
PROVIDED_BY_ORGANIZER
EOL

Test your lnd setup:

lnd-ws --version

You should see something like this:

lnd version 0.5.0-beta commit=7fe095c54128b502e20b3482ef73f1b6ba737850

Great! Now let's move on to the next step.

Create and Fund Your Wallet

Start lnd:

lnd-ws

You should see something like this:

2018-10-02 22:03:15.710 [INF] LTND: Version 0.5.0-beta commit=7fe095c54128b502e20b3482ef73f1b6ba737850
2018-10-02 22:03:15.710 [INF] LTND: Active chain: Bitcoin (network=simnet)
2018-10-02 22:03:15.714 [INF] CHDB: Checking for schema update: latest_version=6, db_version=6
2018-10-02 22:03:15.714 [INF] RPCS: Generating TLS certificates...
2018-10-02 22:03:15.731 [INF] RPCS: Done generating TLS certificates
2018-10-02 22:03:15.731 [INF] RPCS: password RPC server listening on 127.0.0.1:10009
2018-10-02 22:03:15.732 [INF] RPCS: password gRPC proxy started at 127.0.0.1:8080
2018-10-02 22:03:15.732 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.

Leave lnd running.

In a new terminal window, run the following command:

lncli-ws create

Follow the prompts to complete the LN wallet creation process.

If successful you should see the following message:

lnd successfully initialized!

Switch back to the terminal window where your lnd is running. You should now see something like this:

2018-10-02 22:29:07.280 [INF] LNWL: Opened wallet
2018-10-02 22:29:07.340 [INF] LTND: Primary chain is set to: bitcoin
2018-10-02 22:29:08.233 [INF] LNWL: The wallet has been unlocked without a time limit
2018-10-02 22:29:08.233 [INF] LNWL: Catching up block hashes to height 0, this will take a while...
2018-10-02 22:29:08.237 [INF] LTND: LightningWallet opened
2018-10-02 22:29:08.242 [INF] LNWL: Caught up to height 0
2018-10-02 22:29:08.243 [INF] HSWC: Restoring in-memory circuit state from disk
2018-10-02 22:29:08.244 [INF] LNWL: Done catching up block hashes
2018-10-02 22:29:08.245 [INF] HSWC: Payment circuits loaded: num_pending=0, num_open=0
2018-10-02 22:29:08.246 [INF] LNWL: Started rescan from block 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6 (height 0) for 0 addresses
2018-10-02 22:29:08.247 [INF] LNWL: Catching up block hashes to height 0, this might take a while
2018-10-02 22:29:08.249 [INF] LNWL: Done catching up block hashes
2018-10-02 22:29:08.249 [INF] LNWL: Finished rescan for 0 addresses (synced to block 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6, height 0)
2018-10-02 22:29:08.250 [INF] RPCS: RPC server listening on 127.0.0.1:10009
2018-10-02 22:29:08.250 [INF] RPCS: gRPC proxy started at 127.0.0.1:8080
2018-10-02 22:29:08.302 [INF] HSWC: Starting HTLC Switch
2018-10-02 22:29:08.302 [INF] NTFN: New block epoch subscription
2018-10-02 22:29:08.302 [INF] NTFN: New block epoch subscription
2018-10-02 22:29:08.302 [INF] NTFN: New block epoch subscription
2018-10-02 22:29:08.302 [INF] DISC: Authenticated Gossiper is starting
2018-10-02 22:29:08.302 [INF] NTFN: New block epoch subscription
2018-10-02 22:29:08.302 [INF] BRAR: Starting contract observer, watching for breaches.
2018-10-02 22:29:08.302 [INF] CRTR: FilteredChainView starting
2018-10-02 22:29:08.350 [INF] CRTR: Filtering chain using 0 channels active
2018-10-02 22:29:08.350 [INF] CRTR: Prune tip for Channel Graph: height=0, hash=683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6
2018-10-02 22:29:08.351 [INF] CMGR: Server listening on [::]:9735
2018-10-02 22:29:08.352 [INF] SRVR: Auto peer bootstrapping is disabled

Funding Your New Wallet

Create a new bitcoin address:

lncli-ws newaddress np2wkh

Note that the address type here is important - np2wkh ("Pay to nested witness key hash").

Example output:

{
    "address": "<new address printed here>"
}

Copy and paste this address to the IRC channel.

The workshop organizer should now re-start the btcd node with the address as the recipient "mining" address:

btcd --miningaddr=<ADDRESS>

Then the organizer should generate new blocks using the btcctl utility:

btcctl generate 100

This will generate 100 new blocks.

Example output:

[
  "68d12bead9ac4a87599582d5186bf08634c9dfe86678d123b27d7d682ecd39df",
  "14cd20d79a0b2786b6a24710e75822eadd949f57cac93ea0dc8abcf858f7bf18",
  "67e5d6ceb6db58d334a195ab85613753aa909a2783c61c600320733614d6c7c7",
  "4e92a4c34792ac9394efb8470d6251656b09ae1e9d8bdfb95f5f7a69569ce925",
  "5ba8cbc7d51c80372aa48512fc81760f48f26167df518e6845ace3fab0978882",
  "... more block hashes omitted"
]

Normally blocks are mined about once every 10 minutes. But we don't have that kind of time, so we use the above command to generate blocks on-demand as we need them.

Now check your wallet balance:

lncli-ws walletbalance

Example output:

{
    "total_balance": "5000000000",
    "confirmed_balance": "5000000000",
    "unconfirmed_balance": "0"
}

• The amounts shown are in satoshis
• 1 bitcoin = 100 million satoshis

The above steps should be repeated for each workshop participant.

Opening a Channel

Before you can open channels, you will need to be connected to other lightning nodes ("peers"). To see your current peer connections:

lncli-ws listpeers

Since you haven't connected to any peers yet, the result will be an empty array:

{
  "peers": []
}

To connect to "Bob":

lncli-ws connect 0200eacbb299639eb19f4afd39d572e87398062e500e6b63ceb1ebded31fddd3b7@159.89.23.5

Now when you list your peers, you should see Bob's node listed:

lncli-ws listpeers

Result:

{
    "peers": [
        {
            "pub_key": "0200eacbb299639eb19f4afd39d572e87398062e500e6b63ceb1ebded31fddd3b7",
            "address": "159.89.23.5:9735",
            "bytes_sent": "279",
            "bytes_recv": "279",
            "sat_sent": "0",
            "sat_recv": "0",
            "inbound": false,
            "ping_time": "0"
        }
    ]
}

But all we've done so far is make two lightning nodes aware of each other. We haven't opened any channels yet.

Try to open a channel with your new peer:

lncli-ws openchannel --node_key=<PEER_PUBKEY> --local_amt=1000000

Each channel has two sides ("local" and "remote"). The local_amt argument in the above command is how much you want to add to the channel on the local (your) side.

Example output on success:

{
  "funding_txid": "41f045e8d3a33cceff237183e37eeabff3aea0cf4d64e28238807afe5df2dfa5"
}

If you see the following error:

[lncli] rpc error: code = Unknown desc = not enough witness outputs to create funding transaction, need 0.01 BTC only have 0 BTC  available

Then you need to go back to the Create and Fund Your Wallet step above and check that you are using the correct address type.

If you see an error similar to the following:

[lncli] rpc error: code = Unknown desc = has witness data, but segwit isn't active yet

This means that the block height of the simnet blockchain is not high enough. The threshold for segwit activation is 300 blocks on simnet.

When your lightning node tries to open a new channel, it creates and broadcasts a special bitcoin transaction. Before the channel is considered "open", this transaction needs to be confirmed. The default number of confirmations (blocks) is 6. Whoever is controlling the btcd node should generate 6 new blocks:

btcctl generate 6

Check that the channel was created:

lncli-ws listchannels

You should see your new channel:

{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "0200eacbb299639eb19f4afd39d572e87398062e500e6b63ceb1ebded31fddd3b7",
            "channel_point": "a8ea16394e1c759f57b3486eea33e3977b0a39d7ea70f409c3519bd581b52573:0",
            "chan_id": "1181974999924736",
            "capacity": "1000000",
            "local_balance": "990950",
            "remote_balance": "0",
            "commit_fee": "9050",
            "commit_weight": "600",
            "fee_per_kw": "12500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [
            ],
            "csv_delay": 144,
            "private": false
        }
    ]
}

Super! With your newly opened channel, you can start sending and receiving lightning payments.

Sending and Receiving Payments

The first step in receiving a payment via the Lightning Network, is to generate an invoice. Use the following command to create your first invoice:

lncli-ws addinvoice --amt=10000

Example output:

{
  "r_hash": "a9b22eca10de08426f11f3f59b8a733f1af831a699c1b3f6ca632533239dc1dd",
  "pay_req": "lnsb1pd8pxdzpp54xezajssmcyyymc3706ehznn8ud0svdxn8qm8ak2vvjnxguac8wsdqqcqzyse0qkh2fdn4adwlz598s4v9l2ulner3jalncsjf33za0r3hksv2u3m7vw2663ypaqcc4fjsuzeh5n5hfsqyggwk3rzp6neng4hza8stgp4aaszp"
}

The pay_req field is what we will share with our customer (or whoever will pay the invoice).

Share the above payment request invoice via the IRC chat so that one of the other workshop participants can send you a payment.

Some other participant should run the following command to send the payment:

lncli-ws payinvoice <The Payment Request>

But wait, something went wrong:

{
    "payment_error": "unable to route payment to destination: TemporaryChannelFailure: insufficient capacity in available outgoing links: need 9002018 mSAT, max available is 1312000 mSAT",
    "payment_preimage": "",
    "payment_route": null
}

We are seeing this error because Bob doesn't have any funds on its side of your channel. In order for the payment to be successfully routed from the other participant to you via Bob, his node must have a local balance equal to or greater than the requested amount. Here is a series of diagrams to illustrate:

+ ----- +                      + --- +                      + --------- +
| payor | <-- 1000000 -- 0 --> | Bob | <-- 0 -- 1000000 --> | recipient |
+ ----- +                      + --- +                      + --------- +

The next step of the failing payment looks like this:

+ ----- +                         + --- +                      + --------- +
| payor | <-- 990000 -- 10000 --> | Bob | <-- 0 -- 1000000 --> | recipient |
+ ----- +                         + --- +                      + --------- +

And then where the payment finally fails:

+ ----- +                         + --- +                            + --------- +
| payor | <-- 990000 -- 10000 --> | Bob | <-- (10000) -- 1010000 --> | recipient |
+ ----- +                         + --- +       !!!                  + --------- +

Payments are routed by transferring value from one side of a channel to another so that the transacted amount is effectively moved through the network until it reaches its final destination. You can only ever transfer value between the two peers involved in a channel by manipulating the local and remote balances in that channel. It is not possible to transfer value directly between channels. If one side of a channel does not have enough value to transfer to the other side, the routing will fail - as in the example above.

To fix this, you must first send some funds to Bob.

Whoever is controlling Bob's node should create a second payment request:

lncli-ws addinvoice --amt=10000

Once they've shared the payment request via IRC, go ahead and pay it:

lncli-ws payinvoice <Bob's Payment Request>

Check the balances in your channel to see that the payment was sent:

lncli-ws listchannels

Now you should see that the remote balance is enough for your previous payment request to be routed.

{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "0200eacbb299639eb19f4afd39d572e87398062e500e6b63ceb1ebded31fddd3b7",
            "channel_point": "a8ea16394e1c759f57b3486eea33e3977b0a39d7ea70f409c3519bd581b52573:0",
            "chan_id": "1181974999924736",
            "capacity": "1000000",
            "local_balance": "980950",
            "remote_balance": "10000",
            "commit_fee": "9050",
            "commit_weight": "724",
            "fee_per_kw": "12500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "10000",
            "total_satoshis_received": "0",
            "num_updates": "2",
            "pending_htlcs": [
            ],
            "csv_delay": 144,
            "private": false
        }
    ]
}

Ask another workshop participant to try to pay your invoice. This time it should be successful. Cool!

That's it for this workshop. The remaining sections are part of the setup guide for the workshop organizer.

Thanks for participating!

Setup Private Bitcoin Network

In order to create a private bitcoin network, you will need to setup at least one bitcoin node. In this workshop, we will be using btcd. Steps to install btcd:

go get -v -u github.com/Masterminds/glide
git clone https://github.com/roasbeef/btcd $GOPATH/src/github.com/roasbeef/btcd
cd $GOPATH/src/github.com/roasbeef/btcd
glide install
go install . ./cmd/...

Create btcd configuration file. RPC usernames and passwords will be "randomly" generated when you create the file:

mkdir -p ~/.btcd;
cat > ~/.btcd/btcd.conf << EOL
simnet=1
txindex=1
rpclisten=$(host=($(hostname -I)); echo ${host[0]})
rpcuser=$(date '+%s%N-ln-workshop' | sha256sum | head -c 20)
rpcpass=$(date '+%s%N-ln-workshop' | sha256sum | head -c 32)
rpclimituser=$(date '+%s%N-ln-workshop' | sha256sum | head -c 20)
rpclimitpass=$(date '+%s%N-ln-workshop' | sha256sum | head -c 32)
rpcmaxclients=100
rpcmaxwebsockets=300
debuglevel=debug
EOL

Create the btcctl configuration file:

mkdir -p ~/.btcctl;
cat > ~/.btcctl/btcctl.conf << EOL
simnet=1
$(cat ~/.btcd/btcd.conf | grep rpcuser=)
$(cat ~/.btcd/btcd.conf | grep rpcpass=)
EOL

Start btcd:

btcd

You should see something like the following:

2018-10-02 21:58:48.039 [INF] BTCD: Version 0.12.0-beta
2018-10-02 21:58:48.039 [INF] BTCD: Loading block database from '/home/user/.btcd/data/simnet/blocks_ffldb'
2018-10-02 21:58:48.045 [INF] BTCD: Block database loaded
2018-10-02 21:58:48.055 [INF] INDX: Transaction index is enabled
2018-10-02 21:58:48.055 [INF] INDX: cf index is enabled
2018-10-02 21:58:48.056 [DBG] INDX: Current internal block ID: 0
2018-10-02 21:58:48.056 [DBG] INDX: Current transaction index tip (height -1, hash 0000000000000000000000000000000000000000000000000000000000000000)
2018-10-02 21:58:48.056 [DBG] INDX: Current committed filter index tip (height -1, hash 0000000000000000000000000000000000000000000000000000000000000000)
2018-10-02 21:58:48.056 [INF] INDX: Catching up indexes from height -1 to 0
2018-10-02 21:58:48.056 [INF] INDX: Indexes caught up to height 0
2018-10-02 21:58:48.056 [INF] CHAN: Chain state (height 0, hash 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6, totaltx 1, work 2)
2018-10-02 21:58:48.073 [INF] AMGR: Loaded 0 addresses from file '/home/user/.btcd/data/simnet/peers.json'
2018-10-02 21:58:48.073 [INF] RPCS: RPC server listening on 10.0.0.1:18556
2018-10-02 21:58:48.073 [INF] CMGR: Server listening on 0.0.0.0:18555
2018-10-02 21:58:48.073 [INF] CMGR: Server listening on [::]:18555

If you have a firewall enabled (always a good idea!), then you will need to allow connections to the btcd node's RPC port:

sudo ufw allow 18556

Print your local area network IP address:

hostname -I

And the credentials for the limited RPC user:

cat ~/.btcd/btcd.conf | grep rpclimit

And finally the btcd node's RPC certificate:

cat ~/.btcd/rpc.cert

Share the output of the above commands with the workshop's IRC channel, so that the workshop attendees can finish the configuration of their lnd nodes.

For the RPC TLS certificate, create a "gist" and share the link instead of posting directly to the IRC chat.

Follow the participant instructions (links at the top of the page), to setup your lnd node and to create and fund its bitcoin wallet. Your lnd node will act as "Bob" during this workshop.

Gotcha's for Organizer

There are some edge cases that can cause problems and waste time:

• If you've already had your btcd node setup once before and you change its listener IP address, you will need to re-generate the RPC TLS certificate. Delete the current certificate (and key file) then restart btcd.

More Resources

To learn more about the Lightning Network, have a look at the following additional resource(s):

• https://dev.lightning.community/ - Community-driven website with many guides and detailed technical documentation about the LN protocol
• https://github.com/ElementsProject/lightning - Alternate LN implementation written in C
• https://github.com/ACINQ/eclair - Alternate LN implementation written in Scala

There are many potential applications that need a reliable, fast stream of bitcoin transactions: payment processing, gathering transaction data for statistics and network analysis, fee estimation, and many more that I can't think of at the moment. You might think about using some existing third-party services (or APIs), but you will quickly realize that most of those services are not reliable, too slow, inefficient, or too expensive to license. And let's not forget the biggest problem of all: They are a third-party. In the world of cryptocurrencies, it's always better to verify yourself and cut out the middleman.

So in this guide, we will walk through the steps needed to setup a bitcoin node that will stream transactions over TCP using the ZeroMQ protocol. For the sake of simplicity, it is assumed that both your bitcoin node and the subscriber application will be on the same machine.

Setup bitcoind

Download bitcoind:

wget -O bitcoind.tar.gz https://bitcoin.org/bin/bitcoin-core-0.16.2/bitcoin-0.16.2-x86_64-linux-gnu.tar.gz;

Check bitcoin.org for the latest version number.

Extract the compressed tar file:

tar -xvzf bitcoin.tar.gz

And move the bitcoind and bitcoin-cli binaries to the current directory:

mv bitcoin-0.16.2/bin/bitcoind ./
mv bitcoin-0.16.2/bin/bitcoin-cli ./

Create a new folder to be used as the data directory:

mkdir -p .bitcoin

Create a bitcoin configuration file:

cat > ./.bitcoin/bitcoin.conf << EOL
# Use the regtest network, because we can generate blocks as needed.
regtest=1

# In this example, we will keep bitcoind running in one terminal window.
# So we don't need it to run as a daemon.
daemon=0

# RPC is required for bitcoin-cli.
server=1
rpcuser=test
rpcpassword=test

# In this example we are only interested in receiving raw transactions.
# The address here is the URL where bitcoind will listen for new ZeroMQ connection requests.
zmqpubrawtx=tcp://127.0.0.1:3000
EOL

For more details on how ZeroMQ is implemented and configured in bitcoind, see bitcoin/zmq.md.

Start bitcoind:

./bitcoind -datadir=./.bitcoin

Open a new terminal window (or tab), leaving bitcoind running in the first one.

Use bitcoin-cli to generate new blocks:

./bitcoin-cli -datadir=./.bitcoin generate 101

This will give your local wallet some regtest bitcoin to play with.

Get a new address:

./bitcoin-cli -datadir=./.bitcoin getnewaddress

Send some bitcoin to the new address:

./bitcoin-cli -datadir=./.bitcoin sendtoaddress "ADDRESS" 1.000

You will need to repeat this last step when testing the example node.js application later.

Example Subscriber Application

In the following example node.js script, we will use zeromq.js and bitcoinjs-lib to subscribe to a TCP socket using the ZeroMQ protocol and parse the received raw transaction data. The code is commented to help understand how it works.

// Library for working with the bitcoin protocol.
// For working with transactions, hd wallets, etc.
var bitcoin = require('bitcoinjs-lib');

// Implementation of ZeroMQ in node.js.
// From the maintainers of the ZeroMQ protocol.
var zmq = require('zeromq');

// Create a subscriber socket.
var sock = zmq.socket('sub');
var addr = 'tcp://127.0.0.1:3000';

// Initiate connection to TCP socket.
sock.connect(addr);

// Subscribe to receive messages for a specific topic.
// This can be "rawblock", "hashblock", "rawtx", or "hashtx".
sock.subscribe('rawtx');

sock.on('message', function(topic, message) {

    if (topic.toString() === 'rawtx') {

        // Message is a buffer. But we want it as a hex string.
        var rawTx = message.toString('hex');

        // Use bitcoinjs-lib to decode the raw transaction.
        var tx = bitcoin.Transaction.fromHex(rawTx);

        // Get the txid as a reference.
        var txid = tx.getId();

        console.log('received transaction', txid, tx);

        // To go further you can get the address for a specific output as follows:
        // var address = bitcoin.address.fromOutputScript(tx.outs[0].script, bitcoin.networks.testnet);
    }
});

Copy/paste the above code into a new file named example.js.

In this example, we are listening for raw transactions. But there are other "topics" that you can subscribe to via zeromq:

• "rawblock" - Receive raw block data for new blocks.
• "hashblock" - Receive only the block hash for new blocks.
• "rawtx" - Receive raw transaction data for new transactions.
• "hashtx" - Receive only the transaction hash for new transactions.

Initialize a new npm project:

npm init

Hold down the <enter> key to accept all the defaults.

Install bitcoinjs-lib and zeromq modules via npm:

npm install bitcoinjs-lib zeromq

Run the example node.js application:

node example.js

Leave this running and open another terminal window (or tab).

To test whether everything is working properly, you will need to send a test transaction. Send some bitcoin using the command you tried earlier:

./bitcoin-cli -datadir=./.bitcoin sendtoaddress "ADDRESS" 1.000

If all is well, you should see something like the following printed in the terminal window of the example node application:

received transaction ffe5030e79e3092d6781095e61bf19996297fc5650bf8329ccb880e9c6141015 Transaction {
  version: 2,
  locktime: 101,
  ins: 
   [ { hash: <Buffer b4 0a 3f 90 c4 ab b5 f3 37 3a 22 ac d0 47 1a cd 5b 61 5a ea dd 50 7c 87 9d b5 7e e6 82 d6 da 46>,
       index: 1,
       script: <Buffer 16 00 14 d6 08 78 ca 69 0b 3e b9 3e f1 c8 8d 67 8b ea 8f 86 59 0d 59>,
       sequence: 4294967294,
       witness: [Array] } ],
  outs: 
   [ { value: 100000000,
       script: <Buffer a9 14 6c d8 77 4d e7 1c 89 91 f9 f3 c3 3b d2 de 65 bf 21 da 2c 64 87> },
     { value: 4799992920,
       script: <Buffer a9 14 2f 0f b0 dc ab 89 06 e8 e0 5e ef 7e 53 94 82 22 cb 78 82 29 87> } ] }

If you don't see the above output, then something is wrong. Double check that your bitcoind instance is still running. You can also check the debug log of bitcoind:

tail .bitcoin/regtest/debug.log

Remote Setup and Security Considerations

In a production environment, the bitcoin node will run on a separate server from your node.js application. Before you go and configure your ZeroMQ end-point to listen on 0.0.0.0, it's important to understand that the ZeroMQ protocol uses TCP so it lacks any encryption or authentication mechanism.

To ensure that the bitcoin transactions being streamed to your node.js application are authentic, it's a good idea to use port forwarding via an SSH tunnel. This will provide a strong layer of encryption and authentication. For a detailed guide to configuring such a setup see Secure Cloud Services via SSH Tunneling.

For most web-based business applications these days, it is necessary to run secondary services such as databases, search indexes, shared caches, and so on. Typically these services will be running on their own dedicated box (VPS or dedicated hardware). As a security best practice, these services should not be listening for incoming requests and also inbound requests should be blocked by a firewall.

This presents a problem: How to allow the applications that depend on these services to reach them? Enter port forwarding via SSH tunneling. This guide will walk you through the process of configuring and debugging such a setup so that you can have the best of both worlds.

Configuring the Service Server

First we will need to allow SSH tunneling on the service server. To do this we will create a dedicated user that is restricted to a sub-set of commands. We will further restrict the incoming SSH tunnels to only allow port-forwarding.

Create a new user ("remote_access") with the following command:

useradd --shell /bin/rbash --home-dir /home/remote_access --create-home remote_access;

• Use rbash (restricted bash) to limit the user to a small sub-set of commands.

Create an authorized keys file for SSH authorization:

mkdir -p /home/remote_access/.ssh;
touch /home/remote_access/.ssh/authorized_keys;
chown -R remote_access:remote_access /home/remote_access/.ssh;

Prepend each public key in the authorized keys file with the following:

no-pty,no-X11-forwarding,permitopen="localhost:3306",command="/bin/echo do-not-send-commands" 

This will restrict the remote user to port-forwarding on a specific port ("3306" in this example).

You can allow more than one port by specifying the permitopen option once for each port. For example, to allow ports 3306 and 8080:

no-pty,no-X11-forwarding,permitopen="localhost:3306",permitopen="localhost:8080",command="/bin/echo do-not-send-commands" 

Configuring the Application Server

Create a new RSA public/private key pair:

mkdir -p ~/.ssh;
ssh-keygen -f ~/.ssh/id_rsa -t rsa;

Print the new RSA public key with the prepended SSH options:

echo "no-pty,no-X11-forwarding,permitopen=\"localhost:3306\",command=\"/bin/echo do-not-send-commands\" $(cat ~/.ssh/id_rsa.pub)"

Copy/paste the output from the above command to the "remote_access" user's authorized keys file on the service server.

Now you can try connecting from the application server to the service server:

ssh -vNTL 3307:localhost:3306 remote_access@REMOTE_HOST

• -v Print verbose log messages.
• -N Do not execute a remote command.
• -T Disable pseudo-terminal allocation.
• -L Specifies that the given port (3307) on the local host is to be forwarded to the given host and port (localhost:3306) on the remote side.
  • The local and remote ports must be different otherwise the tunnel won't work. There are some work-arounds to this, but that is outside the scope of this article.

Have a look at the output. You should see something like the following:

debug1: Authentication succeeded (publickey).
Authenticated to REMOTE_HOST ([192.168.0.1]:22).
debug1: Local connections to LOCALHOST:3307 forwarded to remote address localhost:3306
debug1: Local forwarding listening on ::1 port 3307.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 3307.
debug1: channel 1: new [port listener]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Remote: PTY allocation disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Forced command.

If you see the following message:

debug1: No more authentication methods to try.
remote_access@bitcoind-1.cryptoterminal.eu: Permission denied (publickey).

This means that the server couldn't authenticate you as the "remote_access" user. Most likely you forgot to add or copied incorrectly your public key. Double check the "remote_access" user's authorized keys file.

If you see the following message while attempting to use the SSH tunnel:

channel 2: open failed: administratively prohibited: open failed

This means that the permitopen option is not set correctly. Remember that the second port number is the remote port that should be allowed given the following value for -L: 3307:localhost:3306.

If you are still having trouble, maybe take a break and come back to the problem later ;)

Automate the SSH Tunnel

Once you've got the SSH tunnel running, you can move on to configuring your application server to connect automatically on server start.

For managing the SSH tunnel (in case of network problems), it's a good idea to use autossh:

sudo apt-get install autossh

This will automatically handle reconnects in case the remote server is rebooted, or if either server temporarily loses connectivity.

To start the SSH tunnel whenever the application server starts, add a crontask:

crontab -e

And then add the following on a new line:

@reboot autossh -fnNTL 3307:localhost:3306 remote_access@REMOTE_HOST

• -f Requests ssh to go to background just before command execution.
• -n Redirects stdin from /dev/null (actually, prevents reading from stdin). This must be used when ssh is run in the background.

Conclusion

Now you have the best of both worlds: Your services are safe and secure behind a firewall while still permitting your dependent applications to reach them. This limits any potential damage that can be caused by a compromised application server. And as a bonus, communications between the two servers are encrypted via the SSH tunnel. Yay!

This guide will walk you through the steps needed to setup an SSH tunnel from a Windows machine using PuTTY. You can download PuTTY here. An SSH tunnel is useful for port-forwarding, in the case of connecting securely to a remote database server.

If you're using Linux or Mac, it's probably easier to simply use the terminal (or Secure Shell program) that is already installed on your system.

RSA key

If you already have a private RSA key, you will need to convert it to ppk format so that PuTTY can use it. To convert your RSA private key to this format you can use PuTTYgen (can be found here; search for "puttygen.exe" in that page). The program is simple enough to use, I don't think you'll have any problems with it.

If you do not yet have an RSA key pair (public and private), the above linked PuTTYgen tool can generate one for you.

Once you have your private RSA key as a ppk file, you can move on to the next step.

Configure PuTTY

There are a few configurations that you must set precisely to get the SSH tunneling working with your RSA key.

First screen:

• Host Name (or IP address) - Set the host name equal to USER@HOST; where USER is the user on the server with which you will be connecting and HOST is the IP address or hostname of the server.
• Close window on exit - This should be set to "Never". This allows you to debug the situation if a connection error occurs.

Next screen:

• Don't start a shell session or command at all - This should be checked.

Next screen:

• Click the Browse button and select the ppk file that you generated earlier.

Next screen:

• Add new forwarded port:
  • Source port - the port number on your local machine which all requests to will be forwarded through the SSH tunnel to the remote server.
  • Destination - the host and port number on the remote server to which requests will be forwarded. In this case, localhost:3306 which is where a remote server's MySQL server is listening.
  • Don't forget to click the Add button.

That's it! It might be a good idea to save the configurations you've set so that you can easily go back and change things if you need to debug a problem. To save your configurations:

• Go back to the main screen
• Give your session a name
• Save

Try Connecting

Now that everything is configured, try to click the Open button to initialize the connection to the server. You should see a black terminal window open up. Something like this:

To open the logs view, like shown above, you right-click the title bar of the window and click the menu item called Event Log.

The internet can be a hostile place, not just because of trolls and soul-crushing comments on youtube videos. Websites and internet-based services are being attacked and their users' personal information stolen by the millions. But we don't have to resign ourselves to being victims. We can protect ourselves. In this post, I will explain how you can minimize your risk and improve your security online.

Guard Your Personal Information

The best way to reduce the risk of your personal information being stolen, is to simply not give it out in the first place. Almost all websites and services these days will vacuum up as much personal user information as they can. This puts you at risk of your information being stolen by hackers. These hackers will re-package and sell your information to bad actors to be used in phishing scams, identity theft, and other nasty things. Practically speaking, it can be difficult to use a lot of websites and services without providing at least some personal information. Or is it?

If when you are filling out some form online (e.g for an order at an e-shop), certain information fields might be "required" for the form to be accepted (phone number, email, address, etc). However those fields might not actually be necessary to complete the order. It's very unlikely that they will really need your phone number or email address. And if they are not going to be shipping you a physical item by post, they don't need your real postal address either. So in these cases here are some tricks you can use:

• Email - Use a free, public email account from Mailinator. This is a quick, easy, and free way to get a real, functioning email address. This is nice because you don't have to sign-up or login to use it, and you can check all the emails that an account receives.
• Phone number - Input a fake, "real-looking" phone number. This is fine if you don't expect a phone-based account verification process and you don't need to receive text messages (SMS) from the website. If you need SMS capabilities, there are a few sites that allow you to receive SMS messages for free. These sites are like Mailinator, but for SMS:
  • www.receive-sms-online.info
  • smsreceivefree.com
• Postal Address - Look-up a real-world address in any location that makes sense. I like to use Google Maps for this because it's usually quick and easy.

You might be saying to yourself: "Wow, that's kind of extreme. And I don't like the idea of lying." Get over it. There's a battle happening every day for consumer's personal information. You can either be a victim or you can protect yourself.

Online Payments

If you're feeling adventurous, have a look at bitcoin. With bitcoin it is possible to transact on the internet without the need for banks or other intermediaries. This has many benefits, including protecting your privacy while making online purchases.

Otherwise if using a card to pay online (and you're in the United States), always use a credit card. The reason for this is that you are only liable for up to $50 in the case of fraudulent charges. Debit cards do not have this protection. If your debit card information is stolen and used fraudulently, the funds will be transferred out of your bank account and you will be forced to endure a lengthy process to have your money returned to you.

If you don't use a credit card, it might be a good idea to open a secondary checking account with a limited balance. This will allow you to transact online without exposing your full bank balance to the risk of fraudulant charges. Doing this likely will cost you some money every month in fees, but that could be a small price to pay for the added security and peace-of-mind.

Browser Choice and Setup

Use Firefox or Chromium (non-Googled version of Chrome) as your primary browser. On mobile, use Firefox because Chrome on mobile does not allow extensions to be installed. Must-have browser extensions:

• uBlock Origin (for Chrome and Firefox) - Blocks ads and most third-party tracking services. As a bonus it also makes pages load faster, reduces data usage, reduces power consumption, reduces visual clutter on the screen (so reading articles is easier), protects your privacy, blocks potential drive-by virus installers and other malicious code served by ad-networks.
• HTTPS Everywhere (for Chrome and Firefox) - Forces encrypted communications with many major websites.

Always use private browsing mode; in Chrome this is called "Incognito" mode. The reasoning here is that your browser will forget everything you've done when you close it. Yes, that means you will have to login to websites everytime you open your browser. But this also means that your accounts are protected against a whole class of browser-based vulnerabilities.

You may also want to consider disabling JavaScript in your browser. This makes pages load faster, saves your device's battery, and blocks most attack vectors. The down-side is that some website functions won't work, but you can easily white-list these sites as needed.

Chrome Settings

Firefox already does a good job of providing reasonable defaults to protect your privacy; Chrome not so much. Here are some settings you may want to change:

• Block third-party cookies - Can be found at chrome://settings/content/cookies
• The following settings under "Advanced" in chrome://settings:
  • "Use a web service to help resolve navigation errors"
  • "Use a prediction service to help complete searches and URLs typed in the address bar"
  • "Use a prediction service to load pages more quickly"
  • "Automatically send some system information and page content to Google to help detect dangerous apps and sites"
  • [ ✓ ] "Protect you and your device from dangerous sites"
  • "Automatically send usage statistics and crash reports to Google"
  • "Send a 'Do Not Track' request with your browsing traffic"
  • "Use a web service to help resolve spelling errors"
• Disable autofill - Can be found at chrome://settings/autofill
• Disable manage passwords - Can be found at chrome://settings/passwords

Use a Password Manager

Using a password manager is a critical step towards improving your security online. Some of the benefits of a password manager include:

• One password to remember - The only password you will need to remember is your password manager's master password.
• Stronger passwords - Humans are bad at creating strong, random passwords. This is why it's better to let your password manager do this for you.
• Unique password for each site - Since you don't need to remember them, you can use a unique password for each website. This is important because when a website is hacked, your accounts for other sites are not at risk of being compromised.

Recommended password managers:

• KeePassX
  • Available for Windows, Mac, and Linux
  • Free
  • Requires your own backup and syncing scheme
  • For Android support try one of the following apps: KeePassDroid, Keepass2Android
• 1Password
  • Available for Windows, Mac, Android, and iOS
  • $3 per month
  • Cloud-based backup and sync between devices

Two-factor Authentication

For your most important accounts (primary email, online banking, etc), you should think about enabling two-factor authentication (2FA). This typically involves a secondary device (e.g your phone) which generates a random code every 30 seconds that must be used in addition to your account password to login. This dramatically improves your account security because an attacker would have to know your password and also have access to the device that generates your second-factor authentication codes.

SMS-based 2FA should be avoided. Never link your phone number as a "backup/recovery" method on any of your accounts. Phone numbers can be hijacked via social engineering. A determined attacker can call your phone company, convince some unmotivated customer service representative that they are you, and then switch the phone number on your account to a new SIM card. Once they do that, they will be able to gain access to your accounts via the account recovery mechanism using a code sent via SMS.

Recommended 2FA apps:

• Google Authenticator (for Android and iOS)
• Authy

Conclusion

That's not everything, but it's a solid start to improving your defense posture online. Don't worry if you can't (or don't want to) change all of your habits right now. Pick a couple things today and do those. Maybe come back and add a few more in the future.

Until next time, good luck!

This article assumes that your current password storage mechanism involves some irreversible hashing function. In this case we are forced to migrate each user to the new storage algorithm when they login successfully, because we need the original plain text password to generate a new hash. If you're storing user passwords as plain text, or you're using some reversible encryption scheme, you can migrate all your users' password right away without the need for this article.

Here's a simplified example users table from MySQL:

+--------------+------------------+------+-----+---------+----------------+
| Field        | Type             | Null | Key | Default | Extra          |
+--------------+------------------+------+-----+---------+----------------+
| id           | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| username     | varchar(255)     | NO   | UNI | NULL    |                |
| password     | varchar(255)     | NO   |     | NULL    |                |
+--------------+------------------+------+-----+---------+----------------+

In the above example table:

• The id field is an auto-incremented integer field unique to each user record.
• The username field is a variable-length text field which must be unique to each user record.
• The password field is a variable-length text field that will contain the hash of each user's password.

We will need to add another field named pw_storage_algorithm:

+-----------------------+------------------+------+-----+---------+----------------+
| Field                 | Type             | Null | Key | Default | Extra          |
+-----------------------+------------------+------+-----+---------+----------------+
| id                    | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| username              | varchar(255)     | NO   | UNI | NULL    |                |
| password              | varchar(255)     | NO   |     | NULL    |                |
| pw_storage_algorithm  | varchar(255)     | YES  |     | NULL    |                |
+-----------------------+------------------+------+-----+---------+----------------+

The default value (empty or null) for this new field is assumed to be your current password storage algorithm.

Your current login logic might look something like this (as pseudo-code):

Search for user record in database (by username or email).

If user exists:
    If currentPasswordStorageAlgorithm(password) equals user.password:
        Login success.
    else
        Login failure.

Else:
    Login failure.

Modify the password checking logic in your application to take into account the new field:

Search for user record in database (by username or email).

If user exists:
    If user.pw_storage_algorithm is "new_algorithm":
        If newPasswordStorageAlgorithm(password) equals user.password:
            Login success.
        Else:
            Login failure.
    Else:
        If legacyPasswordStorageAlgorithm(password) equals user.password:
            Set user.pw_storage_algorithm equal to "new_algorithm".
            Set user.password equal to newPasswordStorageAlgorithm(password).
            Save the user record.
            Login success.
        Else:
            Login failure.

Else:
    Login failure.

Notice the branch where we check the password using the legacy password storage algorithm. If the check is successful we hash the plain text password using the new password storage algorithm, then store this hash in the database. With this setup, whenever a user successfully logs in, the stored password will be swapped for the new algorithm. The next time the user logs in, the new password storage algorithm will be used to check their password.

That's it! Good luck and don't forget to add some tests ;)

Almost every Monday for the past year, I've been organizing a programming meetup in Prague. I tried multiple different formats and teaching styles. I bumbled through some poorly thought-out presentations and had some great sessions where I think everyone learned and had fun. In the past few months it has settled in to a smooth stride, and I am finally feeling good about it. So with over 50 individual meetups hosted and hundreds of attendees, now is probably a good time for a little reflection.

I believe my original motivation for the group was to meet smart, interesting people with whom I could work on some fun projects. At the time I was suffering from a personal lack of motivation and needed a change to get things moving in the right direction again. With that said, I think it might be helpful if I dive into a few points which could help others who are considering organizing a similar group.

Pick a Theme

It's not possible to be everything to everyone. Instead focus on an area where your personal strengths will be of most use.

My group is focused on full-stack web programming, specifically Node.js and JavaScript, because this is where my hard skills are strongest. I also felt that it was going to be important to have a unique theme for the group. I tried to think back to how I started programming. Being a self-taught programmer, I learned by working with open source software. Modify an existing thing to change how ti works or to create something new. I settled on the name "Learning by doing".

Be Clear

Describe the meetup clearly enough so that potential attendees can self-select for the right fit. This is especially important if your space is limited. It's not fun when someone shows up to a meetup expecting something completely different. Though, this will happen sometimes no matter what you do.

Interactive Learning > Talking at People

I don't know about you, but I dislike lectures where the presenter drones on and on about a topic without any kind of interaction with their audience. If it's not possible to have a fully interactive presentation, where the attendees can follow along with some real programming tasks, then at least stop for the occasional Q&A.

Go Slowly

This is something I still struggle with, but I think I am doing much better these days. It's easy to forget how complex some programming concepts are, once you've already crossed the threshold of understanding. Have empathy for those who are struggling. A good technique is to stop and check for raised hands or confused faces.

Try to Get Others Involved

Don't try to do everything yourself forever. The people who show up regularly would probably love to help out somehow. Talk to them. Ask them what their goals are, and if they would like to be more involved.

In my group, some of the regulars are beginning to prepare their own presentations. I have had to be very persistent in asking attendees if they would like to present a topic sometime. The difficulty in finding presenters is mostly due to lack of time on their part, which is understandable.

Onward

Overall it's been a great experience. I've met a bunch of nice people, and it has pushed me to improve my soft skills. I look forward to seeing where it goes.

Cheers!

Since you're reading this, you probably have a pile of sound files in a web application wondering if there's a better way. And since you already know how image sprites work, you got to thinking that maybe you could do the same thing for audio. Well, you're in luck. You can! And, with a bit of third-party-library magic, it's possible across all the major browsers.

Right off the bat, here's a working demo of an audio sprite:

Each button plays a different sound, but all from a single audio file. This is done by specifying an offset and a length for each individual sound that is contained within the audio sprite file. When you want to play a particular sound, the audio file begins playing at that sound's offset and finishes playing after a specific amount of time specified by that sound's length.

If you inspect the source of the demo, you will notice that I am using the createjs sound library for audio playback. SoundJS does not support audio sprites, as I was attempting to do, so I had to create a wrapper class to handle this part. Here is a sample of the wrapper class in use:

// Create an instance of the wrapper class.
var Sound = new SoundManager({
	// The base URI path for downloading sound files.
	basePath: 'sounds/',
	// Define all the sounds we want to use in our application.
	// You can reference separate audio files, or the same audio file for each sound.
	sounds: [
		{ fileName: 'all.ogg', offset: 0, length: 115, id: 'sound1' },
		{ fileName: 'all.ogg', offset: 1000, length: 205, id: 'sound2' },
		{ fileName: 'all.ogg', offset: 2000, length: 524, id: 'sound3' }
	],
	// Fallback file extension(s).
	alternateExtensions: ['m4a']
});

// Listen for some user event to trigger a sound.
var button2 = document.getElementById('play-sound-button-2');
button2.addEventListener('click', function(evt) {
	Sound.play('sound2');
});

Note the offset value of each proceeding sound. I have added some padding between each sound so that it is less likely that the audio playback from one track will overlap with the next one. The reason for this is the inaccuracy of JavaScript's setTimeout function. This is also why I've included a custom setTimeout function in the wrapper class:

SoundManager.prototype.setTimeout = function(callback, delay) {
	var start = Date.now();
	var end = start + delay;
	var expected = start + interval;
	var interval = 50;
	setTimeout(function step() {
		var now = Date.now();
		if (now > end) {
			// Enough time has passed.
			// Execute the callback.
			return callback();
		}
		var drift = now - expected;
		expected += interval;
		setTimeout(step, Math.max(0, interval - drift));
	}, interval);
};

The above function creates a loop internally which checks and adjusts for the drift since the last loop. It will continue until the original delay time has passed, then it will execute the callback function it was given. The isn't a perfect solution, because there could still be significant delays in the event loop caused by slowly executing code somewhere else in the application.

Well, that's about it for audio sprites. Have some noisy fun out there!

Have you ever needed to manage your remote MySQL databases, and ended up settling on the less-than-ideal setup of having an instance of phpMyAdmin on the same server as the MySQL server? Well, I am about to make your day. I am going to show you how to manage any number of remote MySQL databases from your local instance of phpMyAdmin; without compromising on security.

This tutorial assumes you've already got phpMyAdmin up and running locally. If you need help with that, there are plenty of articles floating around to guide you.

SSH Tunnel

It is not absolutely necessary, but I suggest setting up Passwordless SSH between your local machine and the remote server. This will allow you to run the SSH tunnel as a background process.

We will be using SSH to set up a connection to our remote host, through which all requests to a specific port locally will be forwarded to a specific port on the remote machine. This is where the magic happens:

ssh -fNL 3307:localhost:3306 root@REMOTE_HOST

Replace REMOTE_HOST with the IP address (or host name) of the remote server.

Let's break that command down a bit:

• -f Requests ssh to go to background just before command execution. Useful for having the SSH tunnel run in the background. If you are using a password to connect to the remote server, you'll want to remove this argument.
• -N Do not execute a remote command. This is useful for just forwarding ports.
• -L Specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side.

To verify that the SSH tunnel was started successfully, run the following command:

ps aux | grep ssh

You should see the ssh command you entered earlier in this list of processes.

Configure phpMyAdmin

Now that we've got the SSH tunnel running in the background, we can configure phpMyAdmin to connect to the remote machine over the tunnel. Edit the following configuration file:

sudo vim /etc/phpmyadmin/config.inc.php

Add the following to the end of the file:

# Add the following after all the existing server configurations:
$cfg['Servers'][$i]['verbose']       = 'Local';
$cfg['Servers'][$i]['host']          = 'localhost';
$cfg['Servers'][$i]['port']          = '3306';
$cfg['Servers'][$i]['connect_type']  = 'tcp';
$cfg['Servers'][$i]['extension']     = 'mysqli';
$cfg['Servers'][$i]['compress']      = FALSE;
$cfg['Servers'][$i]['auth_type']     = 'cookie';
$i++;

$cfg['Servers'][$i]['verbose']       = 'Remote Server 1';// Change this to whatever you like.
$cfg['Servers'][$i]['host']          = '127.0.0.1';
$cfg['Servers'][$i]['port']          = '3307';
$cfg['Servers'][$i]['connect_type']  = 'tcp';
$cfg['Servers'][$i]['extension']     = 'mysqli';
$cfg['Servers'][$i]['compress']      = FALSE;
$cfg['Servers'][$i]['auth_type']     = 'cookie';
$i++;

Save the file. Now, open up your browser and point it at your local phpMyAdmin instance. If you modified the phpMyAdmin config file correctly, you should now see a select drop down at the login screen. This will list both of the servers: "Local" and "Remote Server 1". You can add any number of servers to phpMyAdmin in this way.

Now, finally, the moment of truth. Try logging in with a valid MySQL user on your remote server. If all is well, you should be logged in to the account and should see all the databases to which that user has access.

In this post I will walk you through the process of setting up a scheduled, automatic remote database backup on Linux.

If you haven't already done so, you'll need to set up passwordless SSH from the Server with the database(s) to the Server that will be storing the database backup files.

Create New MySQL User to Perform Backups

This step is to be performed on the server with the database(s).

To be on the safe side, it's a good idea not to use the root MySQL user to perform the database backups. So, let's create a new user with just enough privileges to create backups:

CREATE USER '__DB_USER__'@'localhost' IDENTIFIED BY  '__DB_PASSWORD__';

GRANT SELECT , 
RELOAD , 
FILE , 
SUPER , 
LOCK TABLES , 
SHOW VIEW ON *.* TO '__DB_USER__'@'localhost' IDENTIFIED BY  '__DB_PASSWORD__' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0;

FLUSH PRIVILEGES;

Replace __DB_USER__ with the name for your new user.

Replace __DB_PASSWORD__ with a strong, unique password.

Create Bash Script

This step is to be performed on the server with the database(s).

Create a new file called backup.sh:

sudo vim /home/backup.sh

Add the contents of the following Gist to your new file:

mysql_backup.sh

Read the comments for instructions on how to use the script.

Set strictest possible permissions; so that the only thing that can be done is script execution by the file owner:

sudo chmod 0100 backup.sh

Create directory to store the database backups:

cd /home
sudo mkdir backups
cd backups
sudo mkdir db

Test the backup script:

sudo /home/backup.sh

If it worked, you should see a .sql.gz file for each of the databases you specified in the backup script.

Add Cron Task

This step is to be performed on the server with the database(s).

Finally, you'll need to add a cron task to execute the script on a set schedule. Edit crontab for current the user:

crontab -e

If you've never used crontab on the current server before, it will ask you which editor you would like to use. I prefer vim, but you can use whichever you wish.

Append the following to the end of the crontab file:

30 2 * * * sudo /home/backup.sh

This will run the backup script daily at 2:30 AM (server time). For more info on crontab: https://en.wikipedia.org/wiki/Crontab_(Unix_command.

It might be a good idea to verify your server time with the following command:

date

The output should be similar to the following:

Wed Jul 24 23:12:13 EDT 2013

On Ubuntu you can use the following to set the timezone temporarily and get instructions on how to set it permanently:

tzselect

If you need help setting the timezone on your linux distro:

https://duckduckgo.com/?q=how+to+set+timezone+on+linux+server

Additional Resources

These are not necessary to perform the remote, automatic backup, but you may find them useful.

prune_backups.sh - Automatically delete old backup files

There are a number of use cases where by logging in via SSH (without a password) is the best (or maybe only) option. For example, if you wanted to run an automated backup on a remote server that would upload files to another remote server via scp, you would need SSH to work without a password. This post will guide you through the process of setting up passwordless SSH, such that you will be able to use scp and other utilities that rely upon SSH for authentication without having to use a password.

Step 1: Generate a Private/Public Key Pair

Open up a terminal window on the machine from which you wish to access a remote machine.

Check to see if you already have an RSA key for the current user:

cd ~/.ssh
ls -l

If the following files already exist, then you can skip this step:

-rw------- 1 user group 1502 Jul  18  2013 id_rsa
-rw-r--r-- 1 user group  360 Jul  18  2013 id_rsa.pub

If you don't already have the files, run the following command:

ssh-keygen -f ~/.ssh/id_rsa -t rsa

When prompted to enter a pass phrase, just hit enter (both times). This will allow the use of the RSA key without a password.

Step 2: Copy the Public Key to the Remote Server

Use scp to transfer your public key file to the remote server that you wish to access:

scp ~/.ssh/id_rsa.pub user_name@ip_address:~/

Replace user_name with the username you usually use to SSH into the remote server. You'll be prompted for this user's password.

Replace ip_address with the IP Address of the remote server that has the database export on it.

Step 3: Add Public Key to Remote Server's Authorized Keys File

Secure shell (SSH) into the remote server:

ssh user_name@ip_address

Replace user_name with the username you usually use to SSH into the remote server. You'll be prompted for this user's password.

Replace ip_address with the IP Address of the remote server that has the database export on it.

Append the Public Key to the remote server's authorized keys file:

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

Set the appropriate permissions on the authorized keys file:

chmod 0644 ~/.ssh/authorized_keys

Delete the Public Key file, just to tidy up:

rm ~/id_rsa.pub

Step 4: Test

Open a new terminal window, and try to secure shell into the remote server again:

ssh user_name@ip_address

You should no longer be prompted for a password and the secure shell session should start right away.

I recently I needed to organize the controllers of a CodeIgniter instance into sub-sub folders. By default, CodeIgniter only allows routing to sub folders in the controller directory:

application
-- controllers
---- api
------ users.php
------ items.php
---- welcome.php

With the above structure, you'd access your welcome controller at the following URI:

/welcome

And, you'd access the users API controller at this URI:

/api/users

The problem comes when you want to have a deeper controllers directory structure:

application
-- controllers
---- admin
------ api
-------- users.php
-------- items.php
------ home.php
------ shop.php
---- shop
------ api
-------- cart.php
------ catalog.php
---- blog
------ api
-------- article.php
-------- comments.php
------ home.php

By default, CodeIgniter will not route requests for the following URIs correctly:

/admin/api/item
/shop/api/cart
/blog/api/comments

How to Make it Work

First, if you have not already, you'll need to add mod_rewrite rules to your site's webroot to allow the routing of requests to your CodeIgniter's index.php file:

<IfModule mod_rewrite.c>

  Options +FollowSymLinks
  RewriteEngine On
  RewriteBase /

  # If your default controller is something other than 'welcome' you should probably change this.
  RewriteRule ^(welcome(/index)?|index(\.php)?)/?$ / [L,R=301]
  RewriteRule ^(.*)/index/?$ $1 [L,R=301]

  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteRule ^(.*)$ /index.php/$1 [L]

  SetEnvIfNoCase X-Forwarded-For .+ proxy=yes
  SetEnvIfNoCase X-moz prefetch no_access=yes
   
  # Block pre-fetch requests with X-moz headers.
  RewriteCond %{ENV:no_access} yes
  RewriteRule .* - [F,L]

  # Fix for infinite redirect loops.
  RewriteCond %{ENV:REDIRECT_STATUS} 200
  RewriteRule .* - [L]

</IfModule>

The above rules tell Apache to internally route all requests through the index.php file, but only for requests that do not match an existing file or directory. Once you've got rewrite rules in place and working, you can move on to the sub-sub-folder magic.

The Magic

You will need to extend the core Router class of CodeIgniter. All you have to do is add one file, application/core/MY_Router.php:

<?php  if ( ! defined('BASEPATH')) exit('No direct script access allowed');

/*
	Extended the core Router class to allow for sub-sub-folders in the controllers directory.
*/
class MY_Router extends CI_Router {

	function __construct()
	{
		parent::__construct();
	}

	function _validate_request($segments)
	{
		if (count($segments) == 0)
		{
			return $segments;
		}

		// Does the requested controller exist in the root folder?
		if (file_exists(APPPATH.'controllers/'.$segments[0].'.php'))
		{
			return $segments;
		}

		// Is the controller in a sub-folder?
		if (is_dir(APPPATH.'controllers/'.$segments[0]))
		{
			// Set the directory and remove it from the segment array
			$this->set_directory($segments[0]);
			$segments = array_slice($segments, 1);

			while (count($segments) > 0 && is_dir(APPPATH.'controllers/'.$this->directory.$segments[0]))
			{
				// Set the directory and remove it from the segment array
				$this->set_directory($this->directory . $segments[0]);
				$segments = array_slice($segments, 1);
			}

			if (count($segments) > 0)
			{
				// Does the requested controller exist in the sub-folder?
				if ( ! file_exists(APPPATH.'controllers/'.$this->fetch_directory().$segments[0].'.php'))
				{
					if ( ! empty($this->routes['404_override']))
					{
						$x = explode('/', $this->routes['404_override']);

						$this->set_directory('');
						$this->set_class($x[0]);
						$this->set_method(isset($x[1]) ? $x[1] : 'index');

						return $x;
					}
					else
					{
						show_404($this->fetch_directory().$segments[0]);
					}
				}
			}
			else
			{
				// Is the method being specified in the route?
				if (strpos($this->default_controller, '/') !== FALSE)
				{
					$x = explode('/', $this->default_controller);

					$this->set_class($x[0]);
					$this->set_method($x[1]);
				}
				else
				{
					$this->set_class($this->default_controller);
					$this->set_method('index');
				}

				// Does the default controller exist in the sub-folder?
				if ( ! file_exists(APPPATH.'controllers/'.$this->fetch_directory().$this->default_controller.'.php'))
				{
					$this->directory = '';
					return array();
				}

			}

			return $segments;
		}


		// If we've gotten this far it means that the URI does not correlate to a valid
		// controller class.  We will now see if there is an override
		if ( ! empty($this->routes['404_override']))
		{
			$x = explode('/', $this->routes['404_override']);

			$this->set_class($x[0]);
			$this->set_method(isset($x[1]) ? $x[1] : 'index');

			return $x;
		}


		// Nothing else to do at this point but show a 404
		show_404($segments[0]);
	}

	function set_directory($dir)
	{
		// Allow forward slash, but don't allow periods.
		$this->directory = str_replace('.', '', $dir).'/';
	}

}

/* End of file MY_Router.php */
/* Location: ./application/core/MY_Router.php */

That's it! CodeIgniter will automatically include this file and instantiate the class within. You should now be able to organize your controllers into as many sub folders, sub sub folders, sub sub sub folders as you like.

I recently had the need to manage multiple host names within a single instance of CodeIgniter. Setting the virtual hosts to all point to the same directory in the web root was the easy part. Intelligently routing the requests, once they got to CodeIgniter, such that I can be sure a domain has its own group of controllers, not so much.

To better illustrate what I am going for, let's say you had 3 different host names that you wanted to all go to your one instance of CodeIgniter:

domain-name.com
admin.domain-name.com
shop.domain-name.com

Now let's say you want to route requests for each of these host names to their own group of controllers:

application
-- controllers
---- admin
------ api
-------- users.php
-------- items.php
------ home.php
------ shop.php
---- shop
------ api
-------- cart.php
------ catalog.php
---- home.php

The Solution

Hooks! The way I made all of this work was by utilizing CodeIgniter's hook system to trick CodeIgniter's routing methods into mapping a request for a specific domain to its corresponding controller group. I also had to add an additional hook to restore the original URI information from the actual request; so that I wouldn't have to rewrite any of the code in my controllers.

I moved the code to a Gist:

HostNameRouter.php

The comments in the files of the Gist should be sufficient to guide you with the set up / configuration.

I had to go one step further for this solution to work 100% for my situation; I needed controllers in sub-sub folders to work.

If all is well, you should now be able to assign any number of host names to your single instance of CodeIgniter and have those requests routed to the controller group of your choice. Fun, huh?

This article contains a few useful things that can be done in terminal; with explanations of what they do and why. I'll continually add to this list as I learn more.

Exclude SVN and GIT Files While Backing Up a Directory

If you're like me, you have quite a few projects that use a mix of SVN and GIT version control systems. You may want to exclude all those hidden SVN and GIT files from your backup:

sudo tar --exclude-vcs -zcvf ./backup.tar.gz /path/to/projects/directory

Limit Download Speed for Large File Downloads

If you've ever needed to download a very large file, but didn't want to completely tie up your internet connection's bandwidth, you're going to love this:

cd ~/Downloads
wget --limit-rate=500k http://url-of-file-to-download

wget does a GET request to a specified URL. The --limit-rate option sets a maximum download speed for this download.

Output Result of a Terminal Command to a File

Sometimes when using terminal, a command will generate such a large amount of output that it gets cut-off towards the beginning. The way to view the full output is by writing it to a file:

ls -l > output.txt

Now if you open output.txt in gedit you should see a list of all the visible files and directories within the current user's home directory.

Pipe the Output of One Command into Another

Sometimes it's helpful to chain commands together, like this:

ls -l | grep somefile

ls lists the contents of the current directory. And, grep does pattern matching on the string that is passed to it. The net result of this combination is finding all files in the current directory that at least partially match the text somefile

Search Running Processes by Keyword

Here we're going to pipe the output of the ps aux command - which lists all running processes - into grep with a keyword, to retrieve a list of running processes that matches our keyword:

ps aux | grep ssh

Where ssh is our keyword.

Manually Expire/Exit Sudo Session

After you've used sudo to execute a terminal command, the sudo session will persist for some time after. What this means is that you won't be prompted for the password for the root user when using sudo during this session. To expire the sudo session do the following:

sudo -k

Exit su Session

To manually exit from a su - someuser session:

exit

For the most recent stable release of CodeIgniter (2.1.3), there is a rather annoying simultaneous request problem that will kill active sessions. You might have experienced this yourself if you had a website or application with lots of AJAX requests or other simultaneous requests. The tell-tail sign was that your users would be logged out after the update session time had passed (5 minutes by default).

The solution that seemed to work for most folks involved extending the session library with your own MY_Session.php library that overwrote the sess_update method with one that only executed the update method when not an AJAX request:

<?php if ( ! defined('BASEPATH')) exit('No direct script access allowed');

require_once BASEPATH . '/libraries/Session.php';

class MY_Session extends CI_Session
{

	function __construct()
	{
		parent::__construct();

		$this->CI->session = $this;
	}

	function sess_update()
	{
		// Do NOT update an existing session on AJAX calls.
		if (!$this->CI->input->is_ajax_request())
			return parent::sess_update();
	}

}

/* End of file MY_Session.php */
/* Location: ./application/libraries/MY_Session.php */

You can either auto-load this library from config/autoload.php:

$autoload['libraries'] = array( 'MY_Session');

Or, you can load it later:

$this->load->library('MY_Session');

If you have already installed GIMP 2.8+, then you're off to a good start. Now we're going to change a couple things from the default configuration to make GIMP a little bit more useful and intuitive; or at the very least, more like the image editing programs we've become accustomed to.

Layer Effects

This is a big one. First, you'll need to download layerfx for GIMP 2.8. Yes, for some reason it opens as a text file in your browser instead of downloading as a file. That's ok. Left click once on the text (anywhere). Select all (CTRL + A) and copy (CTRL + C).

Note: The reason I suggest using the python version (.py) is because it allows Previewing.

Now, open up terminal. Create a new python plugin in your GIMP plugins directory:

cd ~/.gimp-2.8/plug-ins
sudo gedit layerfx.py

Paste the contents of the layerfx python plugin (CTRL + V). Save and close the file.

Now back in the terminal window, set the permissions on the layerfx plugin file to be executable:

chmod +x layerfx.py

Finally, open GIMP. If you had it open already, close it then reopen it.

If you installed the plugin correctly, you should be able to reach the layer effects menu at Layer -> Layer Effects.

For more info about the Layer Effects plug-in, see:

http://registry.gimp.org/node/186

Select Layer with Left-Click

If you have ever had to chop up a layered image composition for a website or GUI, you know why this is useful. Being able to left click on a layer to select it can save you tons of time that would otherwise have been wasted searching through hundreds of layers manually.

In GIMP, go to Edit -> Preferences. Now go to Tool Options. At the bottom of this screen, you'll see a check box with the following text next to it:

Set layer or path as active

Check this box and hit "OK."

Now when you are using the Move Tool and left click somewhere in an image, the layer you just clicked will become the selected / active layer.

Do you use console.log() to debug your JavaScript? Well, I've got news for you.. console.log() breaks Internet Explorer; even if all instances of it are commented out! A quick check you can do to see if console.log() is indeed the source of your problem, is to open the console in Internet Explorer and see if the problem goes away. The reason for this is that console.log() does not exist in Internet Explorer unless the developer tools are active. So, for regular IE users,console.log() is a show stopper.

The Solution

You can either make sure you don't ship your JavaScript with any instances of console.log() in it, or you can place the following code near the top of your JavaScript to ensure that it exists (albeit non-functional):

if ( ! window.console ) console = { log: function(){} };

Update

Since this article was written, there is a new initiative in broadening the adoption of HTTPS/SSL across the web: LetsEncrypt. They provide free, automated SSL certificates that actually work.

Defining the Problem

I encountered a peculiar problem with my signed SSL certificates the other day. In the latest versions of Firefox and Chrome, the SSL certificate was being trusted and worked just fine. However, in Chrome in iPad (and likely other browsers with similarly limited capabilities), the certificate was deemed "untrusted."

I ran an SSL Test on the domain with which I was having the problem. This yielded a bit of very useful information:

Chain issues     Incomplete

This gave me what I needed to further debug the problem. I discovered that I needed to have the server send a "Certificate Chain" with the initial SSL hand shake in order for browsers that do not support "certificate discovery" to find the root certificate.

For additional information, see Intermediate Certificate Authorities.

Fixing the Problem

First, you will need to search your Certificate Authority's (CA) website to download their Intermediate CA (or "Certificate Chain") file. This file will contain the concatenated chain of trusted CA certificates needed to reach the root certificate. Once you find and download the chain file, you will need to upload it to your server and configure your web server to provide it to clients during SSL handshakes.

Configuring nginx

In your site's server configuration add the following:

ssl_certificate /path/to/full-certificate-chain.pem;
ssl_trusted_certificate /path/to/certificate-chain.pem;

• certificate-chain.pem - This file should contain the CA certificate chain (in descending order).
• full-certificate-chain.pem - This should contain the same chain of certificates as the certificate-chain.pem file, but with the addition of your site's certificate at the top of the file.

After you're done modifying your site's configuration file, test the changes with the following command:

nginx -t

If everything's ok, it's safe to reload all nginx configurations:

service nginx reload

It's not necessary to do a full restart.

Configuring Apache

In your website's virtual host for port 443, edit the following line:

#SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt

Uncomment the line and have it reference the path to the chain file you just uploaded to the server:

SSLCACertificateFile /path/to/full-certificate-chain.pem;

• full-certificate-chain.pem - This file should contain your site's certificate along with every CA's certificate added in descending order.

Restart Apache.

Is the Problem Fixed?

Run the SSL Test again. If all is well, the chain issue should be resolved. If not, you can further debug the problem by using the following command in a terminal window on your local machine (not on the server with the SSL issue):

openssl s_client -showcerts -verify 32 -connect domain-name:443

• You will need to have openssl installed to run this command
• Be sure to replace domain-name with the domain that is having the SSL issue

The output of this command is quite dense and can be difficult to sort through. You will want to first find the top of the output, and then search for something like this:

verify error:num=20:unable to get local issuer certificate

In this tutorial I will walk you through the process of creating and restoring a database backup from Terminal in Ubuntu.

Create a Database Backup

We're going to use mysqldump to create a backup of the database, and we're going to compress the backup with gzip.

mysqldump -u user_name -h localhost -p database_name | gzip -9 > backup_name.sql.gz

Replace user_name with your MySQL user name.

Replace database_name with the name of the database you wish to export.

Restore the Database Backup

From the terminal of the server to which you wish to import the database backup, use the following command:

zcat /path/to/database/backup/backup_name.sql.gz | mysql -u user_name -p database_name

Replace user_name with your MySQL user name. You'll be prompted for this user's password.

Replace database_name with the name of the database you wish to import.

That's it!

Now, if you are feeling extra fancy, I've included additional instructions for deploying a database backup from one remote server to another:

Deploying a Database to Another Server

Exporting and importing your database backups via terminal is also useful if you need to deploy a copy of a database from one server to another. Since you've already done the export on one server, let's focus on the server to which you want to import the database.

Transferring the Database Backup

Do NOT use File Transfer Protocal (FTP). FTP is fun and all, but here we're going to use a more efficient mechanism. Secure Copy has a number of benefits over the lesser methods of transferring files:

• One Less Trip - No need to download from one server just to upload to another.
• Faster Transfers - Servers tend to have good internet connections. Once your computer is out of the equation, the transfer rate is going to be orders of magnitude better.
• It's Secure - Not only does it have secure in the name, but it also uses SSH for authentication and data transfer.

So let's get down to it then. From the server to which you wish to import the database, use the following command:

sudo scp user_name@ip_address:/file/path/to/backup/backup_name.sql.gz ./

Replace user_name with the username you usually use to SSH into the remote server. You'll be prompted for this user's password.

Replace ip_address with the IP Address of the remote server that has the database export on it.

Replace /file/path/to/backup/backup_name.sql.gz with the full file path of the database export on the remote server.

Now that you've got the database backup on the second server, all you have to do is Restore the Database Backup using the instructions from earlier in this tutorial.

Cross browser testing with Internet Explorer and Windows just got a whole lot easier. In this article I'll walk you through the full process of downloading, installing, and configuring a Windows Virtual Machine.

Get the Virtual Machine

Since this is a rather large file, you may want to limit the download speed for this particular download, so that it doesn't use up all of your bandwidth. From terminal run the following:

cd ~/Downloads
wget --limit-rate=500k http://virtualization.modern.ie/vhd/IEKitV1_Final/VirtualBox/Linux/IE9_Win7.zip

If the direct download link above does not work, you can get a fresh one here.

Once the download is complete, you'll need to "unzip" the zip archive. I use quotes there because I had issues using unzip on the file. I had to use 7z to extract it. If you don't have this command, you'll need to install p7zip-full:

sudo apt-get install p7zip-full

Now you should be able to extract the zip file:

7z e IE9\ -\ Win7.zip

Once that's done, you should have a file named IE9 - Win7.ova

Install VirtualBox

First you will need to download VirtualBox. Then you'll need to install it by running the .deb.

Alternatively, you can get any previous version of VirtualBox and for many different platforms here.

Set Up the Virtual Machine

1. Start up VirtualBox
2. Go to File
3. Then Import Appliance
4. Click Open Appliance
5. Select the IE9 - Win7.ova file
6. Click Next

This should take a little while. Once it's done, start the Virtual Machine.

Fitting the VM to Your Screen Resolution

To make using the Virtual Machine bearable, you'll want to have it fit your screen resolution. To do this, you'll need to install Guest Additions in the Guest OS.

Download the guest additions ISO for your version of VirtualBox. It should be in the directory for your version. If you do not know what version of VirtualBox you're running:

1. Open VirtualBox
2. Go to Help
3. Then About VirtualBox

Installing Guest Additions

1. In VirtualBox, click on the IE9 - Win7 virtual machine.
2. Go to Settings
3. Go to Storage
4. Under Controller: IDE Controller, click Add CD/DVD Device
5. Select the Guest Additions ISO you just downloaded
6. Click OK to save the settings
7. Start the Virtual Machine, if it isn't already
8. In Windows, go to Start
9. Then, Computer
10. You should see the VirtualBox Guest Additions CD under Devices with Removable Storage
11. Open the VirtualBox Guest Additions CD
12. Run the VBoxWindowsAdditions installer
13. Follow the instructions in the installer
14. When the installation is complete, you'll need to restart the virtual machine

After the virtual machine reboots, it should now utilize the full resolution of your screen when in full screen mode.

Extras

These are solutions for some edge use-cases that you may find useful.

Disabling Mouse Integration Mode

You may need to disable "Mouse Integration Mode" in order to use your mouse in the Virtual Machine. You can do this simply with your keyboard while in the VM window: Host Key + I. The Host Key is usually the right Ctrl key.

Accessing Your Host Machine's Web Root

If you've got a local environment set up for your projects, you'll probably want to be able to access those projects locally while on your Virtual Machine. To accomplish this, you will have to modify Windows 7's hosts file; which is located at:

C:\Windows\System32\drivers\etc\hosts

You'll have to open NotePad as Administrator in order to modify the hosts file.

If you only need to be able to access localhost, just add the following line:

10.0.2.2   outer

If you want to be able to use your virtual hosts too, follow the pattern below for each of your virtual hosts:

10.0.2.2   domain-name.local

That's it!

If you're not familiar with the The Prisoner's Dilemma, it's the go-to example for describing game theory. Here's a quick explanation of how it works:

Two individuals are each presented with a choice between two options: to Defect or to Cooperate.

Defecting means an individual will betray the other in order to receive a beneficial outcome for them self and a negative outcome for the other.

Cooperating means an individual is hoping the other individual will choose to cooperate as well, in which case they would both get a slightly beneficial outcome.

Neither will know what choice the other has made until after they have both made a decision.

If you were to play this scenario out only once, the best course of action for either individual would be to defect. However, that's not all that interesting. What's more interesting is when this scenario is played out in a slightly different way dozens, or hundreds, or millions of times. When given the ability to remember at least the last few encounters you have with one individual, the rules of the game change dramatically. Knowing what prior choices your opponent has made open up a world of possibilities; many different strategies. This leads us to the Iterated Prisoner's Dilemma.

So what did I do with this information? Coded it in PHP, of course. I created over a dozen different strategies, and pitted them against one another in over two and a half million iterations of the Prisoner's Dilemma. For each iteration, the two opponents' strategies were chosen randomly. Here are the results:

An interesting next step would be to extend this further to test the effect population has on the success of each individual strategy. In other words, if Tit for Tat made up only 2% of the population, and some more malicious strategies made up a large chunk of the population, would the Tit for Tat strategy fair better or worse?

If you're interested, the code is available in a GitHub repository. If you are feeling inspired, please feel free to fork it or submit a pull request.

Creeped out by just how much companies know about you? Maybe you heard about how Target figured out a teenage girl was pregnant before her own father. Or maybe you've found the ads on the websites you visit to be a little too specific. Or maybe you've heard about the seemingly endless stream of major security breaches involving hundreds of thousands of detailed customer records:

• 1.5 million credit cards exported in hack of payments processor
• MasterCard and Visa Investigate Data Breach
• Citi Credit Card Data Breached for 200,000 Customers
• Big-Box Breach: The Inside Story of Wal-Mart's Hacker Attack

Think for a moment just how much information you give to these companies. How many companies know where you live? How many companies have your phone number? How many companies have your credit card or bank account information? Are those companies selling all or some of that data to other companies? And what are the odds any of those companies could suffer a security breach that could expose your personal information? Odds are some of your personal information has already fallen into the hands of criminals who could use it to do any number of nasty things:

• Steal your identity
• Make fraudulent purchases on your credit cards
• Take funds directly from your bank accounts
• Target you with sophisticated phishing scams

What You Can Do

All that being said, there are things you can do to protect yourself.

1. Avoid giving personal information to any website or service
2. Use an ad-blocker
3. Configure your browser to protect your privacy