Teslabs Engineering

A way to calibrate a magnetometer

Sun, 06 Dec 2015 00:00:00 +0000

My first contact with a digital magnetometer dates back to the summer of 2009. At that time, as many arduinoers, I was more concerned about coding the I2C/SPI driver other than understanding the device itself. A few months later I got an iPhone 4 and realized that the compass application, which used the internal phone magnetometer, required calibration every time it had to be used. And so, I became interested on why was this step required and how can it be done. In this post I will try to explain, together with a practical example, a way to do it. It is in essence the method described here, but with some more details and references.

A digital triple-axis magnetometer sold nowadays. Source: Sparkfun (CC-BY-NC-SA)

Preliminary concepts

Before we start it is important to have a few concepts clear. It seems logical that we need to have a basic understanding of the Earth magnetic field, as it is what we want to measure. Furthermore, basic knowledge of quadrics (e.g. spheres or ellipsoids) will turn out to be important. It may seem awkward to talk about quadrics here, but it will quickly make sense.

The Earth magnetic field

At any location on Earth, the magnetic field can be locally represented as a constant three dimensional vector ($\mathbf{h}_0$). This vector can be characterized by three properties. Firstly, the intensity or magnitude ($\mathcal{F}$), normally measured in nanoteslas (nT) with an approximate range of 25000 nT to 65000 nT. Secondly, the inclination ($\mathcal{I}$), with negative values (up to -90°) if pointing up and positive (up to 90°) if pointing down. Thirdly, the declination ($\mathcal{D}$) which measures the deviation of the field relative to the geographical north and is positive eastward.

Magnetic field frame of reference. Source: Wikimedia Commons

Using the frame of reference in the figure shown above, the Earth magnetic field vector $\mathbf{h}_0$ can be written as:

\[\mathbf{h}_0 = \mathcal{F} \begin{bmatrix} \cos{(\mathcal{I})} \cos{(\mathcal{D})} \\ \cos{(\mathcal{I})} \sin{(\mathcal{D})} \\ \sin{(\mathcal{I})} \end{bmatrix}. \label{eq_h0} \tag{1}\]

Given a geographical point the WMM model can be used to obtain the expected values for $\mathcal{F}$, $\mathcal{I}$ and $\mathcal{D}$. The magnetic declination is specially useful in order to calculate the geographical north in a compass.

WMM 2015.0 intensity, inclination and declination maps. Source: NOAA/NGDC & CIRES

Quadrics

Quadrics are all surfaces that can be expressed as a second degree polynomial in $x$, $y$ and $z$. They include popular surfaces such as spheres, ellipsoids, paraboloids, etc. The general implicit equation of a quadric surface $\mathit{S}$ is given by:

\[\begin{align} \mathit{S} :~ & ax^2 + by^2 + cz^2 + \\ & 2fyz + 2gxz + 2hxy + \\ & px + qy + rz + d = 0. \end{align} \label{eq_quad_gen} \tag{2}\]

Eq. $\eqref{eq_quad_gen}$ can be written in a semi-matricial form, which will be particularly useful for our problem:

\[\mathit{S}: \mathbf{x}^T \mathbf{M} \mathbf{x} + \mathbf{x}^T \mathbf{n} + d = 0 \label{eq_quad_semat} \tag{3}\]

where $\mathbf{x}$ is:

\[\mathbf{x} = \begin{bmatrix} x & y & z \end{bmatrix}^T\]

and $\mathbf{M}$, $\mathbf{n}$ are, respectively:

\[\mathbf{M} = \begin{bmatrix} a & h & g \\ h & b & f \\ g & f & c \end{bmatrix}, ~~ \mathbf{n} = \begin{bmatrix} p \\ q \\ r \end{bmatrix}.\]

A sphere of radius 1 centered at origin would be defined by:

\[\mathbf{M} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, ~~ \mathbf{n} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} ~ \text{and} ~ d=-1.\]

Eq. $\eqref{eq_quad_gen}$ can also be written purely in matrix form as:

\[\mathit{S}: \mathbf{x}^T \mathbf{Q} \mathbf{x} = 0\]

where $\mathbf{x}$ and $\mathbf{Q}$ are, respectively:

\[\mathbf{x} = \begin{bmatrix} x & y & z & 1 \end{bmatrix}^T, ~~ \mathbf{Q} = \left[ \begin{array}{c|c} \mathbf{M} & \mathbf{n}\\ \hline \mathbf{n}^T & d \end{array} \right].\]

The type of quadric is determined by some properties of the matrices $\mathbf{M}$ and $\mathbf{Q}$ as detailed here.

Measuring with a magnetometer

Initially, let’s assume that we are in a magnetic disturbance free environment and that we have an ideal 3-axis magnetometer. Under these conditions, a magnetometer reading $\mathbf{h}$ taken with an arbitrary orientation will be given by:

\[\mathbf{h} = \mathbf{R}_x(\phi) \mathbf{R}_y(\theta) \mathbf{R}_z(\varphi) \mathbf{h}_0, \label{eq_h} \tag{4}\]

where $\mathbf{h}_0$ is the local Earth magnetic field given in $\eqref{eq_h0}$ and $\mathbf{R}_x(\phi)$, $\mathbf{R}_y(\theta)$, $\mathbf{R}_z(\varphi)$ are rotation matrices around the frame of reference $x$, $y$, $z$ axis, respectively. Complementary sources of information would be needed to determine the orientation. It is easy to see that samples locus is a sphere of radius $\mathcal{F}$:

\[\mathbf{h}^T \mathbf{h} = \ldots = \mathcal{F}^2. \label{eq_h_const} \tag{5}\]

This way way of looking at the magnetometer samples gives us a geometric perspective of the problem. A calculated example is shown in the interactive plot found below.

Expected samples of a 3-axis magnetometer around Irvine, CA by the date of writing

Distortion sources

In real life nothing is ideal neither the environment is disturbance free. As detailed in this publication, there are two main categories of measurement distortion sources: instrumentation errors and magnetic interferences. In the next sections they will be briefly described, concluding with the measurement model.

Instrumentation errors

Instrumentation errors are unique and constant for each device. They can be modeled as a result of three components. Firstly, a scale factor, which can be modeled as a diagonal matrix: $$ \mathbf{S} = \begin{bmatrix} s_x & 0 & 0 \\ 0 & s_y & 0 \\ 0 & 0 & s_z \end{bmatrix} \label{eq_s} \tag{6} $$ Secondly, the non-orthogonality of the sensor axis, which can be modeled as: $$ \mathbf{N} = \begin{bmatrix} \mathbf{n}_x & \mathbf{n}_y & \mathbf{n}_z \end{bmatrix} \tag{7} $$ where each column represents a vector of size 3 corresponding to each sensor axis with respect to the sensor frame. Finally, a sensor offset which can be simply modeled as: $$ \mathbf{b}_{so} = \left[b_{so_x} ~ b_{so_y} ~ b_{so_z} \right]^T. \tag{8} $$

Schematic representation of each instrumentation error (-)

Magnetic interferences

The magnetic interferences are caused by ferromagnetic elements present in the surroundings of the sensor. It is composed by permanent (hard iron) and induced magnetism (soft iron). Note that we ignore any non-constant magnetic interference.

Hard iron results from permanent magnets and magnetic hysteresis (remanence of magnetized iron). It is equivalent to a bias, which can be modeled as: $$ \mathbf{b}_{hi} = \left[b_{hi_x} ~ b_{hi_y} ~ b_{hi_z} \right]^T. \tag{9} $$ Soft iron is due by the interaction of an external magnetic field with ferromagnetic materials, causing a change in the intensity and direction of the sensed field. It can be modeled as: $$ \mathbf{A}_{si} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{12} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \label{eq_si} \tag{10} $$

You may find useful to read this application note, where it is discussed which elements on a PCB can cause magnetic interference.

Schematic representation of each magnetic interference (-)

Measurement model

After detailing each distortion source, we can easily write down the measurement model. All the distortion sources described in $\eqref{eq_s}-\eqref{eq_si}$ can be combined to express the measured magnetic field $\mathbf{h}_m$ as:

\[\mathbf{h}_m = \mathbf{SN}(\mathbf{A}_{si}\mathbf{h} + \mathbf{b}_{hi}) + \mathbf{b}_{so} \label{eq_h_meas_ns} \tag{11}\]

where $\mathbf{h}$ is the true magnetic field given by $\eqref{eq_h}$. It is worth to note at this point that our measurement model does not include any stochastic noise. This is in some way unrealistic, however, it simplifies the solution approach at the cost of a slightly biased solution (assuming a small amplitude noise). You may use the method described in this paper to extend our solution by including noise.

Grouping terms $\eqref{eq_h_meas_ns}$ can be re-written as:

\[\mathbf{h}_m = \mathbf{A}\mathbf{h} + \mathbf{b} \label{eq_h_meas} \tag{12}\]

where:

\[\begin{matrix} \mathbf{A} = \mathbf{SNA}_{si} \\ \mathbf{b} = \mathbf{SNb}_{hi} + \mathbf{b}_{so}. \end{matrix}\]

$\mathbf{A}$ is a matrix combining scale factors, misalignments and soft-iron effects, while $\mathbf{b}$ is the combined bias vector. It can be proved that the linear transformation of $\mathbf{h}$ in $\eqref{eq_h_meas}$ makes the measurements $\mathbf{h}_m$ to lie on an ellipsoid. Therefore, we will see in the next section that the calibration process reduces to an ellipsoid fitting problem.

Calibration

It becomes clear from $\eqref{eq_h_meas}$ that we need to find an estimate of $\mathbf{A}$ and $\mathbf{b}$, which we will note as $\mathbf{\hat{A}}$ and $\mathbf{\hat{b}}$. Then, any new measurement $\mathbf{h}_n$ can be simply calibrated as:

\[\mathbf{\hat{h}}_n = \mathbf{\hat{A}}^{-1}(\mathbf{h}_n - \mathbf{\hat{b}}).\]

We first start expressing $\mathbf{h}$, using $\eqref{eq_h_meas}$, as:

\[\mathbf{h} = \mathbf{A}^{-1}(\mathbf{h}_m - \mathbf{b})\]

which combined with $\eqref{eq_h_const}$ leads to:

\[\begin{align} & \mathbf{h}_m^T \mathbf{A}^{-T} \mathbf{A}^{-1} \mathbf{h}_m - \\ & 2 \mathbf{h}_m^T \mathbf{A}^{-T} \mathbf{A}^{-1} \mathbf{b} + \\ & \mathbf{b}^T \mathbf{A}^{-T} \mathbf{A}^{-1} \mathbf{b} - \mathcal{F}^2 = 0. \end{align} \label{eq_cal_quad_ns} \tag{13}\]

Eq. $\eqref{eq_cal_quad_ns}$ can be re-written in the familiar quadric form:

\[\mathbf{h}_m^T \mathbf{M} \mathbf{h}_m + \mathbf{h}_m^T \mathbf{n} + d = 0 \label{eq_cal_quad} \tag{14}\]

where:

\[\mathbf{M} = \mathbf{A}^{-T} \mathbf{A}^{-1} \label{eq_cal_M} \tag{15a}\] \[\mathbf{n} = -2 \mathbf{A}^{-T} \mathbf{A}^{-1} \mathbf{b} \label{eq_cal_n} \tag{15b}\] \[d = \mathbf{b}^T \mathbf{A}^{-T} \mathbf{A}^{-1} \mathbf{b} - \mathcal{F}^2. \label{eq_cal_d} \tag{15c}\]

As we noted in the previous section our quadric surface will be an ellipsoid. Thus, we need to use an ellipsoid fitting algorithm which provides us with estimates of the quadric surface parameters, $\mathbf{\hat{M}}$, $\mathbf{\hat{n}}$ and $\hat{d}$. Multiple algorithms have been developed to fit a set of points to an ellipsoid surface. We will use the LS fitting method described in this paper by Q. Li et al. This method is easy to implement and there is also a MATLAB version readily available. Even though presented as an iterative method, a single step fit is enough if the points describe well the ellipsoid (which will be our case). Furthermore, it is shown that it performs reasonably well for low noise levels.

Using $\eqref{eq_cal_quad}$ and $\eqref{eq_cal_M}$-$\eqref{eq_cal_d}$ with the estimated values $\mathbf{\hat{M}}$, $\mathbf{\hat{n}}$ and $\hat{d}$ we can find $\mathbf{\hat{A}}$ (or $\mathbf{\hat{A}}^{-1}$) and $\mathbf{\hat{b}}$. As detailed in this paper, you will need to find the proper scale of the solution, which is found to be:

\[\mathbf{\hat{b}} = - \mathbf{\hat{M}}^{-1} \mathbf{\hat{n}} \label{eq_cal_b} \tag{16}\] \[\mathbf{\hat{A}}^{-1} = \frac{\mathcal{F}} {\sqrt{\mathbf{\hat{n}}^T\mathbf{\hat{M}}^{-1}\mathbf{\hat{n}} - \hat{d}}} \mathbf{\hat{M}}^{\frac{1}{2}}. \label{eq_cal_A_1} \tag{17}\]

As you can observe from $\eqref{eq_cal_A_1}$, knowledge of $\mathcal{F}$ is required in order to calculate $\mathbf{\hat{A}}^{-1}$. This will make the calibrated sphere to have radius $\mathcal{F}$. However, in many applications such as in attitude estimation, the magnitude is irrelevant. If that is your case, you can make it one or any other convenient value.

In order to conclude this section, the proposed code to perform the calibration is shown below. It uses Numpy and SciPy for numeric calculations as well as the sensor class detailed in the next section.

import sys
import numpy as np
from scipy import linalg

class Magnetometer(object):
    ''' Magnetometer class with calibration capabilities.

        Parameters
        ----------
        sensor : str
            Sensor to use.
        bus : int
            Bus where the sensor is attached.
        F : float (optional)
            Expected earth magnetic field intensity, default=1.
    '''

    # available sensors
    _sensors = {'ak8975c': SensorAK8975C}

    def __init__(self, sensor, bus, F=1.):

        # sensor
        if sensor not in self._sensors:
            raise NotImplementedError('Sensor %s not available' % sensor)
        self.sensor = self._sensors[sensor](bus)

        # initialize values
        self.F   = F
        self.b   = np.zeros([3, 1])
        self.A_1 = np.eye(3)

    def read(self):
        ''' Get a sample.

            Returns
            -------
            s : list
                The sample in uT, [x,y,z] (corrected if performed calibration).
        '''
        s = np.array(self.sensor.read()).reshape(3, 1)
        s = np.dot(self.A_1, s - self.b)
        return [s[0,0], s[1,0], s[2,0]]

    def calibrate(self):
        ''' Performs calibration. '''

        print('Collecting samples (Ctrl-C to stop and perform calibration)')

        try:
            s = []
            n = 0
            while True:
                s.append(self.sensor.read())
                n += 1
                sys.stdout.write('\rTotal: %d' % n)
                sys.stdout.flush()
        except KeyboardInterrupt:
            pass

        # ellipsoid fit
        s = np.array(s).T
        M, n, d = self.__ellipsoid_fit(s)

        # calibration parameters
        # note: some implementations of sqrtm return complex type, taking real
        M_1 = linalg.inv(M)
        self.b = -np.dot(M_1, n)
        self.A_1 = np.real(self.F / np.sqrt(np.dot(n.T, np.dot(M_1, n)) - d) *
                           linalg.sqrtm(M))

    def __ellipsoid_fit(self, s):
        ''' Estimate ellipsoid parameters from a set of points.

            Parameters
            ----------
            s : array_like
              The samples (M,N) where M=3 (x,y,z) and N=number of samples.

            Returns
            -------
            M, n, d : array_like, array_like, float
              The ellipsoid parameters M, n, d.

            References
            ----------
            .. [1] Qingde Li; Griffiths, J.G., "Least squares ellipsoid specific
               fitting," in Geometric Modeling and Processing, 2004.
               Proceedings, vol., no., pp.335-340, 2004
        '''

        # D (samples)
        D = np.array([s[0]**2., s[1]**2., s[2]**2.,
                      2.*s[1]*s[2], 2.*s[0]*s[2], 2.*s[0]*s[1],
                      2.*s[0], 2.*s[1], 2.*s[2], np.ones_like(s[0])])

        # S, S_11, S_12, S_21, S_22 (eq. 11)
        S = np.dot(D, D.T)
        S_11 = S[:6,:6]
        S_12 = S[:6,6:]
        S_21 = S[6:,:6]
        S_22 = S[6:,6:]

        # C (Eq. 8, k=4)
        C = np.array([[-1,  1,  1,  0,  0,  0],
                      [ 1, -1,  1,  0,  0,  0],
                      [ 1,  1, -1,  0,  0,  0],
                      [ 0,  0,  0, -4,  0,  0],
                      [ 0,  0,  0,  0, -4,  0],
                      [ 0,  0,  0,  0,  0, -4]])

        # v_1 (eq. 15, solution)
        E = np.dot(linalg.inv(C),
                   S_11 - np.dot(S_12, np.dot(linalg.inv(S_22), S_21)))

        E_w, E_v = np.linalg.eig(E)

        v_1 = E_v[:, np.argmax(E_w)]
        if v_1[0] < 0: v_1 = -v_1

        # v_2 (eq. 13, solution)
        v_2 = np.dot(np.dot(-np.linalg.inv(S_22), S_21), v_1)

        # quadric-form parameters
        M = np.array([[v_1[0], v_1[3], v_1[4]],
                      [v_1[3], v_1[1], v_1[5]],
                      [v_1[4], v_1[5], v_1[2]]])
        n = np.array([[v_2[0]],
                      [v_2[1]],
                      [v_2[2]]])
        d = v_2[3]

        return M, n, d

Example

To conclude this post we will use samples from a real magnetometer, which will allow us to see if the proposed solution behaves as expected. It is important to note that we will not evaluate the accuracy of our solution, neither other things such as how the number of samples or their spatial distribution affects the calibration. These are topics that would require further study, out of the scope of this post.

The chosen magnetometer is the one found inside the Invensense MPU-9150: AK8975C. It is a module that also contains an accelerometer and a gyroscope, not required for our purposes. A Raspberry Pi 2 will be used as a host platform, as shown below. This will allow us to quickly code a proof of concept by just using a few lines of Python. Note that you will need to activate the I2C driver and install python-smbus on the Raspberry Pi (details here).

The setup used for this example

The proposed code for the AK8975C drive is:

import smbus

# MPU9150 used registers
MPU9150_ADDR                       = 0x68
MPU9150_PWR_MGMT_1                 = 0x6B
MPU9150_INT_PIN_CFG                = 0x37
MPU9150_INT_PIN_CFG_I2C_BYPASS_EN  = 0x02

# AK8975C used registers
AK8975C_ADDR                       = 0x0C
AK8975C_ST1                        = 0x02
AK8975C_ST1_DRDY                   = 0x01
AK8975C_HXL                        = 0x03
AK8975C_CNTL                       = 0x0A
AK8975C_CNTL_SINGLE                = 0x01

# convert to s16 from two's complement
to_s16 = lambda n: n - 0x10000 if n > 0x7FFF else n


class SensorAK8975C(object):
    ''' AK8975C Simple Driver.

        Parameters
        ----------
        bus : int
            I2C bus number (e.g. 1).
    '''

    # sensor sensitivity (uT/LSB)
    _sensitivity = 0.3

    def __init__(self, bus):
        self.bus = smbus.SMBus(bus)

        # wake up and set bypass for AK8975C
        self.bus.write_byte_data(MPU9150_ADDR, MPU9150_PWR_MGMT_1, 0)
        self.bus.write_byte_data(MPU9150_ADDR, MPU9150_INT_PIN_CFG,
                                 MPU9150_INT_PIN_CFG_I2C_BYPASS_EN)

    def __del__(self):
        self.bus.close()

    def read(self):
        ''' Get a sample.

            Returns
            -------
            s : list
                The sample in uT, [x, y, z].
        '''

        # request single shot
        self.bus.write_byte_data(AK8975C_ADDR, AK8975C_CNTL,
                                               AK8975C_CNTL_SINGLE)

        # wait for data ready
        r = self.bus.read_byte_data(AK8975C_ADDR, AK8975C_ST1)
        while not (r & AK8975C_ST1_DRDY):
            r = self.bus.read_byte_data(AK8975C_ADDR, AK8975C_ST1)

        # read x, y, z
        data = self.bus.read_i2c_block_data(AK8975C_ADDR, AK8975C_HXL, 6)

        return [self._sensitivity * to_s16(data[0] | (data[1] << 8)),
                self._sensitivity * to_s16(data[2] | (data[3] << 8)),
                self._sensitivity * to_s16(data[4] | (data[5] << 8))]

Merging the sensor driver shown above with the code in the previous section will allow us to both sample and calibrate. We can also append the code shown below, which will allow us to store non-calibrated and calibrated samples to then visualize the results with any plotting library:

from time import sleep

def collect(fn, fs=10):
    ''' Collect magnetometer samples

        Parameters
        ----------
        fn : str
            Output file.
        fs : int (optional)
            Approximate sampling frequency (Hz), default 10 Hz.
    '''

    print('Collecting [%s]. Ctrl-C to finish' % fn)
    with open(fn, 'w') as f:
        f.write('x,y,z\n')
        try:
            while True:
                s = m.read()
                f.write('%.1f,%.1f,%.1f\n' % (s[0], s[1], s[2]))
                sleep(1./fs)
        except KeyboardInterrupt: pass

m = Magnetometer('ak8975c', 1, 46.85)
collect('ncal.csv')
m.calibrate()
collect('cal.csv')

Real results are displayed in the following interactive plot:

Samples taken before and after calibration shown together with the reference sphere.

TopoJSON maps for Catalonia

Fri, 20 Nov 2015 00:00:00 +0000

After trying some of the TopoJSON examples I could not resist the temptation of learning more about it. As a Catalan citizen, I decided to focus on Catalonia (or Catalunya in Catalan). Before starting I did a quick search to see if somebody else had already used TopoJSON in my region. I found some nice examples from a couple of authors: martgnz and rveciana. However in these examples little information is given on how maps are generated. Particularly topics like map simplification or projection are not discussed. So I decided to write this post where I explain in some detail the process of generating the TopoJSON maps for Catalonia.

Together with this post I have created the cat-topojson utility which will allow you to easily generate the maps.

Example: hover the mouse on any county to know its area!

Required tools

The first requirement is the TopoJSON host tool, topojson, which in turn requires Node.js. If you are on OS X, and assuming you use brew, you can easily get both by typing:

$ brew install node
$ npm install -g topojson

In this post we will also use some tools part of GDAL, useful to convert from one coordinate system to another or simply to retrieve information from map files. It can also be installed using brew:

$ brew install gdal

Source maps

Before we proceed further, we need to obtain the source maps. They are freely available from ICGC (registration is required). The maps of interest are called Base Municipal and are offered in multiple scales (1:50, 1:250 and 1:1000) and formats (DXF, DGN and SHP). In the examples found in this post we will use the scale 1:50 and the format SHP (shapefile), which is the only one of the list supported by topojson. The specification for these maps can be found here. A summary of the relevant information is given below.

The shapefiles of interest contained in the downloaded file are the following:

File name	Description
`bm[eeee]mv33sh1fpm[m]_[aaaammdd]_[c].shp`	Municipality Polygons
`bm[eeee]mv33sh1fpc[m]_[aaaammdd]_[c].shp`	County Polygons
`bm[eeee]mv33sh1fpp[m]_[aaaammdd]_[c].shp`	Province Polygons

where eeee is the scale (i.e., 50, 250, 1000), m is the frame coordinate reference (only 1 is available, meaning EPSG:25831 - ETRS89 / UTM zone 31N), aaaammdd is the maps reference date (e.g. 20150501) and c is the distribution revision (e.g. 0). Other files are provided, e.g. the location of municipality capital cities, but they will not be analyzed in this post.

We can easily get detailed information of each file using the ogrinfo utility provided by GDAL, e.g.

$ ogrinfo -al bm50mv33sh1fpm1_20150501_0.shp | less
Geometry: Polygon
Feature Count: 947
Extent: (260188.973378, 4488778.587563) - (527401.970714, 4747980.912187)
Layer SRS WKT:
PROJCS["ETRS_1989_UTM_Zone_31N",
    GEOGCS["GCS_ETRS_1989",
        DATUM["European_Terrestrial_Reference_System_1989",
            SPHEROID["GRS_1980",6378137.0,298.257222101]],
        PRIMEM["Greenwich",0.0],
        UNIT["Degree",0.0174532925199433]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["False_Easting",500000.0],
    PARAMETER["False_Northing",0.0],
    PARAMETER["Central_Meridian",3.0],
    PARAMETER["Scale_Factor",0.9996],
    PARAMETER["Latitude_Of_Origin",0.0],
    UNIT["Meter",1.0]]
MUNICIPI: String (6.0)
COMARCA: String (2.0)
PROVINCIA: String (2.0)
NOM_MUNI: String (45.0)
NOMN_MUNI: String (45.0)
NOMG_MUNI: String (45.0)
CAP_MUNI: String (30.0)
CAPN_MUNI: String (30.0)
CAPG_MUNI: String (30.0)
SUP_MUNI: Real (6.2)
ORSUP_MUNI: String (1.0)
...

Generation of the TopoJSON maps

In the following sections the process of generating the TopoJSON maps from the ICGC shapefiles is discussed.

Geometry

There are two types of coordinate systems used in maps: projected (X, Y) or geographic ($\lambda$, $\varphi$). As we have just seen in the previous section, the maps provided by ICGC are projected in UTM. Advantages of using projected coordinates include better simplification results as well as reduced overhead on the rendering process as stated in this example. You can use either one or the other with D3.js, so choose what best fits your application. In this post examples for both cases are provided.

In order to change from one coordinate system to another, you can use the ogr2ogr utility provided by GDAL. Actually, ogr2ogr offers many more possibilities (e.g. filtering), but it is left to the reader to explore them. As a quick example, if you want to convert the counties shapefile to the World Geodetic System, EPSG:4326:

$ ogr2ogr \
    -f 'ESRI Shapefile' \
    -t_srs EPSG:4326 \
    counties-wgs.shp \
    bm50mv33sh1fpc1_20150501_0.shp

Provided that your input maps are in geographical coordinates, you could also use the topojson --projection option to apply one of the D3.js projections.

Finally, when using projected coordinates, we can specify --width, --height and --margin to fit a viewport of the specified size. This will simplify the map usage if we know in advance where we are going to display the map, as we will not need to translate or scale it (see the examples).

Simplification and quantization

An important thing to care about when distributing maps is its complexity and size, both intrinsically related. Two techniques can be used to reduce both: geometry simplification and quantization.

As illustrated, in this article geometry simplification reduces the number of points given an area threshold. This area threshold is specified through the -s argument. It is given in steradians in geographical coordinates. In case of projected coordinates it is given as squared base units, e.g. if we use --width and/or --height it will be in px².

Quantization is about numeric precision. TopoJSON uses fixed point coordinates, and allows you to specify the number of differentiable values. It is controlled by the -q parameter, or --pre-quantization and --post-quantization to specify input and output quantizations, respectively. A value of 10.000 differentiable values is used by default, which in most cases will be fine. Roughly speaking, the more values, the larger the size of the map.

In any case, the simplest way to choose the right simplification and quantization parameters is to try and see how map looks. Three illustrative examples are shown below, exaggerating the effects of simplification and quantization in the second and third maps, respectively.

Properties

The ICGC shapefiles contain some properties which are useful to retain (e.g. province names, area, etc.). They are detailed in the specification file.

By default, topojson does not copy the properties contained in the input files. The -p argument can be used to copy all properties, or -p target=source to only keep the source property while renaming it to target. More than one property can be copied by passing multiple -p arguments or by appending multiple properties, e.g. -p name=NAME,surname=SURN. Furthermore, prepending a + sign to a property name will force the property to be a number, e.g. -p area=+AREA.

Another important argument is --id-property. It is used to assign any of the properties to the ID of the geometry.

Examples

Finally to conclude this post a couple of minimal examples are provided. Note that not many details are given on how to use D3.js/TopoJSON, as it is not the aim of this post. If you want to learn more about the topic, you can find great tutorials online like this.

In order to run the examples you will need an HTTP server, e.g.

$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Projected map

In this example we will use the counties (comarques) map and fit it to a viewport of 500x500 px. As the source map is already projected, we can just feed it into topojson:

$ topojson \
    -o cat-comarques.json \
    --width=500 --height=500 \
    --simplify=2 \
    --id-property=+COMARCA \
    -p nom=NOM_COMAR \
    -p cap=CAP_COMAR \
    -p sup=SUP_COMAR \
    -- comarques=bm50mv33sh1fpc1_20150501_0.shp

Then, we can quickly visualize the results creating an html file with the following code:

<!DOCTYPE html>
<meta charset="utf-8">
<style>

.land {
  fill: #ddc;
  stroke: white;
}

</style>

<body>
<script src="//d3js.org/d3.v3.min.js" charset="utf-8"></script>
<script src="//d3js.org/topojson.v1.min.js"></script>

<script>

var width  = 500,
    height = 500;

// no need to project!
var path = d3.geo.path()
    .projection(null);

var svg = d3.select("body").append("svg")
    .attr("width", width)
    .attr("height", height);

d3.json("cat-comarques.json", function(error, cat) {
  if (error) throw error;

  svg.append("path")
      .datum(topojson.feature(cat, cat.objects.comarques))
      .attr("d", path)
      .attr("class", "land")
});

</script>

…and voilà:

Open in a new window

Non-projected map

In this second example we will use the municipalities (municipis) map but the projection will be done on the browser. We first use ogr2ogr to convert our map to the WGS-84 geographical coordinate system:

$ ogr2ogr \
    -f 'ESRI Shapefile' \
    -t_srs EPSG:4326 \
    municipis-wgs.shp \
    bm50mv33sh1fpm1_20150501_0.shp

Then, we use topojson to generate the map:

$ topojson \
    -o cat-municipis.json \
    --simplify=1e-8 \
    --id-property=+MUNICIPI \
    -p nom=NOM_MUNI \
    -p comarca=+COMARCA \
    -p provincia=+PROVINCIA \
    -p sup=SUP_MUNI \
    -- municipis=municipis-wgs.shp

Note that an area threshold of 10^-8 sr has proved to deliver a good compromise after a few tries.

When using a projected map already adjusted to a viewport we do not need to care about centering or scaling it, but now we do. Finding the right values is a tedious task, as you need to go through a trial and error process until you find an acceptable result. However, there is an easy way to automatically center and scale a map described in this great example, which could even be used to calculate the right static values. Summarizing, the example code looks like this:

<!DOCTYPE html>
<meta charset="utf-8">
<style>

.land {
  fill: #ddc;
  stroke: white;
}

</style>

<body>
<script src="//d3js.org/d3.v3.min.js" charset="utf-8"></script>
<script src="//d3js.org/topojson.v1.min.js"></script>

<script>

var width  = 500,
    height = 500;

var projection = d3.geo.mercator();

var path = d3.geo.path()
    .projection(projection);

var svg = d3.select("body").append("svg")
    .attr("width", width)
    .attr("height", height);

d3.json("cat-municipis.json", function(error, cat) {
  if (error) throw error;

  var municipis = topojson.feature(cat, cat.objects.municipis);

  // automatic center and scale (see http://bl.ocks.org/mbostock/4707858)
  projection
      .scale(1)
      .translate([0, 0]);

  var b = path.bounds(municipis),
      s = .95 / Math.max((b[1][0] - b[0][0]) / width, (b[1][1] - b[0][1]) / height),
      t = [(width - s * (b[1][0] + b[0][0])) / 2, (height - s * (b[1][1] + b[0][1])) / 2];

  projection
      .scale(s)
      .translate(t);

  svg.append("path")
      .datum(municipis)
      .attr("class", "land")
      .attr("d", path);

});

</script>

…and produces:

Open in a new window

NOTE: Source maps are produced by ICGC and are subject to terms of use.