staticnotes.org

UK house price changes in real terms

Sun, 11 Jan 2026 00:00:00 +0000

This post is a response to a newspaper article I read this morning. Every month the FT is reporting on how the housing market in the UK is performing as people love to talk about it:

UK house prices rise less than expected in 2025 as growth slows

UK house prices rose by 0.6 per cent in 2025 after a slowdown at the end of the year, according to lender Nationwide. […] Prices fell 0.4 per cent between November and December to an average of £271,068. Both figures were below analysts’ expectations of a 1.2 per cent annual rise and a 0.1 per cent month-on-month expansion. FT Weekend (02.01.2026)

I am asking myself how useful this statistic is for the average reader, especially for a reader who wants to buy or sell a property.

Two issues are immediately obvious:

The average yearly house price change across all of the UK might not agree with the house price performance in the areas I live in or I want to move to. There are also difference by house type, e.g. flat vs. detached house vs. terraced house. Different types of property might even perform in opposite directions.
British people often view property, at least partially, as an investment opportunity. However, in those cases it would help to at least report the real (inflation-adjusted) house price change, not just the nominal change. An increase of 0.6% in nominal house prices sounds different when one doesn’t mention that inflation in 2025 was around 3.6% (UK CPI). Meanwhile other forms of investments did quite well. The FTSE 100 nominally returned 21.4% (which to be fair was a positive outlier this year compared to an average of 6% over the last 10 years).

Fortunately, it is not that difficult to compute real house prices more closely to my situation. The raw data is openly available via API:

The HM Land Registry registers the ownership of property in the UK and provides aggregate monthly data of the average transactions by region and property type.
The Office for National Statistics (ONS) provides monthly inflation data (CPI) for the UK.

So, in this post I want to compute and visualize how real flat and detached house prices in specific areas performed both nominally and in real terms.

Querying the housing price data

I am using the UK House Price Index of the Land Registry API:

http://landregistry.data.gov.uk/data/ukhpi/region

This allows me to query the monthly average house price value for different regions and property types. Important to note here is that this data is based on the succesful property transactions in a region which we use here as an approximation of the average value of similar properties in the region (that were or weren’t sold).

I am based in London and interested how the housing market performs close to me or areas that I am interested living in. So I will query the data for all London boroughs as well as some South England regions around London: Kent, Surrey, Buckinghamshire, and Oxfordshire.

I then resample the monthly data to quarterly data by taking the mean across the 3 months of a quarter.

I then store all the data in parquet files. You can find them here .

Inspecting the data

Let’s have a look at how the data is stored in the file:

import pandas as pd
import warnings
import matplotlib.pyplot as plt
pd.set_option('display.float_format', lambda x: '%.2f' % x)
warnings.simplefilter('always', category=UserWarning)

Here I am loading the file for the different counties that I am interested in:

df_regions = pd.read_parquet('./data/uk_house_prices.parquet')
df_regions['is_london_borough'] = False
df_regions.sample(5,ignore_index=True)

	Quarter	Region	Price	PropertyType	is_london_borough
0	2023Q4	Surrey	517233.67	All	False
1	2011Q3	Kent	192827.67	All	False
2	2014Q2	Oxfordshire	277195.00	Semi_detached	False
3	2013Q4	London	415838.00	Semi_detached	False
4	2021Q3	Buckinghamshire	431147.67	All	False

and here I am loading the file for the London boroughs:

df_boroughs = pd.read_parquet('./data/london_borough_prices.parquet')
df_boroughs['is_london_borough'] = True
df_boroughs.sample(5,ignore_index=True)

	Quarter	Region	Price	PropertyType	is_london_borough
0	2020Q4	Hillingdon	406610.67	Terraced	True
1	2024Q4	Redbridge	490087.67	All	True
2	2017Q2	Lewisham	910228.33	Detached	True
3	2011Q3	Islington	369223.67	Flat	True
4	2010Q3	Enfield	661067.33	Detached	True

You can see that the dataset shows the average quarterly housing transaction values by region. The column PropertyType shows what type of property was sold (Flat, Terraced, Detached, Semi_detached).

Computing real house prices

Next we adjust the nominal house prices from the Land Registry for inflation and compute the real house prices in terms of “2025 pounds”. I have queried the Office for National statistics API to get the UK’s quarterly CPI data and stored it in a parquet file. I am loading this data here:

cpi_df = pd.read_parquet('./data/cpi_data.parquet')
cpi_df.tail(5)

	Quarter	CPI
146	2024Q3	134.10
147	2024Q4	135.20
148	2025Q1	136.00
149	2025Q2	138.50
150	2025Q3	139.20

With the consumer price index, we can compute the real price of housing. This real price answers the question: “How much was the property worth in last quarter’s pounds?” If you experienced inflation, your money is worth less, and you will need to pay more for the same house in today’s money terms. If you experienced deflation, your money is worth more, and you will need to pay less for a house in today’s terms (all else equal).

The formula to compute the real house price in terms of the current CPI:

real_price_current = nominal_price_t * (CPI_current / CPI_t)

# Merging the dataframes here to do real price calculation in one step
df = pd.concat([df_regions, df_boroughs])

# Get the most recent CPI value
current_cpi = cpi_df['CPI'].iloc[-1]
latest_quarter = cpi_df['Quarter'].iloc[-1]

# Merge CPI df with house prices df
df = df.merge(cpi_df, on='Quarter', how='left')

# For quarters without CPI data yet, use the most recent CPI
df['CPI'] = df['CPI'].fillna(current_cpi)

# Calculate real prices using above formula 
df['RealPrice'] = df['Price'] * (current_cpi / df['CPI'])

df.sample(5,ignore_index=True)

	Quarter	Region	Price	PropertyType	is_london_borough	CPI	RealPrice
0	2012Q3	City Of Westminster	2138137.67	Semi_detached	True	96.10	3097073.50
1	2011Q3	Croydon	280930.00	Semi_detached	True	93.80	416902.52
2	2016Q3	Bromley	898828.67	Detached	True	100.90	1240009.42
3	2017Q4	Ealing	1160573.67	Detached	True	104.60	1544472.80
4	2025Q2	City Of London	792392.33	Flat	True	138.50	796397.20

As you can see we now have the column RealPrice which gives us the mean house price transation per quarter and region in real terms. Now we can visualize the data to finally see what happened.

Visualizing nominal and real house prices over time

My goal is to plot the nominal and real house prices over time. To keep things visually simpler on the charts I am only selecting a few London boroughs that I am interested in and only look at data since 2015. For London boroughs I am looking at Flats and Terraced houses and for the counties I am only looking at detached houses.

df = df[df['Quarter'] >= '2015Q1'].copy()

SELECTED_BOROUGHS = [
    'Ealing',
    'Hackney',
    'Hammersmith And Fulham',
    'Hounslow',
    'Islington',
    'Richmond Upon Thames',
    'Wandsworth',
]
df_boroughs = df[df['Region'].isin(SELECTED_BOROUGHS) & (df['PropertyType'].isin(['Flat', 'Terraced']))].copy()
df_regions = df[(df['is_london_borough'] == False) & (df['PropertyType'] == 'Detached')].copy()

In the file utils.py I added some code to create the plots and tables. You can safely skip this if not interested. Importing them here in the notebook:

from utils import create_table, plot_prices_by_type

Detached Houses in South England Counties

I am first looking at the house price performance of detached houses in five different English counties. I am plotting both the nominal prices (dashed line) and the real prices (solid line).

I am also creating a table with the real price changes vs. 1 year ago, 2 years ago, 5 years ago, 10 years ago.

fig, axes = plt.subplots(1, 1, figsize=(14, 8))
plot_prices_by_type(df_regions, 'Detached', axes, 'Price', 'Detached Houses in South England counties')

You can see that within the last 10 years most counties saw a decent increase in nominal terms and a decline in real house prices for detached properties. The only exception is Kent with a small real increase of 0.9%. London and Surrey both lost around 10% in real terms.

regions = sorted(df['Region'].unique())
gt_regions_flat = create_table(
        df_regions, 'Detached',
        'South England Counties',
        regions
    )
gt_regions_flat

	2025 (Current)	2024 (1y ago)	2023 (2y ago)	2020 (5y ago)	2015 (10y ago)
South England Counties
Detached properties - Real prices (CPI-adjusted) - Q4 2025 vs 1, 2, 5, 10 years ago
Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%
Buckinghamshire	£853,221	0.0	£883,178	−3.4	£894,115	−4.6	£974,110	−12.4	£924,706	−7.7
Kent	£605,363	0.0	£620,638	−2.5	£633,552	−4.4	£692,871	−12.6	£599,898	0.9
London	£1,132,911	0.0	£1,178,785	−3.9	£1,179,862	−4.0	£1,331,199	−14.9	£1,265,328	−10.5
Oxfordshire	£680,124	0.0	£675,726	0.7	£676,613	0.5	£757,910	−10.3	£723,218	−6.0
Surrey	£975,634	0.0	£1,007,309	−3.1	£1,023,827	−4.7	£1,118,116	−12.7	£1,083,197	−9.9

Flats in London Boroughs

Next, I am looking at flats in London. Again I am plotting nominal and real quarterly prices for selected London boroughs and create a table with the corresponding numbers.

fig, axes = plt.subplots(1, 1, figsize=(14, 8))
plot_prices_by_type(df_boroughs, 'Flat', axes, 'Price', 'Flats in selected London boroughs')

Both the chart and the table show somewhat stable or slightly increasing nominal prices, but drastic reductions in real prices. Both Hammersmith/Fulham and Wandsworth are down about 30% in real terms since 2015. Moreover compared to last year and two years ago we still see price declines in real terms.

regions = sorted(df['Region'].unique())
gt_regions_flat = create_table(
        df_boroughs, 'Flat',
        'Selected London Boroughs',
        regions
    )
gt_regions_flat

	2025 (Current)	2024 (1y ago)	2023 (2y ago)	2020 (5y ago)	2015 (10y ago)
Selected London Boroughs
Flat properties - Real prices (CPI-adjusted) - Q4 2025 vs 1, 2, 5, 10 years ago
Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%
Ealing	£404,408	0.0	£419,269	−3.5	£419,733	−3.7	£489,258	−17.3	£521,670	−22.5
Hackney	£546,932	0.0	£554,931	−1.4	£583,301	−6.2	£685,095	−20.2	£667,622	−18.1
Hammersmith And Fulham	£577,801	0.0	£639,856	−9.7	£707,815	−18.4	£760,674	−24.0	£883,098	−34.6
Hounslow	£367,011	0.0	£375,391	−2.2	£373,890	−1.8	£441,401	−16.9	£419,450	−12.5
Islington	£566,442	0.0	£599,386	−5.5	£623,593	−9.2	£725,791	−22.0	£771,081	−26.5
Richmond Upon Thames	£483,018	0.0	£524,730	−7.9	£520,433	−7.2	£601,897	−19.8	£633,108	−23.7
Wandsworth	£540,034	0.0	£568,240	−5.0	£603,492	−10.5	£715,552	−24.5	£761,426	−29.1

Terraced houses in London Boroughs

Next, I am looking at terraced houses in the same London boroughs. My expectation is that they did better than flats, especially since they were more popular during the COVID pandemic, where people wanted a garden.

fig, axes = plt.subplots(1, 1, figsize=(14, 8))
plot_prices_by_type(df_boroughs, 'Terraced', axes, 'Price', 'Terraced houses in selected London boroughs')

Indeed, terraced houses did better in nominal terms with all of them up. Again in real terms we see price decreases over all considered comparison years (1,2,5,10). However, it is less drastic than for flats in the same locations.

regions = sorted(df['Region'].unique())
gt_regions_flat = create_table(
        df_boroughs, 'Terraced',
        'Selected London Boroughs',
        regions
    )
gt_regions_flat

	2025 (Current)	2024 (1y ago)	2023 (2y ago)	2020 (5y ago)	2015 (10y ago)
Selected London Boroughs
Terraced properties - Real prices (CPI-adjusted) - Q4 2025 vs 1, 2, 5, 10 years ago
Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%	Price (£)	%
Ealing	£690,513	0.0	£697,849	−1.1	£690,002	0.1	£788,622	−12.4	£800,316	−13.7
Hackney	£972,274	0.0	£966,520	0.6	£1,006,975	−3.4	£1,158,935	−16.1	£1,073,030	−9.4
Hammersmith And Fulham	£1,111,278	0.0	£1,221,472	−9.0	£1,334,009	−16.7	£1,392,109	−20.2	£1,547,731	−28.2
Hounslow	£625,089	0.0	£622,316	0.4	£612,967	2.0	£704,956	−11.3	£637,839	−2.0
Islington	£1,124,856	0.0	£1,175,162	−4.3	£1,214,208	−7.4	£1,380,766	−18.5	£1,380,624	−18.5
Richmond Upon Thames	£890,118	0.0	£935,936	−4.9	£918,218	−3.1	£1,036,584	−14.1	£1,033,227	−13.9
Wandsworth	£965,788	0.0	£1,000,163	−3.4	£1,050,418	−8.1	£1,214,261	−20.5	£1,219,378	−20.8

Winners and Losers in real terms (last 3 years)

Lastly I want to see how the housing market in the different London boroughs compare to each other. I am looking at the prices 3 years ago across all property types. (At first I wanted to use 5 years, but I wanted to stay clear of the COVID years which might introduce short-term anomalies into the result). I am comparing real prices even though for a relative comparison between boroughs nominal prices would be fine as well (they all experienced the same inflation).

selected = df[(df['is_london_borough'] == True) & (df['Quarter'].isin(['2025Q4','2022Q4'])) & (df['PropertyType'] == 'All')]

df_wide = selected.pivot(index=['Region', 'PropertyType'], columns='Quarter', values='RealPrice').reset_index()
df_wide['%_real_price_change'] = (df_wide['2025Q4'] - df_wide['2022Q4']) / df_wide['2022Q4']
df_wide.sort_values('%_real_price_change', ascending=False,ignore_index=True).head(10)

Quarter	Region	PropertyType	2022Q4	2025Q4	%_real_price_change
0	Lewisham	All	529841.12	503542.00	-0.05
1	Havering	All	479610.47	450760.00	-0.06
2	Southwark	All	647219.21	607299.00	-0.06
3	Waltham Forest	All	569427.61	533922.00	-0.06
4	Haringey	All	697080.71	649650.00	-0.07
5	Hackney	All	681376.13	627538.00	-0.08
6	Hounslow	All	588525.22	540799.00	-0.08
7	Sutton	All	493375.92	452726.00	-0.08
8	Hillingdon	All	526316.99	482852.00	-0.08
9	Barking And Dagenham	All	387895.94	354709.00	-0.09

You can see that even the best performing borough Lewisham did saw decreasing real house prices of 5% in the last 3 years. Overall a lot of these top 10 boroughs are further outside of London and generally more affordable areas. Both Hackney and Southwark are more central, but have recently undergone gentrification which might have counteracted the price pressure in other central boroughs.

df_wide.sort_values('%_real_price_change', ascending=False,ignore_index=True).tail(10)

Quarter	Region	PropertyType	2022Q4	2025Q4	%_real_price_change
23	Barnet	All	706765.75	606006.00	-0.14
24	Camden	All	928639.42	792454.00	-0.15
25	Croydon	All	464581.92	393479.00	-0.15
26	Lambeth	All	654875.76	550720.00	-0.16
27	Hammersmith And Fulham	All	883837.60	741308.00	-0.16
28	Wandsworth	All	834257.35	695867.00	-0.17
29	Tower Hamlets	All	595217.88	470209.00	-0.21
30	Kensington And Chelsea	All	1637183.90	1194726.00	-0.27
31	City Of Westminster	All	1273329.16	889935.00	-0.30
32	City Of London	All	993657.65	607399.00	-0.39

Some of the worst performing boroughs over the last 3 years are the City of London, Westminster, Kensington & Chelsea, and Hammersmith and Fulham, which are all very expensive ‘prestige’ areas.

Conclusion

This analysis shows that a prospective home buyer or seller only gains limited information from a national-level house price statistic. Instead, I showed that there are, unsurprisingly, vast differences in price developments between property types and locations.

Moreover, in real terms almost none of the considered regions saw increasing house prices over the last 10 years (notable exception: Kent).

Appendix

Since someone asked, here are the real price changes compare to 3 years ago for all London boroughs:

df_wide.sort_values('%_real_price_change', ascending=False,ignore_index=True)

Quarter	Region	PropertyType	2022Q4	2025Q4	%_real_price_change
0	Lewisham	All	529841.12	503542.00	-0.05
1	Havering	All	479610.47	450760.00	-0.06
2	Southwark	All	647219.21	607299.00	-0.06
3	Waltham Forest	All	569427.61	533922.00	-0.06
4	Haringey	All	697080.71	649650.00	-0.07
5	Hackney	All	681376.13	627538.00	-0.08
6	Hounslow	All	588525.22	540799.00	-0.08
7	Sutton	All	493375.92	452726.00	-0.08
8	Hillingdon	All	526316.99	482852.00	-0.08
9	Barking And Dagenham	All	387895.94	354709.00	-0.09
10	Merton	All	677772.16	618798.00	-0.09
11	Harrow	All	590633.55	538433.00	-0.09
12	Bromley	All	581127.96	529749.00	-0.09
13	Redbridge	All	524996.77	476764.00	-0.09
14	Richmond Upon Thames	All	870795.06	785086.00	-0.10
15	Enfield	All	533218.77	480183.00	-0.10
16	Bexley	All	456886.19	410002.00	-0.10
17	Islington	All	757742.76	677305.00	-0.11
18	Ealing	All	645069.13	572575.00	-0.11
19	Greenwich	All	533193.50	472599.00	-0.11
20	Brent	All	639480.63	558093.00	-0.13
21	Kingston Upon Thames	All	657636.32	573489.00	-0.13
22	Newham	All	471479.67	405808.00	-0.14
23	Barnet	All	706765.75	606006.00	-0.14
24	Camden	All	928639.42	792454.00	-0.15
25	Croydon	All	464581.92	393479.00	-0.15
26	Lambeth	All	654875.76	550720.00	-0.16
27	Hammersmith And Fulham	All	883837.60	741308.00	-0.16
28	Wandsworth	All	834257.35	695867.00	-0.17
29	Tower Hamlets	All	595217.88	470209.00	-0.21
30	Kensington And Chelsea	All	1637183.90	1194726.00	-0.27
31	City Of Westminster	All	1273329.16	889935.00	-0.30
32	City Of London	All	993657.65	607399.00	-0.39

Jupyter Notebook

You can find the jupyter notebook for this post here .

TIL how to set up multiple SSH keys (for multiple Github accounts)

Fri, 28 Nov 2025 16:20:00 +0000

Setting up SSH keys is one of these things I need to do every couple of months. Inevitably, I don’t remember how to do it and then follow some tutorial, e.g. the Github Docs . It kind of always works but I haven’t fully internalized how to properly manage those different SSH keys.

Disclaimer: This learning was provoked by setting up multiple SSH keys for different Github accounts (I know how to do that) and then being able to git clone {private-repository} a private repository with the correct ssh key (I didn’t know how to do that). An example use case could be a personal and professional Github account that don’t have access to the same private repositories. So ideally when we git clone we want to automatically pick the right ssh key.

1. Generate the SSH keys for each account

Let’s assume we want to generate new separate ssh keys for each account. I like to have them all in my ~/.ssh folder. We can use the key generation command

$ cd ~/.ssh
$ ssh-keygen -t ed25519 -C "your_email or your_name"

where we specify the encryption type (-t) as ed25519 which is Edwards-curve Digital Signature Algorithm and generally recommended over RSA-based keys . The comment (-C) argument allows us to optionally add a comment that is added at the back of the key so that you remember who it created. People often use their email address. However, this is not used anywhere to select the key.

$ Generating public/private ed25519 key pair.
$ Enter file in which to save the key (/Users/youruser/.ssh/id_ed25519): /Users/youruser/.ssh/github_personal

$ Enter passphrase for "github_personal" (empty for no passphrase):

This generates both the private and public key github_personal (private) and github_personal.pub (public) in the specified path. If we entered a passphrase the private key will be encrypted as well using 3DES . We can check out what the keys look like:

$ cat github_personal.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDycaSFXnekEe32AZFhhDp3Q14JRXF7bu8CNfGf7Z6hN your_email

and

$ cat github_personal

-----BEGIN OPENSSH PRIVATE KEY-----
somegibberylongkeyperhapsfurtherencryptedusingpassphrase
-----END OPENSSH PRIVATE KEY-----

Let’s also use the same method to create a keypair for work (github_work, github_work.pub). Once we have these keys, we can add them to our two Github accounts.

2. Associate the public keys with the Github accounts

In both Github accounts, navigate to Access:SSH and GPG Keys -> New SSH key and add the public key as an Authentication Key to the account. Github now has the ability to verify messages that were signed by the corresponding private key of the pair.

3. Set up OpenSSH client config file

The file ~/.ssh/config allows us to configure “Host” specifications that we can refer to when using SSH (e.g. when git clone uses the SSH protocol for data transfer). The file looks like this:

Host hostname1
    SSH_OPTION value
    SSH_OPTION value

Host hostname2
    SSH_OPTION value

In our case, we define the two different github identities that we want to be able to select:

Host github-personal
    User git
    PreferredAuthentications publickey
    Hostname github.com
    IdentityFile ~/.ssh/github_personal

Host github-work
    User git
    PreferredAuthentications publickey
    Hostname github.com
    IdentityFile ~/.ssh/github_work

This defines which keys to use when we connect using SSH via user@host. So when we the test the config by running ssh -T git@github-personal SSH will use the SSH key ~/.ssh/github_personal as specified in the file. (Note: We can name the host and user whatever we want.)

4. Use the right hostname when cloning repositories

Let’s say we want to git clone a repository that only our work account has access to, then I can use the git clone command git clone {user}/{host}:{repo_path}.git:

$ git clone git@github-work:/org1/project1.git

or more explicitely:

$ git clone ssh://git@github-work/org1/project.git

git/ssh will look into the config file and select the configured private key for the provided host.

GitHub will send a challenge (message) to our machine for authentication. The private key will be used to sign this challenge and our machine then sends the signed message back to the Github server. The Github server will use the stored public key (from step 2) to verify the signature and allow a connection.

Note: If in step 1) we decided to encrypt the private key with a passphrase we will be prompted to provide it. Normally, at that point MacOS will offer us to add this passphrase to our Keychain .

TIL how to re-run only failed tests

Fri, 21 Nov 2025 13:00:00 +0000

This is a super small thing I stumbled over. When I implement a new feature, there is inevitably a step where I go through the failing pytest tests and fix or clarify them.

Unless I use an IDE, my work loop is like this:

Make code change
Run python -m pytest
Check which tests are failing.
Go back to 1.

Now, 2. runs all tests which can take quite some time. There is always the option to run a specific test only, but multiple failing tests can be scattered across different files.

So instead, what you can do is to only run the tests that failed at your last test run. Simply use:

python -m pytest --last-failed

Once you have fixed those tests, it’s worth running all tests one more time to check that nothing else was broken by the new changes.

Another option is to use the test view of your IDE, if you have access to one in that environment.

TIL how to investigate missing data easily with DuckDB

Fri, 16 May 2025 14:00:00 +0000

In data engineering pipelines we commonly store output data in S3 buckets at various stages. In this particular case we stored the input data received from customers as csv files in the S3 bucket s3://customer-ingested/customerA.csv and after some transformations we stored the transformed data as csv files in another S3 bucket s3://customer-transformed/customerA.csv.

Problem

For some of the customers the csv file was missing in the first bucket customer-ingested while we were sure that all customers had a file in customer-transformed. However, there was a legit reason why the file could be missing which was that in the application the customer had a state of is_deactivated = True.

The goal of this investigation was to find all the active customers for which we were missing data in the bucket customer-ingested.

Solution

The nice thing about DuckDB in this scenario is that I can combine filesystem data with information from an application table and work in (familiar) SQL land.

Here is the query:

with transformed as (
	select regexp_extract(file, '/([^/]+)\.csv$', 1) AS filenames_transformed from glob('s3://customer-transformed/*.csv')
	),
ingested as (
	select regexp_extract(file, '/([^/]+)\.csv$', 1) AS filenames_ingested from glob('s3://customer-ingested/*.csv')
	),
customer_status as (
	select customer_name, is_deactivated from read_parquet('s3://application_table_snapshot/customers.parquet')
)

select * from transformed
left join ingested on transformed.filenames_transformed = ingested.filenames_ingested 
left join customer_status as cs on transformed.filenames_transformed = cs.customer_name
where is_deactivated = False and filenames_ingested is null;

┌───────────────────────┬────────────────────┬────────────────────┬────────────────┐
│ filenames_transformed │ filenames_ingested │   customer_name    │ is_deactivated │
│        varchar        │      varchar       │      varchar       │    varchar     │
├───────────────────────┼────────────────────┼────────────────────┼────────────────┤
│ customerT             │ NULL               │ customerT          │ False          │
│ customerW             │ NULL               │ customerW          │ False          │
│ customerZ             │ NULL               │ customerZ          │ False          │
└───────────────────────┴────────────────────┴────────────────────┴────────────────┘

In the CTE, I am creating a table transformed with all the filenames in the S3 bucket customer-transformed. I am using DuckDB’s utility function glob(search_path) to retrieve the filenames. I do the same for the bucket customer-ingested.

To get the customer status I read from a recent replication snapshot of the application database to get customer_status.

Finally, I join the three expressions to identify missing files (missing values) in the column filenames_ingested for active customers.

TIL that DuckDB now supports Avro files

Wed, 12 Mar 2025 16:00:00 +0000

In the data engineering pipelines of my current company we use two main file formats:

Parquet files for analytical workloads (columnar-storage)
Avro files for transactional / event-based messages (row-storage)

In Querying remote S3 files I wrote about how I use DuckDB to query parquet files stored in S3. Recently, I noticed that DuckDB started supporting reading of Avro files in the same way.

Install extension

To install the extension open DuckDB and run:

INSTALL avro FROM community;
LOAD avro;

Read local files

Local files can be queried with the read_avro function:

select * FROM read_avro('my_dataset.avro');

Read files stored in S3

Remote files can be queried by providing the S3 URI (after logging into the AWS account ):

select * FROM read_avro('s3://my_bucket/my_remote_dataset.avro');

A few limitations of the extension are listed here .

Building a local data warehouse with DuckDB, dbt, and Superset

Sun, 09 Mar 2025 15:00:00 +0000

In previous blog posts, I described two DuckDB use cases for data scientists and data engineers: Querying remote parquet files and processing larger-than memory datasets .

Now I want to explore if I can set up a local and open-source version of the analytics data stack that we use at my company. At work, we are using Snowflake, dbt cloud, and Google’s Looker which cost us several thousand EUR per month. I am going to use the following open-source tools in my local setup:

	cloud	local
analytical database	snowflake	DuckDB
data modelling tool	dbt cloud	local dbt
dashboard tool	Looker	Apache Superset

Figure1 shows how the components work together.

Figure 1. Components of my local data warehouse stack.

I will use a toy visualization problem to demonstrate the setup. My goal is to load the race time dataset that I used in Data analysis: Vätternrundan 2024 results into DuckDB. I then use dbt to create data models and Superset to create an interactive dashboard to visualize the data.

Loading the raw data into DuckDB

I want to use DuckDB for all data processing. Therefore, I will load the raw race time data into DuckDB.

I start by installing DuckDB as my analytical database. Since I use homebrew as a package manager, I can run:

brew install duckdb

I then create a persistent database (a file on my machine):

duckdb database.duckdb

and create a new DuckDB table from the parquet file that contains the raw data.

CREATE TABLE vatternrundan AS select startnumber, city, country, result_time, start_time from read_parquet('results_vatternrundan24.parquet');

To verify that everything works, I run:

describe table vatternrundan;

┌─────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │  null   │   key   │ default │  extra  │
│   varchar   │   varchar   │ varchar │ varchar │ varchar │ varchar │
├─────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ startnumber │ BIGINT      │ YES     │ NULL    │ NULL    │ NULL    │
│ city        │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ country     │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ result_time │ BIGINT      │ YES     │ NULL    │ NULL    │ NULL    │
│ start_time  │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
└─────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘

Setting up dbt for data transformations

I want to use dbt to create data models on top of the raw data. I start by creating a new poetry project to install dependencies:

poetry init
poetry add duckdb-dbt

Then I initialize the dbt project local_warehouse with

poetry run dbt init local_warehouse

Next, I configure dbt to use DuckDB as the data processing backend. In my work setup, this would point at a snowflake instance instead. I point at the local database file in the dbt profiles file profiles.yml:

local_warehouse:
  outputs:
    dev:
      type: duckdb
      path: ../duckdb/database.duckdb
  target: dev

dbt models

I define the DuckDB table vatternrundan as a dbt source in sources.yml:

version: 2
sources:
  - name: vatternrundan_db
    schema: main 
    tables:
    - name: vatternrundan

This means I can now refer to this raw data in dbt. Moreover, I want to create two more models. One thin staging layer over the raw data stg_vatternrundan.sql, and one refined model that aggregates the rider data by country average_speed_by_country.sql.

The staging model is stored in the dbt project as models/staging/stg_vatternrundan.sql:

select 
    startnumber::int64 as startnumber,
    city::varchar as city,
    country::varchar as country,
    to_microseconds((result_time/1000)::int64) as result_time,
from {{ source('vatternrundan_db', 'vatternrundan') }}

As you can see, I refer to the DuckDB source table, define the column types, and do some light conversions.

The aggregation model is stored in models/refined/average_speed_by_country.sql:

with speeds as (
    select startnumber,
    country,
    60 * datepart('hours', result_time) + datepart('minutes', result_time) as result_time_minutes,
    315 * 60 / result_time_minutes as average_speed,
    from {{ref('stg_vatternrundan')}}
)

select country, count(*) as num_riders, mean(average_speed) as average_speed_of_country 
from speeds
group by country
order by average_speed_of_country desc

I use this dbt model to compute the average rider speed for every country in the dataset.

Now that I have defined my dbt models, I configure dbt to run them by adding the following to dbt_project.yml:

models:
  local_warehouse:
    staging:
      +schema: staging
      +materialized: table
    refined:
      +schema: refined
      +materialized: table

This tells dbt how to materialize the dbt models in DuckDB. So let’s run dbt to build the models:

poetry run dbt run --profiles-dir=.

Check tables were created

I can verify in DuckDB that these tables have been created

Run

duckdb database.duckdb

followed by:

show all tables;

which should now show:

database.main_staging.stg_vatternrundan
database.main_refined.average_speed_by_country

Superset for data analytics dashboards

After modelling my data, I want to create a dashboard and visualize the data. I use Apache Superset as my dashboarding tool. Superset works with many analytical databases. However, setting it up to work with DuckDB is a bit clunky and I had to troubleshoot quite a bit.

Install Superset with DuckDB support

I follow the Quickstart Guide with some modifications. First, I download the git repository and checkout the last tagged commit:

git clone https://github.com/apache/superset
cd superset
git checkout tags/4.1.1

I need to make two modifications before building the container.

Install duckdb-engine inside the container. This allows us later to select DuckDB as a database in Superset. To do this, add

RUN pip install duckdb-engine

to the Dockerfile.

I need to make the DuckDB database file available in the Docker container. I add its path as a volume. Modify docker-compose-image-tag.yml:

superset:
[...]
    volumes:
        - ./docker:/app/docker
        - superset_home:/app/superset_home
        - <local-machine-path-to-duckdb-database>:/app/duckdb

and force the Superset container to build from the Dockerfile and not from the image:

superset:
[...]
    build:
        context: .
        dockerfile: Dockerfile
#   image: apache/superset: comment out or remove

Then build the container with

docker compose build

and start Superset with

docker compose -f docker-compose-image-tag.yml up

Then, I can access Superset with my browser under http://localhost:8088 and login with username: admin and password: admin.

Configure Superset

Now, I can add DuckDB as a database in the Superset UI. I navigate to Settings -> Data -> Database Connections + Database -> DuckDB

and add the link to the DuckDB database file as the SQLAlchemy URI:

duckdb:////app/duckdb/database.duckdb?access_mode=READ_ONLY

Clicking on Test Connection should respond with “Connection looks good!”.

Add datasets in Superset

In the navbar under Datasets I can add the DuckDB tables stg_vatternrundan and average_speed_by_country as new Superset datasets.

Create dashboard in Superset

Creating the dashboard is self-explanatory. In the navbar, I select Dashboards and add a new Dashboard called “Vätternrundan Dashboard”. Next, I create two charts:

number of riders per country
average speed of riders per country

Just select the relevant Superset Dataset, build the charts, and add them to the dashboard. Figure 2 shows how I did it.

Figure 2. Visualization of Vätternrundan rider speed by country in Apache Superset.

Conclusion

The combination of DuckDB, dbt, and Apache Superset is a local version of “the modern datastack”. This can be useful for personal projects, and to experiment with data transformations and visualizations.

I found that the integration of DuckDB and dbt worked seamlessly. On the other hand, connecting Superset to DuckDB was clunky. Superset would benefit from better DuckDB support out-of-the-box.

Troubleshooting

DuckDB only allows one connection with read-write-access, but multiple concurrent read-only connections. Make sure you are not connected to your DuckDB database with more than one client as access mode allows write by default, e.g. Superset and the DuckDB CLI client in your terminal.
Ensure to connect Superset to DuckDB in read only mode, i.e. add access_mode=READ_ONLY to the SQLAlchemy URI to connect.

Book notes: Poor Charlie’s Almanack

Fri, 21 Feb 2025 00:00:00 +0000

The book is a collection of speeches that Charlie Munger, partner of Warren Buffett at Berkshire Hathaway, gave over the years (1986 - 2007) at universities and institutions.

I found four interesting themes across the speeches:

Advocating interdisciplinary collaboration between university departments

He believes that the social sciences, especially economics and psychology, are focusing on too narrow theoretical problems. Academics suffer from a man-with-a-hammer-syndrome. He suggests that they collaborate more across departments, e.g. economics borrowing from psychology, and within departments, e.g. macroeconomists shouldn’t avoid microeconomic explanations. Moreover, the social science should try to incorporate findings / models from the hard sciences (physics, chemistry, etc.) and attribute properly.

Fraud and advice for endowment funds

In some of his talks, he uses fictional examples of companies to show the problems with accounting and financial engineering fraud.

He thinks that charitable foundations and endowment funds should as much as possible avoid wasteful investment practices. He criticizes the trend of university endowment funds to employ layers of analysts and consultants to use fund of funds. These multilayered systems erode a large chunk of the endowment funds’ returns when compared to simpler alternatives, like unlevered domestic equity indices. He directly criticizes the practices of his audience of consultants and fund managers in talk six, but he uses humour and self-deprecation to get away with it.

(Inverted) Advice for Graduates

In his commencement speeches for university graduates, he gives advice on how to guarantee misery in life.

Be unreliable.
Don’t learn from other’s mistakes. Instead, make common mistakes of others again (join a cult, drive while drunk, gamble) and don’t learn from people that came before you.
Give up after adversity and failures. Just give up when the inevitable hard times occur.
Don’t ever invert. Don’t attempt to learn from thinking about achieving the opposite of your goals. Don’t try to be objective.

Mental checklist to analyse problems

A recurring topic in his speeches is a checklist of mental models and human biases. He developed this checklist over time and iterated through it when analysing a problem or evaluating an investment. Using appropriate checklists and the inversion technique are two main tools that he promotes for better thinking.

Here are 25 human biases from talk eleven: The Psychology of Human Misjudgment .

Reward and Punishment Superresponse Tendency: If you want to persuade people, appeal to their own interests. Ensure the incentives of people you work with are aligned with the outcome you want. Don’t reward them for metrics that they can easily game.
Liking/Loving Tendency: People are seeking love and approval from other people. Moreover, we favour people and products that are merely associated with the target of our affection. We can use this to our advantage by liking truly admirable people or ideas.
Disliking/Hating Tendency: People can have a tendency to dislike things different to them or products or people that are associated with the object of their dislike.
Doubt-Avoidance Tendency: Our brains are conditioned to quickly remove doubt after reaching a first decision. This tendency should be countered by forcing a delay for reflection before an important decision, e.g. jury decisions in court.
Inconsistency-Avoidance Tendency: A tendency to stick to previous conclusions, habits, and ideas to avoid change. Practically, this means that it is much easier to prevent a bad habit than to change it. To counter this tendency, we should force the discussion of counterarguments before a decision can be made. This tendency can be used to manipulate people (see also Ben Franklin effect , Cialdini’s consistency principle .
Curiosity Tendency: Humans have a general tendency to be curious, which can be supercharged with today’s access to information. This fortunate tendency should be used to counteract other psychological tendencies.
Kantian Fairness Tendency: People have a tendency in direct interactions to behave fairly (following Kant’s categorical imperative).
Envy/Jealously Tendency: People’s tendency to envy someone else’s status, wealth, or compensation. “It is not greed that drives the world but envy.” (Buffett)
Reciprocation Tendency: People have a tendency to reciprocate favours and disfavours. This can also be used for manipulation, e.g. a salesman could do you a small favour to get a much better outcome in negotiation.
Influence-from-Mere-Association Tendency: Valuing something by the association with another unrelated factor/idea/concept. Examples:
- association of quality with the highest price
- purchasing of luxury items to boost status
- advertising of products with unrelated but positive images
- associating one’s ability with past successes and making bad decisions
- thinking of someone worse because they are a competitor
Simple Pain-Avoiding Psychological Denial: Denying reality because it is too painful to accept, e.g. addiction, bankruptcy.
- Denial of a reality that’s too painful to accept, e.g. addiction to alcohol.
Excessive Self-Regard Tendency: People’s tendency to overestimate their abilities, decisions, and possessions. Moreover, their preference of people that are similar to them.
Overoptimism Tendency: A tendency to be overly optimistic of the future, especially if one has done well in the past.
Deprival-Superreaction Tendency: Reacting stronger to losses than to gains, e.g. losing $10 is considered worse than gaining $10. Irrational overreaction to threatened loss of status, territory, love, friendship, or property.
Social-Proof Tendency: A tendency to act and think the same way as people around you (Group-think).
Contrast-Misreaction Tendency: Making bad decisions by anchoring on an irrelevant comparison. Examples:
- Adding $1000 of useless add-ons to a car only because the car costs $65k.
- A real estate agent presenting 3 terrible and expensive houses, then showing a merely bad house to make it look more desirable.
Stress-Influence Tendency: Light stress can increase performance temporarily, while heavy stress can cause dysfunctional thinking and bad decision-making.
Availability-Misweighting Tendency: Tendency to overweight or overvalue things (people, decisions, work, ideas) that are close or readily accessible to you. This also holds for metrics that are easy to measure and actions that easy to take. This can be countered by following checklists of actions and by considering more difficult (or less accessible) alternatives.
Use-It-or-Lose-It Tendency: Over time, our skills and knowledge fades. Therefore we should deliberately train and repeat the skills we want to retain. Write them down as a checklist and work through them regularly.
Drug-Misinfluence Tendency: Most people can’t handle drugs responsibly over a long period of time. Not worth trying to prove that you can.
Senescence-Misinfluence Tendency: Older people have a harder time to learn new skills. Knowing this, the best counter is to actively maintain the accumulated knowledge, see 19).
Authority-Misinfluence Tendency: Tendency to blindly follow the leader (see also: HiPPO effect ).
Twaddle Tendency: Some people waste time talking about things they are not an expert in. Try to separate these people from and follow the experts.
Reason-Respecting Tendency: People can learn better when they can think through the reasons behind a directive or action. Therefore, when giving orders explain your reasoning.
Lollapalooza Tendency: Often multiple human biases act together to drive a certain behaviour or outcome.

Conclusion

I gave only 3/5 stars because the format of reading his talks didn’t appeal to me. I was already aware of most of the human biases listed by Munger. Some are discussed in Thinking Fast and Slow (2011) and Influence: Science and Practice (2001) . However, what I found impressive is that he talked about and used these biases already 20 years ago, before they were widely discussed. I also liked that in many speeches he directly criticizes the audience that invited him to speak. However, he does it in a charming and humorous way.

TIL how to fix a broken Joplin sync

Mon, 17 Feb 2025 00:00:00 +0000

I use Joplin for all my note-taking as it’s open-source, markdown-based, and has decent Desktop and Android apps. I use note encryption and sync my notes via WebDAV with my remote Nextcloud instance.

Problem

After not syncing my notes for several weeks, I encountered a couple of errors both on my Desktop app and on my Android app. I don’t have access to the error message anymore, but the errors referred to files and file IDs that caused inconsistencies on the remote server, e.g. broken references:

f1231gdffgasd1asd2.md TypeError: Cannot read property 'type_ of null.

Solution

To solve the issue, I had to delete the offending file on the remote server. However, to not lose it, it’s best to make a local copy of the file’s content if it is present on one of your local devices.

First, I had to determine which file caused the issue, which is a bit harder here as filenames are encrypted. The easiest way to identify the note from the filename is to paste a local markdown reference into a new Joplin note like this:

[Note I am looking for](:/f1231gdffgasd1asd2)

and then click the link in the rendered note. This took me to the offending note.

Second, copy the content into a new note.

Third, I had to delete the file on the remote. I connected on my Mac to the remote WebDAV server via Finder. In Finder, Click on Go -> Connect to Server -> Add address: https://yourserveraddress.org/remote.php/webdav/joplin (look up the link in your Joplin settings), and authenticated with my Nextcloud credentials. Then I was able to search for the file f1231gdffgasd1asd2 and delete it.

The sync now succeeds.

TIL how to debug a broken nextcloud login (again)

Sun, 16 Feb 2025 00:00:00 +0000

The symptoms of this problem were the same as the ones I described in TIL how to debug a broken nextcloud login .

Problem

Phone and Mac apps stopped syncing data with the Nextcloud instance. In my browser I could access the login page, but after entering the details I was being sent back to the same login page without any error message.

Investigation

Last time the issue was caused by excessive logging. However, I reduced the loglevel in the nextcloud config, so this was not the issue here. I am no expert in web admin, PHP, or nextcloud, so I used Claude 3.5 to pair on debugging:

Prompt

I am self-hosting a nextcloud instance on a Hetzner VPC. I currently have the following problem. I can’t log in to nextcloud via my account. The screen just reloads and shows me the login page again. It is not a login detail issue. They are correct. Moreover, other nextcloud related syncs (files, calendar) stopped working. Walk me through how I can debug this issue and get nextcloud up and running again.

I can’t say that the suggestions worked right away, but they put me on the right path and helped me with some useful debugging commands.

The command

df -h

showed me that I didn’t have any free space. While the command

du -sh /var/www/nextcloud/data/*

indicated that in nextcloud a lot of space was occupied by old file versions and deleted files from the nextcloud file sync.

Solution

You can use the nextcloud console occ (ownCloud Console) tool to run maintenance tasks on the nextcloud instance.

The command

sudo -u www-data php /var/www/nextcloud/occ list

lists all available commands.

A couple of clean up commands to remove old file versions and files in the trashbin worked for me:

# Check Nextcloud trashbin size
sudo -u www-data php /var/www/nextcloud/occ trashbin:cleanup --all-users

# Clean up file versions
sudo -u www-data php /var/www/nextcloud/occ versions:cleanup

# Clear Nextcloud cache:
sudo -u www-data php /var/www/nextcloud/occ cache:clear

This freed up enough space for nextcloud to work again.

Unfortunately, the occ commands didn’t work right away because there was apparently not enough space to run any of them. Each one returned the same error message:

fwrite(): write of 4963 bytes failed with errno=28 No space left on device

So I first had to free up a few hundred megabytes:

Clear local repository of retrieved package files:
```
sudo apt-get clean
```

Remove old log files:

sudo find /var/log -type f -name "*.gz" -delete
sudo find /var/log -type f -name "*.1" -delete
sudo truncate -s 0 /var/log/*.log

Remove temporary files:
```
sudo rm -rf /tmp/*
```

This cleared enough space to run the occ commands.

How related posts are computed

Sun, 09 Feb 2025 00:00:00 +0000

The number next to related posts at the bottom of each page is the advertised post’s “similarity” to the currently viewed page (from 1.0 to -1.0).

I am using the following process to compute related posts locally:

Summarize every post and TIL using local Llama 3.1 (llama3.1:8b-instruct-q5_0 via Ollama) with the following prompt:

Prompt

You are an analyst and editor with many years of experience in reading and synthesizing content.

Here is a blog post:

<BLOGPOST>

{ blog_post }

</BLOGPOST>

Please create a comprehensive and concise summary of the blog post. Focus on the main concepts, key details, and central arguments.

<INSTRUCTIONS>

Include any specific technologies, methods, or frameworks mentioned.
Don’t use more than 7 sentences.
Respond in plaintext. Don’t add formatting or linebreak characters to your response.
Don’t repeat the instructions of the task. Respond directly.

</INSTRUCTIONS>

Embed the summary using ChromaDB’s default embedding model all-MiniLM-L6-v2 and store the embedding and metadata about the post in a persistent ChromaDB vector database (a file on my computer).
Compute the cosine similarity between the embeddings for each pair of posts. A score of 1.0 indicates proportional vectors, a score of 0.0 orthogonal vectors, and a score of -1.0 opposite vectors.
Write a yaml file that includes for every post a link to the most similar post and their similarity.
Use a Hugo partial to include the data in the yaml file about the most relevant posts at the bottom of each page.

Noteworthy

Maybe not surprising, but Llama 3.1 8b performed much better than Llama 3.2 3b. The latter would hallucinate quite a few parts of the summary.
I had to explicitly instruct the model not to restate the task. Otherwise, every summary would have started with “Here is your concise blog post summary in not more than 7 sentences.” which would create some artificial similarity in the embeddings.
Most blog posts are shorter than 15000 characters or 4000 tokens which I had to work out to configure the model context correctly.
ChromaDB’s default embedding model all-MiniLM-L6-v2 has 384 dimensions and a maximum input sequence length of 256 BERT tokens (~100–150 words). I checked some of the summaries and they were around 160 – 220 BERT token. There is the option to use another embedding model that has a higher maximum sequence length. A good embedding model leaderboard to help make a choice is available here .

Example

For the blog post Book notes: Four Thousand Weeks by Oliver Burkeman the LLM came up with this summary which was then embedded:

Model: llama3.1:8b-instruct-q5_0 - response

“The blog post discusses the book “Four Thousand Weeks” by Oliver Burkeman, which explores the concept of time and how to make the most of our limited lives. The author argues that modern society’s focus on productivity and efficiency is flawed, as it leads to a never-ending cycle of work and stress. Instead, he advocates for embracing our finitude and focusing on the present moment. Key concepts include “atelic activities” (enjoying leisure time for its own sake), “radical incrementalism” (making small progress towards goals), and “cosmic insignificance therapy” (putting life’s problems into perspective by considering the vastness of the universe). The author also suggests tools for embracing finitude, such as keeping a done list to focus on completed tasks. Overall, the book encourages readers to reevaluate their priorities and find meaning in the present moment.”

Embeddings

I can visualize the embeddings in two dimensions using t-SNE :

Figure 1. Plot of blog post summary embeddings in t-SNE space.

The plot looks somewhat reasonable. The code-heavy jupyter notebook posts are at the bottom, clustered around duckdb-large-datasets. The book reviews (4000-weeks, how_big_things-get-done, how-to-win-friends) are fairly close together. SQL related posts are clustered at the top. On the other hand, I would have expected reading-and-note-taking to be closer to writing-well.

Code

I am using this langchain script to compute the recommendations.

Book notes: How Big Things Get Done by Bent Flyvbjerg and Dan Gardner

Sun, 02 Feb 2025 00:00:00 +0000

Bent Flyvbjerg is an economics professor at IT University of Copenhagen. He maintains a database of megaprojects (power plants, opera halls, tunnels, airports) and their planned and realized timelines and budgets. He researches the reasons why modern megaprojects often fail to deliver on time and on budget. It has recently become quite popular to discuss the apparent decrease in speed with which large-scale projects are realized. Examples are easy to find, e.g. the delayed High Speed Rail 2 (HS2) project or the time and cost overruns of the Berlin Brandenburg Airport (BER) . Counter-examples are less common (or less newsworthy), e.g. the 2023 Notre-Dame Reconstruction.

In their book Flyvbjerg/Gardner explain the common reasons for cost and time overruns of megaprojects and how to mitigate them. I think we can apply the majority of their learnings for smaller home (kitchen renovation) and work projects (cloud migration) as well.

Planning is cheap, building is expensive

In large construction projects, planning is a lot cheaper than building. Once a badly planned project is underway, unforeseen problems will be discovered. Delays tend to cause further delays. The total project time can be in itself a source of additional risk and cost.

When politicians plan prestigious infrastructure projects, they have a bias for strategic misrepresentation of the cost and time to get support for it. Therefore, the government and the public need to scrutinize the project’s goal and its planning to ensure that the plan is realistic and enough alternatives have been considered. Only get involved in such a project if it has the people and funds, including contingencies, to succeed.

Since planning is a lot cheaper than building for large projects, Flyvbjerg/Gardner advise investing significant time into the planning phase:

understand the objective behind the proposed project (assume an outside view)
explore alternatives and don’t commit to the first available solution (resist quick action bias)
don’t forecast using the best-case scenario, instead use similar past projects to anchor your estimate
experiment and iterate in the planning phase using digital modelling and simulations

Understand the why

The project is not a goal in itself, it is how the goal is achieved. We want to talk to the stakeholder to understand the goal. Is the project even the right approach? Good planning needs to explore the problem before jumping to the solution. This includes considering alternatives.

There is a good example in the book about a bridge project. The original goal of the project is to connect an island with the mainland. By jumping to the bridge idea, the stakeholders ignored alternatives, e.g. a tunnel, ferries, a helipad. Or maybe a physical connection is not required. If it’s about improved communication, maybe a high-speed broadband connection achieves the goal.

Another framing of this idea in product development is to work backwards from the customer. Understand their needs and problems, before coming up with any solution.

Reduce uncertainty via experimentation

A crucial part of the planning phase is to experiment with the solution. Ideally, we can simulate and iterate on the project, e.g. use 3D models of the building, low fidelity designs to show customers, or low effort versions of the animation movie we are trying to produce.

The simulation ensures that the majority of aspects of our project are scrutinized before the building phase begins. We can assume that the project will run into problems, so we want most of the problems to occur during the cheap planning phase.

Experience in people and technology

To maximize the chance of a successful project we should maximize experience. Flyvbjerg/Gardner consider both experience in key people and in technology. We should try to ensure that key people have experience in similar projects. Ask: “Have they done it before?”.

“Technology is ‘frozen experience’.”

All things being equal, we should use tried and tested off-the-shelf technology instead of shiny new technology. If we can, we should use existing designs and operational processes.This is the same idea behind McKinkley’s advice to Choose Boring Technology .

The construction of the Empire State Building is used throughout the book as a positive example. One reason for this was the architect William Lamb who insisted to only use proven technology, and who created a design that allowed repeatable non-custom work steps. Moreover, the construction company Starrett Brothers and Eken had built several similar skyscrapers on time and budget before.

Reference-class forecasting

After the project is planned we need to forecast the project duration and its cost. Unfortunately, planners often forecast too optimistic. Instead of deriving it from the project alone, the authors suggest to use reference-class forecasting. The idea is to find a set of comparable projects, and to anchor our forecast based on their outcomes. Then adjust from that anchor.

However, good forecasting can not protect the project from fat-tailed risks. Since those risks can kill our project, we need to identify the (known) high risk events and try to mitigate them.

“Successful project leaders focus every day on not losing, while keeping a keen eye on the […] goal they are trying to achieve.”

After the planning is done, build quickly

The project duration is in itself is a source of budget and time risk. After planning, simulating, and forecasting, we therefore need to act fast once the building phase starts. To do this we should ideally use an experienced team and a modular building pattern.

Build with an experienced team

If possible, hire, what the authors call, a masterbuilder. That is someone who has experience in similar projects and can pick the right team. If such a team doesn’t exist, you or them need to create it.

Moreover, we need to ensure that the incentives of participating contractors are aligned with ours by sharing risks and rewards, e.g. pay a bonus for early completion. Don’t always pick the contractor that submits the lowest bid, because the lowest bid doesn’t necessarily lead to the lowest cost. Moreover, we will try to choose companies that we have successfully collaborated with before.

What’s our Lego?

Nuclear power plant constructions belong to the class of projects that most likely exceed cost and time budgets. Flyvbjerg/Gardner argue that you are building one large thing that has few repeatable, standardized parts. Solar power plants are the opposite. The core ingredient, the solar modules, can be produced rapidly and repeatedly in factories, and then assembled in a modular fashion.

When possible we should try to build modules. We can then produce and assemble the modules in a repeatable process. This delivers value in stages. After we have completed a module, we can use the learnings to iterate on the module’s design and assembly process. So we should always ask ourselves:

“What’s our basic building block, the thing we will repeatedly make, becoming smarter and better each time. What’s our Lego?”

Additional thoughts about the book

Incentives and identity

There is a section in the book that discusses why the Heathrow Terminal 5 project was completed on time. The authors argue that a large part was the alignment of incentives between the project managers, the contractors, and the workers. Contractors were aligned via contractual bonuses and established working relationship. Workers were treated well, their feedback was actively encouraged, and they felt like they are contributing to a historic project in their country. Not part of the book, but the Notre-Dame reconstruction after the fire, was also accomplished on time. I believe that the history of the building, and the feeling of being part of a national project are very powerful motivators that aligned the participants of the project.

Cost overruns by project type

I found the table in the book that shows the mean cost overrun by category interesting.

The table shows the mean (base rate) cost overrun for each project type. The fat-ness of the distribution (% projects in the upper tail), and the (base rate) cost overrun for projects in the upper tail.

Project type	Mean cost overrun (%)	% of projects in tail (>= 50% overrun)	Mean overrun of projects in tail (%)
Nuclear storage	238	48	427
Olympic Games	157	76	200
Nuclear power	120	55	204
IT	73	18	447
…	…	…	…
Buildings	62	39	206
Rail	39	28	116
Airport	39	43	88
Tunnels	37	28	103
…	…	…	…
Wind energy	13	7	97
Energy transmission	8	4	166
Solar power	1	2	50

It makes sense that (non-standard, non-modular) nuclear projects have the highest base rate for cost increase. On the opposite side are wind energy and solar power projects, that can be pre-produced in factories and assembled on-site. Organizing Olympic Games suffers from the fact that they are highly complex, and usually held in a city that hasn’t hosted them before (inexperience). As a tech worker, I am intrigued to see IT projects at the top of the list. Moreover, when IT projects overrun they incur a 447% cost overrun, the highest among all project type.

Learnings for software projects

How applicable is the advice for day-to-day data and software projects? I think much of the advice applies. I found that chunking larger projects into small value-delivering modules is a great way to ensure continuous progress (especially in large refactoring projects).

The emphasis on planning vs. building is probably not as relevant in normal day-to-day software projects. There is less of a cost difference between planning and building a feature (in both cases mostly the software engineers’ time) compared to construction projects.

LLM prompt heuristics that definitely maybe work

Fri, 11 Oct 2024 14:56:00 +0000

Effective prompt writing for large language models continues to be a dark art. Having read the prompt engineering blogs from Meta, Anthropic, and OpenAI and watched some of Anthropic’s prompt discussions online, it does feel more like design than engineering. Even employees from the same lab don’t agree what tricks actually work. If you have time watch this video of one of Anthropic’s prompt engineers running a prompting masterclass. While giving good heuristics to the audience he is careful not to make any definite statements.

I don’t believe it is a good investment to try to become a world-class prompter. All heuristics are highly dependent on the model architecture, training data, and training procedure. This means that with every iteration of the models, heuristics could become obsolete or harmful to performance.

However, the current attention-based architecture doesn’t seem to go anywhere soon. Therefore, it is reasonable to expect that we can continue to use the prompt context to help the model to move the embeddings of the user task into directions that contain a lot of nuance and information about the domain of the task.

Therefore, I am collecting the advice into a few guidelines that I can use with the current models. I will caveat this by saying that in 90% of my use-cases the model response is parsed by a human: me. Therefore, I am not as worried about hallucinations as someone that puts the model outputs in front of their customers or inside data parsing pipelines.

Here is what seems to work in October 2024 with Claude Sonnet 3.5, GPT-4o, and Llama 3.2.

Prompt specificity

Make the prompt as specific to the task as you can. This is probably the biggest return on your time. A good advice given in this discussion is to imagine printing out the prompt and giving it to a new hire at your company. Then see if they could solve the task. This forces you to give all necessary context and constraints of the task.
Components of a role prompt: The role, its part of an organisation, its perspective and the perspective of the person or organisation being addressed.

No lazy role prompts. Make the role and the context clear.

Bad: You are a cab driver.
Good: You are a cab driver, driving people as your full time job in London for twenty years and are knowledgble about the city, its roads, and its sights.

If the model is used inside a product, tell it about it:

Bad: You are an assistant writing document summaries.
Good: You are an assistant used in a product for law firms that summarizes
legal documents.

Order of prompt components

Order the components of your prompts in the following order:

Specific role and context description (put into system prompt if part of interactive use of the model)
Input data, e.g. documents, code snippets, CSV files
Task description
(Optional:) Good examples and bad examples
Task constraints

Use XML tags for prompt components

Use XML tags to separate different components of a prompt.

You are an experienced post-doc at a reputable research institute in the US. You are an expert in the research field of the following paper:

<Research_paper>
{{research_paper}}  
<Research_paper/>

Your task is to summarize the findings of the research paper given to you in
<Research_paper> tags.

<Instructions>
- List between 1 - 5 most important findings of the paper. Don't list more
  than 5 findings.
- more instructions...
</Instructions>

Using examples (few shot prompting)

It can help to provide both good and bad examples for a specific task with an explanation why they are good or bad. In the above example we could provide an example research paper with a summary of findings we wrote ourselves as a good example.

Instructions

Response length limit. Instead of writing “Be concise”, tell the model what that means for your context, e.g. “Answer in no more than 4 sentences.”
Avoid open ended instructions.
Instruct for style, formatting, and restrictions.
Instruct to ask for sources of evidence to reduce hallucinations.

Ask it to respond in a chain-of-thought to increase performance.

You are a logician and love to solve logic puzzles. Carefully read the following puzzle.

<PUZZLE>
Simon is looking at Charlie. Charlie is looking at Sarah. You know that Simon is married and Sarah is not married. Is a married person looking at an unmarried person?
</PUZZLE>
Let's think step by step before giving an answer.

Grammar and style

Avoid typos and wrong punctuation as that can deteriorate the quality of the response.

Parse-able output

Often LLMs add a preamble at the beginning or an epilogue at the end of a response. If you want to force the model to respond only with valid json, you can:

Use the “Prefill Claude’s response” feature
Ask the model to put the json into tags and then extract that block from the response
Prefill the response yourself by adding at the end of your prompt: Here is the JSON: {. The open bracket conditions the model to start the answer with the first JSON key. You then need to prepend the “{” to the response to make it valid.

Other tricks

If your prompt includes logic that could be handled in code, handle it in code.

Bad: You are part of a role-playing game that is used for training of customer support agents in a fortune 500 company. 

You can assume any of the following roles based on user input. 

If the user asks for Role1 assume the role of a customer asking questions about the company's products. If the user asks for Role2, assume the role of a helpful customer support agent.

Good: Define two different prompts for each role and use code to switch prompts based on what role the operator wants the model to assume.

Give the model a way out if it doesn’t know the answer.

Good: {{prompt}}
If something weird happens and you are unsure about what to do, simply print out
"UNSURE".

Use a temperature of 0 for fact-based, less creative tasks.

Iterative prompt design

You can use the LLM to generate examples for a task and select the good. examples. Then use those examples in the prompt that is used “in production”.
If the model responds incorrectly, tell it about the mistake and ask it how you should modify the prompt.

TIL (from Claude) how to create separate RSS feeds for my hugo blog

Thu, 03 Oct 2024 00:00:00 +0000

I really like the idea of providing different content types of my blog in separate RSS feeds. For example a blog could have:

essays
today-I-learned posts
link-list or bloglink posts
short social media style posts
etc.

Providing different RSS feeds for them allows my readers to subscribe to all or a subset of these, depending on their interest. I am using hugo, as the static site generator for this website. Setting up these different feeds for a framework that I am only partially familar with is a great use-case to work with an LLM.

I knew that I would need to:

Distinguish different content types with a flag
Specify different RSS templates that filter on these flags
Tell hugo to generate separate xml files for each content type and use the right templates.

I just didn’t know how to piece these parts together. I first tried googling the answer and read the typical mix of

stackoverflow posts that were not solving the exact same problem
hugo forum discussions that did not have a clear solution or where the solution wasn’t transferable to my unique setup.

I think with those first pointers, spending some time on the hugo documentation, and trial-and-erroring, I could have worked out how to do it. I think that these shallow but unknown problems are the ideal use case for an LLM.

I think of using LLMs as an interface for search + context, where I don’t have to take the laborious work of pre-filtering and connecting the sources. Instead, it helps me get a quick overview and a potential draft of a solution. I can then go deeper and understand the problem better by asking about parts of the solution that I don’t understand. I can then mix this with more googling and get to a solution.

My Goal:

Prompt

I have a hugo blog with the following folder structure:

/content/posts/post1.md
/content/posts/post2.md
/content/posts/_index.md
/content/til/_index.md
/content/til/til1.md
/content/til/til2.md

Markdown pages written in the posts subfolder have type: post in the frontmatter and pages written in the til subfolder have type: til in the submatter.

Explain to me step by step how I can create three separate rss feeds on my website.

Only showing content coming from the til subfolder on til.xml
Only showing content coming from the post subfolder posts.xml
One showing everything on index.xml

Solution

Model: claude-3-5-sonnet-20241022 - response

I’d be happy to explain how you can create three separate RSS feeds for your Hugo blog with the structure you’ve described. Let’s go through this step-by-step:

Create RSS template files: First, you’ll need to create three separate RSS template files in your Hugo project’s layouts folder.

Now, let me explain each step in more detail:

We create three separate RSS template files in the layouts/_default/ directory. Each file corresponds to one of the RSS feeds you want to create.
In each RSS template, we use Hugo’s templating language to generate the RSS feed. The main difference between the templates is in the range statement:
- For til.xml, we use {{ range where .Site.RegularPages "Type" "til" }}
- For posts.xml, we use {{ range where .Site.RegularPages "Type" "post" }}
- For index.xml, we use {{ range .Site.RegularPages }} to include all content
We update the Hugo configuration file to define new output formats for TIL and Posts, and to include these new formats in the home page outputs.
We add links to the new RSS feeds in the site’s HTML, typically in the <head> section or footer.
Finally, we build the site to generate the new RSS feed files.

After following these steps, your Hugo site will generate three separate RSS feeds:

til.xml will only include content from the til subfolder
posts.xml will only include content from the posts subfolder
index.xml will include all content

Would you like me to explain any part of this process in more detail?

I followed these steps, except for using my already existing xml template for til.xml, posts.xml, index.xml. I already read in some hugo discussions that I need to add [outputs] and [outputFormats] to my hugo config.toml file. However, I didn’t really understand the difference between home, page, section and what exactly home = ["HTML", "RSS", "JSON", "TIL", "POSTS"] does. So I asked for clarification:

Prompt

Can you explain to me the config [outputs] and its purpose as well as [outputFormats]?

Model: claude-3-5-sonnet-20241022 - response

Certainly! I’d be happy to explain the [outputs] and [outputFormats] configurations in Hugo and their purposes.

Now, let me provide a more detailed explanation of these configurations:

[outputs]: This configuration section determines what types of files Hugo will generate for different types of pages on your site. In our case, we’re telling Hugo to generate additional output formats for the home page. The line home = ["HTML", "RSS", "TIL", "POSTS"] means:

For the home page, generate the standard HTML and RSS outputs
Also generate our custom TIL and POSTS outputs This is crucial for creating our multiple RSS feeds, as it instructs Hugo to actually generate these files when building the site.

[outputFormats]: This section allows you to define new output formats or modify existing ones. In our case, we’re defining two new output formats: TIL and POSTS. For each new format, we specify:

mediaType: This tells Hugo what kind of content this is. We’re using “application/rss+xml” because we’re creating RSS feeds.
baseName: This determines the filename of the output file. For example, baseName = "til" will result in a file named til.xml.

By defining these custom output formats, we’re giving Hugo the information it needs to generate our specialized RSS feeds.

The interaction between these two configuration sections is key:

[outputFormats] defines what the TIL and POSTS formats are
[outputs] tells Hugo to actually generate these formats for the home page

This setup allows you to have fine-grained control over what files Hugo generates and how they’re named, which is exactly what we need to create multiple, specialized RSS feeds. Is there any part of this you’d like me to elaborate on further?

This helped but I still didn’t really get why I would have to add this to home and not page because I thought that I want to generate rss for page-types.

Prompt

What’s the difference between adding TIL and POSTS to home instead of page?

Model: claude-3-5-sonnet-20241022 - response

That’s an excellent question! The difference between adding output formats to home versus page in Hugo’s configuration is quite significant and affects how and where these outputs are generated. Let me explain:

Now, let me elaborate on the key points:

Scope and Purpose:
- Adding TIL and POSTS to home creates site-wide RSS feeds that include all relevant content from across your site. This is ideal for your use case of creating separate feeds for TIL entries and blog posts.
- If you added these to page, it would create individual RSS feeds for each content page, which is not what you’re aiming for.
Number of Files Generated:
- With home, you get one til.xml and one posts.xml file for your entire site.
- If you used page, you’d get a til.xml and posts.xml for every single page on your site, which would be unnecessary and potentially confusing.
Content Aggregation:
- The home outputs can access and aggregate content from across your entire site, which is perfect for creating comprehensive RSS feeds.
- page outputs typically only contain information from that specific page.
Location of Generated Files:
- home outputs are usually placed at the root of your site, making them easy to find and link to (e.g., https://yoursite.com/til.xml).
- page outputs would be scattered throughout your site structure, mirroring your content organization.
Practical Implementation:
- For your RSS feeds, using home allows you to create the three distinct feeds you want (TIL, posts, and everything) in a clean, organized manner.
- Using page for this purpose would be impractical and wouldn’t achieve your goal of separate, comprehensive feeds.

In summary, for your specific need of creating site-wide RSS feeds for different content types, adding TIL and POSTS to the home outputs is the correct approach. It generates the feeds at the site level, includes all relevant content, and produces a single file for each feed type, which is exactly what you’re looking to achieve.

I followed the suggested approach and added to my config.toml:

# I want to top-level output html, rss, and my two custom outputs
[outputs]
  home = ["html", "rss", "til", "post"]

# Here I define how to process these custom feeds
[outputFormats.til]
  mediaType = "application/rss+xml"
  baseName = "til"

[outputFormats.post]
  mediaType = "application/rss+xml"
  baseName = "posts"

Then I built the page and hugo would build three different xml files:

public/index.xml
public/posts.xml
public/til.xml

However, the files all had the same content as if they were built from the same template instead of the individual templates that filter by page type.

Prompt

This generates the three xml files but they all seem to use the same template file

Model: claude-3-5-sonnet-20241022 - response

I apologize for the confusion in my previous explanation. You’re right to point out that issue. Let’s correct this and explain how to properly set up distinct templates for each RSS feed.

Now, let me explain the key changes and why they solve the problem:

Template File Naming: The crucial change is in how we name and place the template files. Instead of putting them in layouts/_default/, we’re now creating:
- layouts/index.til.xml
- layouts/index.posts.xml
- layouts/index.xml
This naming convention is important because Hugo uses it to match the output formats we defined in the configuration.
Template Content: The content of each template remains largely the same, but we’ve made sure that each one is filtering for the correct content type:
- index.til.xml uses {{ range where .Site.RegularPages "Type" "til" }}
- index.posts.xml uses {{ range where .Site.RegularPages "Type" "post" }}
- index.xml continues to use {{ range .Site.RegularPages }} for all content
Configuration: The configuration in config.toml remains the same as before. The [outputFormats] and [outputs] sections are correctly set up to generate these files.
How It Works:
- When Hugo sees TIL in the [outputs] section, it looks for a template named index.til.xml
- Similarly, for POSTS, it looks for index.posts.xml
- The default RSS output uses index.xml

This setup ensures that each RSS feed uses its own template, allowing you to customize the content for each feed independently.

The reason your previous attempt generated three XML files with the same content was likely because Hugo was falling back to a default RSS template when it couldn’t find specific templates for TIL and POSTS.

With these changes, you should now have three distinct RSS feeds, each with its appropriate content. Does this explanation help clarify the issue? Would you like me to elaborate on any part of this solution?

So I moved the template layout files as suggested to:

layouts/index.til.xml
layouts/index.posts.xml
layouts/index.xml

and they were correctly selected. You can find the correct feeds here:

https://staticnotes.org/index.xml

https://staticnotes.org/posts.xml

https://staticnotes.org/til.xml

Conclusion

A common criticism is that LLMs will confidently hallucinate incorrect information. I think this will likely be true for the next couple of years. However, when I use the old-school google + stackoverflow + documentation + online discussion workflow I also might generate an incorrect approach. This can happen because I misunderstand the content or because I can’t piece the information from different sources correctly together for my use case.

I think viewing an LLM as an interactive interface to search and combine information and context is the right mental model to use them productively.

Processing 112M rows of steam reviews locally with DuckDB

Sat, 28 Sep 2024 13:43:40 +0000

In DuckDB use cases for data scientists: Querying remote S3 files I wrote how I use DuckDB as a convenient way to query data from CSV or Parquet files in S3. Another use case for data scientists and data engineers is DuckDB’s ability to processing larger-than-memory data on your local machine. For us data scientists this fills the gap between

small data that you can transform with pandas
big data that typically requires a multi-node processing engine like PySpark.

In this post I am going to run some exploratory queries against this kaggle Steam review dataset . It contains 112M rows of Steam game reviews and comes as an uncompressed 46GB CSV file (17GB compressed). Let’s find out how my MacBook copes with it. Later I am going to run a small comparison against polars , a DataFrame library, which is often mentioned for out-of-memory processing.I am not comparing against pandas because its inability to work with large datasets was one of the reasons why I explored this topic. However, you could do this with chunking .

Useful commands and settings

Before I start, here are some useful DuckDB commands and settings that I often use:

Command	What is it good for?
`.timer on`	Prints the execution time after each SQL command
`FROM duckdb_memory();`	Gives an overview of how much memory is used by DuckDB
`SET enable_progress_bar = true;`	Prints a progress bar for query runs
`SET memory_limit = '2GB';`	Controls how much data DuckDB can keep in RAM.
`.mode line`	Prints query results one at a time (useful if many columns). Default: `.mode duckbox`
`SUMMARIZE (select *)`	Summarizes the contents of a table.
`.exit`	Stops DuckDB.

Steam review dataset

First, I want to familiarize myself with the dataset. I downloaded the 46GB file all_reviews.csv to my 2023 Macbook Pro M3 with 18GB RAM. After starting a non-persistent session with the duckdb command, I use the above commands to activate the timer and line mode. I then take a look at an example row with:

 select * from read_csv('all_reviews.csv', ignore_errors=true) limit 1;

              recommendationid = 148919893
                         appid = 10
                          game = Counter-Strike
                author_steamid = 76561199036724879
        author_num_games_owned = 0
            author_num_reviews = 3
       author_playtime_forever = 197
author_playtime_last_two_weeks = 197
     author_playtime_at_review = 197
            author_last_played = 1698336369
                      language = russian
                        review = старость
             timestamp_created = 1698336397
             timestamp_updated = 1698336397
                      voted_up = 1
                      votes_up = 0
                   votes_funny = 0
           weighted_vote_score = 0.0
                 comment_count = 0
                steam_purchase = 1
             received_for_free = 0
   written_during_early_access = 0
         hidden_in_steam_china = 1
          steam_china_location =

This gives me an overview of the file columns and an idea of their content.

Processing the file directly

Rather than slowly loading all data into DuckDB, I want to test DuckDB’s stream processing capabilities. Instead of fully materializing the data in memory, the execution engine reads and processes the data in chunks.This is a useful feature if you want to convert larger-than-memory files from one format to another format, e.g. CSV to Parquet. But that’s not what we are here for today..

I am going to use two aggregation queries for my benchmark:

The first query counts the number of English reviews per steam account and sorts them from highest to lowest

D select author_steamid, 
COUNT(*) AS num_reviews  
FROM read_csv('all_reviews.csv', ignore_errors = true)  
WHERE language = 'english' 
GROUP BY author_steamid 
ORDER BY num_reviews DESC;
100% ▕████████████████████████████████████████████████████████████▏ 
┌───────────────────┬─────────────┐
│  author_steamid   │ num_reviews │
│       int64       │    int64    │
├───────────────────┼─────────────┤
│ 76561198030784015 │        9674 │
│ 76561198024340430 │        5930 │
│ 76561198067298289 │        5534 │
│ 76561198094803808 │        4341 │
│ 76561198125392509 │        4124 │
│ 76561198027267313 │        4124 │
│ 76561197960373660 │        3350 │
│ 76561197970602587 │        3212 │
│ 76561197961017729 │        2810 │
│ 76561198155150242 │        2554 │
│ 76561198066590240 │        2490 │
│ 76561198045381877 │        2280 │
│ 76561198069159152 │        2216 │
│ 76561197960319772 │        1978 │
│ 76561198062813911 │        1940 │
│ 76561198025731804 │        1928 │
│ 76561198137285867 │        1898 │
│ 76561198043135631 │        1883 │
│ 76561198036629241 │        1875 │
│ 76561198055119582 │        1843 │
│         ·         │           · │
│         ·         │           · │
│         ·         │           · │
│ 76561198111175247 │           1 │
├───────────────────┴─────────────┤
│    15324507 rows (40 shown)     │
└─────────────────────────────────┘
Run Time (s): real 22.556 user 166.382276 sys 9.815767

This query processes the 112M rows in a surprisingly short 22.5s. The most active Steam user has written 9674 game reviews. You can find them here .

I made the second query intentionally more complex. I filter for reviews that contain the word love and I also compute the mean of the votes_up score and the sum of the votes_funny score for each user. I then filter the aggregation using having for only users that have a sum_votes_funny score of more than 100.

D SELECT author_steamid, 
    COUNT(*) AS num_reviews,
    avg(votes_up) as mean_votes_up, 
    sum(votes_funny) as sum_votes_funny,
    FROM read_csv('all_reviews.csv', ignore_errors = true) 
    WHERE language = 'english' and review LIKE '%love%'
    GROUP BY author_steamid 
    having sum_votes_funny > 100 
    ORDER BY num_reviews DESC; 
100% ▕████████████████████████████████████████████████████████████▏ 
┌───────────────────┬─────────────┬────────────────────┬─────────────────┐
│  author_steamid   │ num_reviews │   mean_votes_up    │ sum_votes_funny │
│       int64       │    int64    │       double       │     int128      │
├───────────────────┼─────────────┼────────────────────┼─────────────────┤
│ 76561198155150242 │         777 │   1.09009009009009 │             362 │
│ 76561198042406453 │         665 │ 13.478195488721804 │             341 │
│ 76561198043135631 │         475 │ 2.9410526315789474 │             116 │
│ 76561198149437416 │         352 │ 1.2386363636363635 │             256 │
│ 76561198007343154 │         342 │  4.038011695906433 │             375 │
│ 76561198066590240 │         273 │  9.293040293040294 │             247 │
│ 76561197970314107 │         241 │  38.15767634854772 │             319 │
│ 76561197961017729 │         237 │ 10.278481012658228 │             251 │
│ 76561197981638563 │         231 │ 26.372294372294373 │             509 │
│ 76561197972040704 │         223 │  8.560538116591928 │             123 │
│ 76561197992694498 │         223 │   73.1390134529148 │             759 │
│ 76561197970761123 │         222 │ 22.603603603603602 │             150 │
│ 76561198043609914 │         217 │ 18.792626728110598 │             129 │
│ 76561198053422627 │         202 │  17.04950495049505 │             284 │
│ 76561198007888370 │         196 │  93.96938775510205 │            3345 │
│ 76561198040884867 │         191 │  38.41884816753927 │             509 │
│ 76561198817597644 │         166 │ 22.542168674698797 │             119 │
│ 76561198356141989 │         166 │ 19.246987951807228 │             216 │
│ 76561198031599084 │         165 │ 17.163636363636364 │             171 │
│ 76561198011647032 │         159 │  6.345911949685535 │             192 │
│         ·         │           · │                ·   │              ·  │
│         ·         │           · │                ·   │              ·  │
│         ·         │           · │                ·   │              ·  │
│ 76561198106232693 │           1 │               96.0 │             155 │
├───────────────────┴─────────────┴────────────────────┴─────────────────┤
│ 2383 rows (40 shown)                                         4 columns │
└────────────────────────────────────────────────────────────────────────┘
Run Time (s): real 26.648 user 213.432443 sys 7.281861

With ~26s this is on par with the previous query. User 76561198155150242 used the word “love” in 777 reviews and collected 362 funny upvotes across their reviews.

These timings show how incredibly convenient DuckDB is to analyse large datasets. <30s is still in the realm where I can interactively work with the dataset and explore different queries.You can push DuckDB to the limit by using a blocking operator, e.g. a rank() window function. This query will be slow because the entire input needs to be buffered to compute the result.

When I explore data it is more common that I work in a notebook instead of the CLI. Fortunately, I can run the same queries using the DuckDB Python client . In the next section I am going to compare it against polars, another relatively new kid on the block for out-of-memory analytics.

Working in a Jupyter notebook

DuckDB Python client

I am going to write the code to run the above queries using the duckdb python package and return the results as a pandas dataframe.

import duckdb
import pandas as pd

# Connect to DuckDB
conn = duckdb.connect(':memory:')


# Define the queries
query1 = """
SELECT author_steamid, COUNT(*) AS num_reviews 
FROM read_csv('./steam_reviews/all_reviews/all_reviews.csv', ignore_errors = true) 
WHERE language = 'english' 
GROUP BY author_steamid 
ORDER BY num_reviews DESC 
"""

query2 = """
SELECT author_steamid, 
COUNT(*) AS num_reviews,
avg(votes_up) as mean_votes_up, 
sum(votes_funny) as sum_votes_funny,
FROM read_csv('./steam_reviews/all_reviews/all_reviews.csv', ignore_errors = true) 
WHERE language = 'english' and review LIKE '%love%'
GROUP BY author_steamid 
having sum_votes_funny > 100 
ORDER BY num_reviews DESC 
"""

%%timeit

# Execute the query
result1 = conn.execute(query1)
# Convert the result to a pandas DataFrame
df1 = result1.df()

16.6 s ± 627 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

df1.head(5)

	author_steamid	num_reviews
0	76561198030784015	9674
1	76561198024340430	5930
2	76561198067298289	5534
3	76561198094803808	4341
4	76561198027267313	4124

%%timeit

# Execute the query
result2 = conn.execute(query2)
# Convert the result to a pandas DataFrame
df2 = result2.df()

19.9 s ± 343 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

df2.head(5)

	author_steamid	num_reviews	mean_votes_up	sum_votes_funny
0	76561198155150242	777	1.090090	362.0
1	76561198042406453	665	13.478195	341.0
2	76561198043135631	475	2.941053	116.0
3	76561198149437416	352	1.238636	256.0
4	76561198007343154	342	4.038012	375.0

Not surprisingly this gives the same results as the queries run from the CLI. Having the aggregated results as a pandas dataframe allows me to use it in downstream work. I don’t need to learn another syntax.

## clear memory 
import gc 
del result1, result2, df1, df2;
gc.collect();

Polars

Since polars is often brought up as a faster and better pandas, I want to do a performance comparison. Polars supports lazy dataframes, which allows me to define operations on the dataframe without loading it fully into memory first.

Below are the two equivalent queries written using polars syntax.

import  polars as pl

# Create a lazy DataFrame
df_lazy = pl.scan_csv("./steam_reviews/all_reviews/all_reviews.csv", ignore_errors=True)

%%time

# Define the query using lazy operations
result_polars1 = (
    df_lazy.filter(pl.col("language") == "english")
    .group_by("author_steamid")
    .agg(pl.len().alias("num_reviews"))
    .sort("num_reviews", descending=True)
    
)

# Execute the query and collect the results
df_polars1 = result_polars1.collect()
df_polars1.head(5)

CPU times: user 1min 3s, sys: 22.1 s, total: 1min 26s
Wall time: 2min 7s

shape: (5, 2)

author_steamid	num_reviews
i64	u32
76561198030784015	9822
76561198024340430	5983
76561198067298289	5577
76561198094803808	4408
76561198125392509	4203

%%time

result_polars2 = (
    df_lazy
    .filter(
        (pl.col("language") == "english") & 
        pl.col("review").str.contains("love")
    )
    .group_by("author_steamid")
    .agg([
        pl.len().alias("num_reviews"),
        pl.col("votes_up").mean().alias("mean_votes_up"),
        pl.col("votes_funny").sum().alias("sum_votes_funny")
    ])
    .filter(pl.col("sum_votes_funny") > 100)
    .sort("num_reviews", descending=True)
)


# Execute the query and collect the results
df_polars2 = result_polars2.collect()
df_polars2.head(5)

<timed exec>:9: DeprecationWarning: `pl.count()` is deprecated. Please use `pl.len()` instead.


CPU times: user 1min 21s, sys: 42.7 s, total: 2min 4s
Wall time: 5min 14s

shape: (5, 4)

author_steamid	num_reviews	mean_votes_up	sum_votes_funny
i64	u32	f64	i64
76561198155150242	786	1.094148	369
76561198042406453	673	13.43685	345
76561198043135631	480	2.922917	116
76561198149437416	360	1.216667	256
76561198007343154	354	4.014124	381

Now we can compare DuckDB vs. polars execution times for the two queries:

wall time	DuckDB	polars
query 1	17s	2min 7s
query 2	20s	5min 14s

Conclusion

This investigation shows that DuckDB is a powerful and convenient tool to process larger-than-memory datasets on a single machine. As a data scientist this is useful, because I can focus on exploring the data in the early exploration phase. I don’t yet have to spend time setting up more complex tools or work on a remote machine. The quick comparison with polars also shows its speed advantages and that it can be used as a drop-in when aggregating data in Jupyter notebooks.

TIL about the QUALIFY statement in SQL

Mon, 16 Sep 2024 00:00:00 +0000

Today I came across the QUALIFY clause which is supported in some SQL dialects. It’s not part of the SQL standard but supported by main analytical databases like BigQuery, Snowflake, Oracle, Databricks, DuckDB, etc. It’s part of Snowflake’s dialect which is the data warehouse that I use at work. The QUALIFY statement lets me filter the result of a query based on the result of a window function.

The order of execution is the following:

-- SELECT <columns>
--   FROM <table>
--   [GROUP BY ...]
--   [HAVING ...]
--   QUALIFY <window function predicate>

QUALIFY is for window functions what HAVING is for group-by statements. It allows to filter based on the outputs of these operations without having to use a subquery or CTE. Therefore I can write more concise SQL.

I often deduplicate data using the row_number() window function. Sometimes I just want the latest event for a particular user in a dataset.

An example would be:

-- logs with some user events and timestamps
with logs_annotated as (
    select
        logs.*,
        row_number() over (partition by user_id order by event_timestamp desc) as row_number
    from logs
)
select * from logs_annotated where row_number = 1

With QUALIFY I can write one statement instead:

select * from logs
qualify row_number() over (partition by user_id order by event_timestamp desc) = 1

This works because the QUALIFY statement is evaluated after the window function result has been computed and can therefore be used in filtering.

Interest rate expectations

Sat, 14 Sep 2024 00:00:00 +0000

This morning, the top headline on the front page of my weekly newspaper was:

Bets rise on bumper rate cut by Fed

Investors have sharply increased their bets on half percentage point interest rate cut by the Federal reserve next week as the US central bank prepares to lower borrowing cost for the first time in more than four years. Traders in swaps markets are pricing in a 43 percent chance the Fed will opt for a bumper cut in a bid to prevent high rates damaging the economy. - FT Weekend (14.9.2024)

I often read news about expectations on how the Fed will set interest rates. I have a vague sense that it is related to the price of some interest rate related futures. I just asked myself if I wanted to look up what the market expectation is next week, where would I actually go to look? This detail is regularly glanced over in news articles. It turns out that the maths behind the expectation approximation is fairly simple. Let us work it out.

The Federal Funds Rate

Banks are required to put a certain percentage of their deposits into accounts at a Federal Reserve Bank. This is to maintain liquidity to cover depositor’s withdrawals in the short-term. This reserve requirement is loosely a percentage of the bank’s deposits at the end of the day averaged over a two-week period. If at the end of the day a bank has excess reserve balances it can lend it overnight to another bank that is undercapitalized. Banks do that a lot and negotiate an interest rate for this overnight loan. The weighted average of all these deals for day is the effective federal funds rate (EFFR).

The Fed’s federal funds rate (FFR) target range is set by the Federal Open Market Committee and is the desired range for the EFFR. While the Fed can’t directly impact the EFFR it can influence it via their own deposit interest rates or changes to the monetary supply. This tells us what today’s FFR range and EFFR are (both are published by the Fed). But how do we compute the market’s expectations on changes to the FFR range?

Federal Funds Futures

We can infer the expectations to FFR range changes at the next committee meeting by looking into how FFR future contracts are priced. Those are traded on the Chicago Mercantile Exchange (CME). You can view the monthly future quotes on the CME website .

These futures are used by banks and fixed-income portfolio managers to hedge against short-term interest rate fluctuations. The 30-day futures are monthly contracts that are settled on the last business day of every month. The contract price payable is the arithmetic mean of the daily EFFR during the contract month as reported by the Federal Reserve Bank of New York, subtracted from 100. For example if the average interest rate was 3.5% for a given month, then the contract price would be 100 - 3.5 = $96.5. Unfortunately, the minimum contract size is the price times 4167, which for this example would be $96.5 * 4167 = $416,796.5. A bit too high for my personal hedging needs.

What do the Federal Funds Future(s) hold?

Now that we know about the EFFR and Federal Funds futures, we can collect all ingredients to compute the expectations:

The current FFR range was set in the July meeting to 5.25% - 5.5%.
The dates of the next committee meetings are published here on federalreserve.gov . The remaining meetings this year are scheduled for: 18.9.2024, 7.11.2024, 18.12.2024.
The expected EFFR from the futures contracts for the month of the next committee meeting and the following month. For the remaining year they were on the 14.9.2024:

import pandas as pd

df = pd.DataFrame(
    {
        'MONTH' : ['09-2024', '10-2024', '11-2024', '12-2024'],
        'PRICE' : [94.81, 95.03, 95.315, 95.58],
        'MEETING': [18, None, 7, 18]
     }
).set_index('MONTH')
display(df)

	PRICE	MEETING
MONTH
09-2024	94.810	18.0
10-2024	95.030	NaN
11-2024	95.315	7.0
12-2024	95.580	18.0

The next month without a committee meeting is October. This means that October’s average EFFR can only be impacted by the September committee meeting. This means that the October future contract price of $95.03 is the expected average EFFR for October.

r_october_avg = 100 - df.at['10-2024','PRICE']
r_september_avg = 100 - df.at['09-2024','PRICE']

It is also the expected EFFR on the last day of September (and the first day of November).

r_september_end = r_october_avg

We are interested in the implied difference in EFFR at the beginning and the end of the month. If there is a difference, then this would imply that the FED committee changed the FFR range.

Keep in mind that the contract price is the arithmetic mean of the realized EFFR on every day in September. Assume that the FED committee does lower the price on the 18th of the month, then we will have 18 days at a higher rate and 30-18 = 12 days at a lower rate. All of that is captured in the future price (the average EFFR) for September.

Therefore we can use:

\[ EFFR(\text{average of Sep}) = \frac{N}{30} EFFR(\text{start of Sep}) +\frac{30-N}{30} EFFR(\text{end of Sep}) \]

\[ \Leftrightarrow EFFR(\text{start of Sep}) = \left( EFFR(\text{average of Sep}) - \frac{30-N}{30}\cdot EFFR(\text{end of Sep}))\right) \cdot \frac{30}{N} \]

where $N$ is the number of days before the committee meeting in September.

days_month = 30
days_before_meeting = df.at['09-2024','MEETING']
days_after_meeting = days_month - days_before_meeting
r_september_start = (r_september_avg - (days_after_meeting/days_month * r_september_end) ) * (days_month/days_before_meeting)
print(f'The expected EFFR for the start of September is {r_september_start:.2f}%.')

The expected EFFR for the start of September is 5.34%.

Since today is the 14th of September, we can already look up the realized EFFR up until today. If today was the 25th of August, then we need to calculate the expected $EFFR(\text{start of Sep})$ with the above formula.

r_september_change = r_september_end - r_september_start
print(f'Comparing the expected EFFR at beginning and end of month we have a delta of {r_september_change:.2f}.')

Comparing the expected EFFR at beginning and end of month we have a delta of -0.37.

The market’s expectation for September is that we will have 0.37bps lower EFFR at the end of the month. If we buy the futures contract today we will make money if the FED committee cuts the rate in such a way that the realized EFFR will decrease more than 0.37bps compared to the beginning of the month.

Traditionally, the committee changes the FFR range by 25bps (however 50 and 100 basis points change do occur, e.g. after 9/11, after the 2007-2008 housing market crash and during the COVID-19 pandemic).

However, we can see that the market prices in a 37bps lower EFFR. This means we can calculate the probability of a 25bps cut vs. a 50bps cut.

print(f'The price implies {r_september_change / -0.25:.2f} x 25bps rate cuts.')

The price implies 1.47 x 25bps rate cuts.

probability_of_25_bps_cut = 1 - (r_september_change/(-0.25) % 1)
print(f'This implies a probability of {100 * probability_of_25_bps_cut:.2f}% that the commitee announces a 25bps rate cut.')

This implies a probability of 53.33% that the commitee announces a 25bps rate cut.

probability_of_50_bps_cut =  (r_september_change/(-0.25) % 1)
print(f'This implies a probability of {100 * probability_of_50_bps_cut:.2f}% that the commitee announces a 50bps rate cut.')

This implies a probability of 46.67% that the commitee announces a 50bps rate cut.

The calculation confirms the newspaper article’s claim that the rate decision is priced as a coin flip between a 25bps and a 50bps rate cut.

Update 18.9.2024: The FOMC decided on a 50bps cut to a target range of 4.75% - 5.0%.

What about Europe?

If you are interested in the expectations for the interest rates set by the Bank of England, you can use One Month SONIA Index Futures . Similarly, for the ECB interest rates expectations you can check ECB Dated ESTR Futures .

Jupyter Notebook

You can find the jupyter notebook for this post here .

TIL about just and justfiles

Fri, 13 Sep 2024 00:00:00 +0000

just is just a command runner that I recently started using to define aliases for commands in my projects, e.g. build, test, or deploy commands.

Installation

just can be installed via homebrew:

brew install just

Example

I want to create alias commands for my dbt project to build my dbt models in production vs. development environments. At the moment, I need to run the commands:

# prod warehouse
poetry run dbt run --target prod
# dev warehouse
poetry run dbt run --target dev

To define shorter just commands for this project I add a justfile into the project folder.

# justfile
run-prod *ARGS:
    poetry run dbt run --target prod {{ARGS}}

run-dev *ARGS:
    poetry run dbt run --target dev {{ARGS}}

If I am inside the project folder I can now run just run-prod and just run-dev. Moreover, I can pass in any dbt keyword arguments, e.g.

just run-dev --select mymodel

will resolve to

poetry run dbt run --target dev --select mymodel

Is this not just `make`?

just is inspired by the Unix build tool make from 1978. However, just is not a build system and focuses solely on command running. This avoids having to deal with things like .PHONY that can can catch you out in make recipes. I also like two additional features of just:

I can run my just commands from any subfolder of the project folder that contains the justfile. This is useful when I work on this blog. I have to run hugo serve from the top-level project folder and jupyter-to-markdown converter nb2hugo from the notebooks/project_x/ subfolder. With just command aliases it doesn’t matter from which folder I run the command.

just allows you to load environmental variables from an .env file.

Example .env:

DATABASE=localhost:1111
USERACCOUNT=rob1234

Add set load-dotenv to your justfile and the environment variables will be available in your shell:

set dotenv-load

check-vars:
    @echo "Env vars are DATABASE:$DATABASE and USER:$USERACCOUNT"

Running just check-vars will output:

Env vars are DATABASE:localhost:1111 and USER:rob1234

Clearing up confusion around IPython, ipykernel, and Jupyter notebooks

Mon, 26 Aug 2024 00:00:00 +0000

One of my big recurring time sinks while doing data science work used to be trying to get my colleagues’ jupyter notebooks to run on my machine. The main contributing factors:

My team generally uses poetry environments to improve reproducibility, but sometimes dependencies aren’t specified
I use VS code and the jupyter extension to edit notebooks in VS code which requires more configuration then running the Jupyter web UI.
I lacked a clear understanding of the differences of IPython, ipykernel, jupyter and which python environments are being used when running a notebook.

So this is my attempt at a friendcatcher . I hope this saves you a few minutes the next time you run into similar issues.

The different components

Let’s distinguish the components that play a role in running a jupyter notebook:

Jupyter Notebook platform: A web-based interactive computing platform that supports different languages via different kernels, e.g. for python (ipykernel), Julia (IJulia), R (IRKernel)
Jupyter notebook (extension .ipynb): is a document in json format that holds metadata and cell code.
IPython command shell: The shell has two components:
1. An interactive Python shell. You can start it with ipython. It’s like the default python REPL but with enhanced features, e.g. object introspection, tab completion, input history, magic commands, etc.
2. A jupyter kernel ipykernel. This is the backend process where user python code runs and which can be connected to different frontends. One frontend is indeed the IPython shell, another one a Jupyter notebook. You can install ipykernel as a standalone package into your python environment.
jupyter python package. This is a metapackage which installs the notebook, qtconsole, and ipykernel.

graph TD;

colab(Google Colab UI) <--> ipykernel(ipykernel)
vs(VS Code UI) <--> ipykernel
ui(jupyter notebook UI) <-->  server
server(jupyter server) <-->  ipykernel
ipykernel <-->  ipython[IPython]

Jupyter kernels vs. shell environment

One source of confusion is that jupyter kernel can point to a different python executable than your shell environment.

To get an overview of available jupyter executables you can use:

List all available jupyter executables in your system:
```
$ type -a jupyter
```
List all available jupyter kernels:
```
$ jupyter kernelspec list
```

Every jupyter kernel folder includes a kernel.json file that links to the python executable that is being used. Note this can be different to the python executable referenced by your current shell. Moreover, the shell environment of a Jupyter notebook uses the python executable used to launch the notebook.

Print path of currently used python executable:
```
$ type python
```
or in a notebook cell:
```
!type python
```
Print path of python executable of current kernel: In a notebook cell:
```
import sys
sys.executable
```

You can create new kernels using the ipykernel package:

$ python -m ipykernel install --user --name envname --display-name "Python (envname)"

Dependency management

Since I use VS Code as my frontend I just need to add the ipykernel package into the virtual environment that I use to manage all other dependencies used to run the notebook. This ensures that the same python executable is used for the kernel and the shell environment.This is well explained here .

These are the steps to create a new environment for a jupyter notebook:

Create project folder:
```
$ mkdir notebook_project
```

Create new virtual environment in the folder, then activate it

$ cd notebook_project
$ python3 -m venv .venv
$ source .venv/bin/activate

Install ipykernel (and other dependencies) using pip (make sure the venv is activated):
```
$ python3 -m pip install ipykernel
$ python3 -m pip install pandas
```
Create a new notebook
```
$touch mynotebook.ipynb
```
Open notebook in VSCode and in the top right corner select Select Kernel –> Python Environment –> .venv (.venv/bin/python)
You should now be able to run the notebook and use the pandas package inside the notebook.
To add new dependencies:
- Use the terminal: $ python3 -m pip install <package_name>
- Install from within a notebook cell:
```
import sys
!{sys.executable} -m pip install <package_name>
```
- Specify your dependencies in a requirements.txt

I use poetry for virtual environments and dependency management. So in step 2 I would instead use:

$ cd notebook_project
$ poetry init

and install packages via:

$ poetry add ipykernel
$ poetry add pandas

If I want to use the default Jupyter UI, I can install the jupyter metapackage into my environment and then start the UI with:

$ poetry add jupyter
$ poetry run jupyter notebook

Links

TIL to create continuously numbered lists with content between items

Wed, 21 Aug 2024 00:00:00 +0000

When writing notes in markdown I commonly encounter the problem that I want to create an automatically numbered list with content between the items. However, this messes up the numbering as it creates two separate numbered lists, e.g.

My first point
My second point

code block or image

My third point
My fourth point

The solution to this is embarrassingly obvious once you know it. Just move the image or code block one level down by indenting it with four spaces. This keeps the list intact.

My first point
My second point

code block or image
My third point
My fourth point

TIL how to set a lower threshold on a column in SQL

Wed, 07 Aug 2024 00:00:00 +0000

For the last couple of months I came across multiple instances where I wanted to write an SQL query to lower threshold all the values in a particular column in one of our Snowflake tables.

I used quite clunky workarounds to do that. It turns out you can use the SQL functions GREATEST or LEAST for it.

Example

I am creating a temporary table in the first CTE with a column DATA_COLUMN that has positive and negative numbers.

with example_table as (
SELECT * 
FROM (
    VALUES 
        (1, -100.423), 
        (2, 5.0), 
        (3, -0.001), 
        (4, 100.0), 
        (5, 12.0)
) AS t (row_nr, data_column)
)

select GREATEST(DATA_COLUMN, 0.) as thresholded 
from example_table

Output:

The function’s primary purpose is to take the greatest value among a number of referenced columns, e.g. GREATEST(column1, column2, column3).

Data analysis: Vätternrundan 2024 results

Sat, 03 Aug 2024 00:00:00 +0000

In 2024, I was foolish enough to participate in the Vätternrundan , which is a 315km distance bike sportive around Sweden’s second biggest lake. It took me and my friends a little less than 11.5 hours of cycling (15h including breaks).

Since this event is a sportive , not a race, the organizers only publish finishing times via a bib number search on their website . However, I was curious to understand how well my group did overall. To get my hands on the data I wrote a small scraping script to collect all finish times from the results page.

I can use this data to find out:

information about the participants, e.g. what countries had a lot of starters
distribution of race times, e.g. how fast is the median rider
what percentage of people finished the race

Scraping the dataset

Since getting the timing information from the results page involves several clicks for each start number, I use selenium to automate the browser actions. The script performs the actions for each start number between 1 and 30,000 and collects the timing information. Some start numbers are not used and many people didn’t finish the race. All the data is stored in a parquet file that is used in the next section. I had to do some light data cleaning, e.g. converting scraped strings to timedelta, removing data for unused startnumbers, etc.

Loading the dataset

I load the data from the parquet file into a pandas dataframe. I perform a few additional data cleaning steps, so it is easier to work with the dataset.

import pandas as pd
import numpy as np
import datetime as dt
import matplotlib.pyplot as plt
import seaborn as sns
from matplotlib.ticker import FuncFormatter

df = pd.read_parquet("results_vatternrundan24.parquet")
df['result_time_minutes'] = df['result_time'].dt.total_seconds() / 60
df['average_speed'] = 1/df['result_time_minutes'] * 315 * 60
df['has_finished'] = ~df['result_time'].isna() & (df['timing_consistent'])

print(f'The dataset contains {df.shape[0]} entries.')

The dataset contains 15813 entries.

Analysing the dataset

The dataset has the following columns:

startnumber
initials: initials of the rider
club: club of the rider
city: home city of the rider
country: home country of the rider
result_time: full duration between crossing start and finish lines (includes breaks at food stations)
average_speed
is_sub9: whether the rider was part of a sub9 group that started later in the day (after 11:30 on Saturday)
has_finished: a boolean whether a rider completed the event or dropped out

display(df[['startnumber', 'initials', 'club','city', 'country', 'result_time', 'average_speed', 'has_finished']].head(5))
df.dtypes

	startnumber	initials	club	city	country	result_time	average_speed	has_finished
40	5121	JK	-	Mantorp	SE	0 days 15:11:00	20.746432	True
41	5122	MI	-	Skärblacka	SE	0 days 15:11:00	20.746432	True
42	5123	TP	-	Höör	SE	0 days 14:35:00	21.600000	True
43	5124	KH	Trimgutta	Løvenstad	NO	0 days 14:47:00	21.307779	True
44	5125	HN	-	Jessheim	NO	NaT	NaN	False

startnumber                      int64
initials                        object
club                            object
city                            object
country                         object
result_time            timedelta64[ns]
start_time                      object
station_records                 object
timing_consistent                 bool
is_sub9                           bool
result_time_minutes            float64
average_speed                  float64
has_finished                      bool
dtype: object

The riders

Let’s first look at the top countries, cities, and clubs that are in this dataset. Not surprisingly, Sweden (SE) has the most participants, followed by Germany (DE), Norway (NO), and Finland (FI).

df.groupby('country').agg(num_starters = ('startnumber', 'count'), num_finishers = ('has_finished', 'sum'), perc_finished = ('has_finished', 'mean'), mean_result_time = ('result_time', 'mean'), fastest_finisher = ('result_time', 'min')).sort_values('num_starters', ascending=False).head(10)

	num_starters	num_finishers	perc_finished	mean_result_time	fastest_finisher
country
SE	12035	9904	0.822933	0 days 13:38:44.486094316	0 days 05:15:00
DE	1437	1174	0.816980	0 days 14:30:54.331914893	0 days 07:37:00
NO	846	658	0.777778	0 days 11:15:29.272727272	0 days 07:16:00
FI	402	335	0.833333	0 days 12:56:00.537313432	0 days 08:09:00
DK	346	281	0.812139	0 days 13:24:26.501766784	0 days 07:57:00
GB	269	194	0.721190	0 days 14:24:04.307692307	0 days 02:54:00
NL	87	77	0.885057	0 days 14:25:03.896103896	0 days 08:53:00
	43	36	0.837209	0 days 15:36:35	0 days 09:01:00
US	38	25	0.657895	0 days 13:39:55.384615384	0 days 09:19:00
PL	38	30	0.789474	0 days 12:52:26	0 days 09:20:00

The cities with most participants are all Swedish: Stockholm, Göteborg, Linköping, Uppsala, Malmö.

df.groupby('city').agg(num_starters = ('startnumber', 'count'), num_finishers = ('has_finished', 'sum'), perc_finished = ('has_finished', 'mean'), mean_result_time = ('result_time', 'mean'), fastest_finisher = ('result_time', 'min')).sort_values('num_starters', ascending=False).head(10)

	num_starters	num_finishers	perc_finished	mean_result_time	fastest_finisher
city
Stockholm	851	726	0.853114	0 days 13:37:09.752066115	0 days 07:31:00
Göteborg	613	516	0.841762	0 days 13:30:09.266409266	0 days 07:35:00
Linköping	378	313	0.828042	0 days 13:48:16.815286624	0 days 07:31:00
Uppsala	323	268	0.829721	0 days 13:50:19.029850746	0 days 07:37:00
Malmö	233	191	0.819742	0 days 13:52:36.125654450	0 days 07:31:00
Örebro	230	184	0.800000	0 days 13:34:19.677419354	0 days 08:19:00
Motala	223	186	0.834081	0 days 14:07:49.354838709	0 days 07:35:00
Västerås	186	142	0.763441	0 days 13:25:54.929577464	0 days 07:37:00
Lund	168	138	0.821429	0 days 13:56:18.260869565	0 days 07:16:00
Jönköping	147	124	0.843537	0 days 13:49:49.354838709	0 days 08:55:00

The following clubs had the most riders.

df.groupby('club').agg(city = ('city', 'first'), num_starters = ('startnumber', 'count'), num_finishers = ('has_finished', 'sum'), perc_finished = ('has_finished', 'mean'), mean_result_time = ('result_time', 'mean'), fastest_finisher = ('result_time', 'min')).sort_values('num_starters', ascending=False)[1:10]

	city	num_starters	num_finishers	perc_finished	mean_result_time	fastest_finisher
club
IMOVEFORCANCER	Karlskrona	69	58	0.840580	0 days 09:54:26.896551724	0 days 07:37:00
Fredrikshof	Stockholm	56	48	0.857143	0 days 10:46:42.500000	0 days 09:20:00
Örebrocyklisterna	Kumla	55	42	0.763636	0 days 11:31:47.142857142	0 days 08:04:00
Team Kungälv	Kungälv	43	29	0.674419	0 days 10:37:20	0 days 08:53:00
schulz sportreisen	Dresden	40	36	0.900000	0 days 13:47:45	0 days 10:20:00
KCE - Kävlinge Cykelentusiaster	Kävlinge	38	32	0.842105	0 days 11:50:01.875000	0 days 09:41:00
Fredrikshofs IF Cykelklubb	Enskede	35	30	0.857143	0 days 11:29:52	0 days 09:44:00
Försvarsmakten	Örebro	32	29	0.906250	0 days 13:51:00	0 days 09:21:00
Team Sportia Uppsala	Uppsala	31	29	0.935484	0 days 10:50:47.586206896	0 days 09:24:00

Number of finishers

315km is a long event. Let’s check how many of the participants finished the event:

num_starters = df.shape[0]
num_finishers = df.query('has_finished').shape[0]
print(f"Among the {num_starters} starting riders in the dataset, we found recorded and consistent finishing times for {num_finishers}. \
This is a finishing rate of {100 * num_finishers / num_starters:.2f}%.")

Among the 15813 starting riders in the dataset, we found recorded and consistent finishing times for 12920. This is a finishing rate of 81.70%.

The podium and top 10

fastest_rider = df.query('has_finished').sort_values('result_time').head(1)
print(f'The fastest rider {fastest_rider.initials.values[0]} from {fastest_rider.city.values[0]} only needed {dt.timedelta(microseconds= float(fastest_rider.result_time.values[0])/1000)} to complete the 315km. \
That is a mindblowing average speed of {(fastest_rider.average_speed.values[0]):.1f} km/h.')

The fastest rider JL from Lund only needed 7:16:00 to complete the 315km. That is a mindblowing average speed of 43.3 km/h.

The 10 fastest riders were below 7h 32min.

df.query('has_finished')[['startnumber', 'initials', 'city', 'club', 'country', 'result_time', 'average_speed']].sort_values('result_time', ascending=True).head(10)

	startnumber	initials	city	club	country	result_time	average_speed
16311	27012	JL	Lund	Kjekkas IF	SE	0 days 07:16:00	43.348624
16301	27002	TJ	Slattum	-	NO	0 days 07:16:00	43.348624
16304	27005	EO	Nittedal	-	NO	0 days 07:16:00	43.348624
16308	27009	BHB	Follebu	-	NO	0 days 07:16:00	43.348624
16310	27011	DT	Oslo	-	NO	0 days 07:16:00	43.348624
16316	27017	JK	Lillehammer	-	NO	0 days 07:16:00	43.348624
16314	27015	SD	Oslo - Norway	-	NO	0 days 07:16:00	43.348624
16781	27482	HD	Kalmar	-	SE	0 days 07:31:00	41.906874
16782	27483	TE	Rockneby	-	SE	0 days 07:31:00	41.906874
16783	27484	JL	Ekerö	-	SE	0 days 07:31:00	41.906874

The 10 fastest riders that didn’t ride as part of a sub-9 registered teams were:

df.query('has_finished and not is_sub9')[['startnumber', 'initials', 'city', 'club', 'country', 'result_time', 'average_speed']].sort_values('result_time', ascending=True).head(10)

	startnumber	initials	city	club	country	result_time	average_speed
8860	16661	MN	Haslum	Team Tøff i Tryne	NO	0 days 08:02:00	39.211618
8850	16651	HL	Oslo	Team Tøff i Trynet	NO	0 days 08:02:00	39.211618
28298	18529	MS	Otalampi	Team Wassu	FI	0 days 08:12:00	38.414634
28297	18528	JK	Kotka	Team Wassu	FI	0 days 08:12:00	38.414634
28295	18526	JP	Helsinki	Team Wassu	FI	0 days 08:12:00	38.414634
8849	16650	SG	Eiksmarka	Team Tøff i Trynet	NO	0 days 08:17:00	38.028169
28296	18527	JS	Hamina	Team Wassu	FI	0 days 08:21:00	37.724551
8861	16662	FA	Oslo	Team Tøff i Trynet	NO	0 days 08:23:00	37.574553
11817	23218	WG	Lemgo	RC Sprintax Bielefeld	DE	0 days 08:24:00	37.500000
11819	23220	CB	Bielefeld	RC Sprintax Bielefeld	DE	0 days 08:24:00	37.500000

Lanterne rouge

The slowest finisher took almost 28h.

df.query('has_finished')[['startnumber', 'initials', 'city', 'country', 'result_time', 'average_speed']].sort_values('result_time', ascending=True).tail(5)

	startnumber	initials	city	country	result_time	average_speed
3416	2367	AH	Danderyd	SE	1 days 02:37:00	11.834690
1464	415	MP	Östersund	SE	1 days 02:49:00	11.746426
1465	416	ZYP	Stockholm	SE	1 days 02:49:00	11.746426
1260	211	HB	Svedala	SE	1 days 03:36:00	11.413043
1109	60	KH	Skara	SE	1 days 03:44:00	11.358173

Finishing times (excluding sub-9 groups)

To get a better sense of the finishing times, I plot the histogram and get some summary statistics of the result_time column for every finisher. I only consider the times of the regular riders below. This means excluding the sub-9 cycling teams that start separately later on Saturday. With a little less than 15h total (11.5h moving time) my group was slower than the median rider. Next time, we might want to take shorter breaks.

times = df.query('has_finished and not is_sub9')
times['result_time'].describe()

count                        12175
mean     0 days 13:55:05.829979466
std      0 days 02:57:47.200342638
min                0 days 08:02:00
25%                0 days 11:37:00
50%                0 days 13:44:00
75%                0 days 15:48:00
max                1 days 03:44:00
Name: result_time, dtype: object

print(f"To be among the top 5% fastest finishers, you need to beat {times['result_time'].quantile(q=0.05)}.")
print(f"To be among the top 10% fastest finishers, you need to beat {times['result_time'].quantile(q=0.1)}.")
print(f"To be among the top 25% fastest finishers, you need to beat {times['result_time'].quantile(q=0.25)}.")

To be among the top 5% fastest finishers, you need to beat 0 days 09:44:00.
To be among the top 10% fastest finishers, you need to beat 0 days 10:10:00.
To be among the top 25% fastest finishers, you need to beat 0 days 11:37:00.

To plot the distribution of result times, I am binning the result time in minutes and also add some vertical lines indicating the 0.1- and 0.25- percentiles. It’s not surprising that the distribution is right-skewed with many riders that have very long finishing times, but nobody faster than 8 hours.


def minutes_to_hours_minutes(x, pos):
    """Define a function to convert minutes to HH:MM. Used for the axis labelling."""
    hours = int(x // 60)
    minutes = int(x % 60)
    return f"{hours:02d}:{minutes:02d}"

# create histogram plot of result time distribution
plt.figure(figsize=(10, 6)) 
sns.histplot(data=times, x="result_time_minutes", binwidth=15)

plt.gca().xaxis.set_major_formatter(FuncFormatter(minutes_to_hours_minutes))

# draw colored vertical percentile lines
plt.axvline(times.result_time_minutes.median(), color='red', linestyle='--', linewidth=1.5)
plt.axvline(times.result_time_minutes.quantile(0.25), color='green', linestyle='--', linewidth=1.5)
plt.axvline(times.result_time_minutes.quantile(0.1), color='orange', linestyle='--', linewidth=1.5)
plt.text(times.result_time_minutes.median(), plt.ylim()[1] * 0.95, f'Median',
         color='red', ha='left', va='bottom')
plt.text(times.result_time_minutes.quantile(0.25), plt.ylim()[1] * 0.95, f'Top 25%',
         color='green', ha='center', va='bottom')
plt.text(times.result_time_minutes.quantile(0.1), plt.ylim()[1] * 0.95, f'Top 10%',
         color='orange', ha='right', va='bottom');
plt.xlabel('Finish time (incl. breaks)');

If you are planning to race next year’s Vatternrundan, this distribution might give you an indication of what to expect and a time to aim for.

Jupyter Notebook

You can find the jupyter notebook and the datasets for this post here .

How to better remember books

Sun, 26 May 2024 00:00:00 +0000

In this article I am going to describe the system that I use to retain more information from the non-fiction books that I read. There seem to be two schools of thought about how to best retain the content of non-fiction books. The first approach is best described by the quote:

“I cannot remember the books I’ve read any more than the meals I have eaten; even so, they have made me.”

- Ralph Waldo Emerson

By reading a lot, our brain retains what it finds interesting. Our prior beliefs are updated and specific ideas are enforced within us when we encounter the same or related concepts in different sources.

Some argue that proper retention can only be achieved by actively working through the material while reading it. This second approach commonly involves taking notes of the major ideas of the book.

While I do experience surprise connection moments occasionally, there are two reasons why I invest time into taking good notes:

The first approach has an inherent survivorship bias. I do remember concepts from books and I am able to link them to other things. This always feels great. However, I don’t know about the missed opportunities of ideas that I have forgotten and that will never resurface.
Reading a non-fiction book is quite a significant time investment. For me it takes between a few days and several weeks. Taking at least basic notes seems to be a marginal additional cost for the certainty of improved retention and better ability to digest the book in the future.

I use a simple multi-pass approach that involves both analog and digital notes. I mostly read physical books, because I like to browse books on my shelf that I have read in the past. However, you can easily adapt this approach to ebooks.

Skim the table of contents, and try to understand the structure of the book.
Read the book and use a pencil to mark interesting sentences or paragraphs with a line in the margin (Fig. 1). In rare cases I use double lines to highlight especially noteworthy ideas. This helps later to quickly re-familiarize with the main concepts of a chapter without having to fully re-read it. I don’t typically write a lot of notes into the margin because there is usually not a lot of space available.

Figure 1. I use a pencil to mark relevant sentences in the margin and occasionally add some notes.

After finishing one or several chapters, I review the marked paragraphs and summarize the main points of each chapter on the empty space at the beginning or end of the chapter (Fig. 2). By writing the notes into the book I get two benefits. Firstly, there is a higher chance for these analog notes to still be accessible to me in a decade. Secondly, these notes might be interesting to friends that I lend the book to.

Figure 2. I summarize the main points of each chapter on the blank space at the beginning or end of the chapter.

Once I finished the book, I transfer the chapter notes into a markdown file (in Joplin ). While copying the notes I sometimes rework or modify them. I might also link to other digital notes that have a connection to the book.
Optional: If I want to share my notes with an audience I will write about the major learnings from the notes and bring them into context with concepts from other books/articles/experiences.
Optional: Most books will have some learnings that I want to periodically and actively recall. For those I create atomic Anki cards which will prompt me to regularly review the concepts. I am trying to write good prompts for these cards. This can be a mix of facts like “Where was Napoleon born?” or higher level concepts I don’t want to forget, e.g. “Give an example for an infinite game as defined by J. P. Carse”.

The most important rule for this note taking system is that I arbitrarily break the rules for certain books and skip some of these steps when I feel they are not worth the time. However, this system ensures that I do multiple passes of the content, that I have both durable analog and searchable digital notes, and that I am periodically prompted for the most interesting concepts.

DuckDB use cases for data scientists: Querying remote S3 files

Sun, 25 Feb 2024 00:00:00 +0000

DuckDB is a pretty cool in-process OLAP analytical database that I started to spin up on the fly for quick data analysis. What SQLite is to Postgres, DuckDB is to Snowflake. It is a single executable without dependencies and stores databases in local files.

I can think of four use cases for data science work:

DuckDB supports larger-than-memory workloads by loading data sequentially. You can use it to analysis datasets that are too large for Pandas (and too small to justify PySpark).
I can query CSV, parquet, and JSON files directly from remote endpoints, e.g. S3, using SQL.
I can replace Snowflake queries with DuckDB queries in unit / integration tests.
I can set up a (DuckDB + dbt) data warehouse for local development.

I want to share here my workflow for the second use case. Inspecting parquet files in AWS S3 is a pain because I can’t easily inspect them in the AWS console. Since a few months I use DuckDB to load, inspect, and analyse parquet files from the command line. I found this reduced my cognitive load in situations where I quickly want to check a remote file, because I don’t have to download the parquet file and write a python script to inspect it.

Installing DuckDB

I use macOS and can install DuckDB via homebrew: brew install duckdb. To work with remote files I also install the httpfs extension.

I start the DuckDB shell with duckdb and run the SQL commands:

INSTALL httpfs;
LOAD httpfs;

Authentication with AWS

To load files from S3, I need to configure AWS credentials. I am assuming here that your workplace has configured AWS SSO temporary credentials, but this work also with static credentials (ACCESS_KEY_ID, SECRET_ACCESS_KEY). There are two ways of doing this:

1. Set AWS credentials in the session:

I can set AWS credentials inside a duck db session like this:

LOAD httpfs;
SET s3_region = 'eu-west-2';
SET s3_access_key_id ='???';
SET s3_secret_access_key ='???';

2. Let aws vault handle credentials

I use the AWS credentials management tool aws vault (see here ) which manages my temporary AWS credentials in the background and exposes them to my shell sub-process. I create the duckdb shell with aws-vault exec [profile-name] | duckdb which ensures that AWS credentials are automatically set and updated.

Querying parquet files

Let’s assume that I have multiple parquet files stored in an S3 bucket work-project and they all have the same schema:

s3://work-project/clickdata_001.parquet
s3://work-project/clickdata_002.parquet
[...]
s3://work-project/clickdata_100.parquet

These files might be placed in this S3 bucket as part of an ETL pipeline before they are loaded into a table in a data warehouse. Let’s imagine I want to investigate a bug in the pipeline and need to inspect the files.

Inspect schema of parquet file

To familiarize myself with the schema I run parquet_schema

LOAD httpfs; -- once at the beginning of the session
SELECT * FROM parquet_schema('s3://work-project/clickdata_001.parquet');

which will return:

clicked_at::timestamp
user_id::int
event_type::string

Query a parquet file

I can use postgres SQL dialect to query the parquet files with the read_parquet function, e.g.

--- count number of events in the file
SELECT count(*) FROM read_parquet('s3://work-project/clickdata_001.parquet');

--- find the last event time of a particular user
SELECT max(clicked_at) FROM read_parquet('s3://work-project/clickdata_001.parquet') where user_id = 1234;

Query multiple parquet files

Say I want to find the users with the highest number of events across all files. I can use glob syntax with * to run a query against all files as if they were one table:

SELECT user_id, count(*) as num_events FROM read_parquet('s3://work-project/*.parquet') group by user_id order by num_events desc;

Create table from files

Let’s assume that all clickdata files together have a size of 500MB. It would be annoying if I had to download these files for every query that I want to run. Let’s instead create a table from the files.

create table clickdata as select * from read_parquet('s3://work-project/*.parquet');

This table is stored in memory until we close the shell, which allows me to run different queries against it until I am done with my analysis, e.g.

select * from clickdata;

I can also store the table in a local database on my computer in case I need to work with the data for a longer time. The following SQL statement

EXPORT DATABASE 'mydatabase' (FORMAT PARQUET);

will create a local folder mydatabase which stores the tables currently in memory. In a later duckdb session I can reload the table from the database using the following command

IMPORT DATABASE 'mydatabase';

Query column statistics

A nice feature of the parquet file format is that it stores statistics about each column , e.g. min, max, null_count, distinct_count. If you are interested, you can inspect parquet file metadata using select * from parquet_metadata('file.parquet'); Assume the click event dataset is several GB large and I want to identify the earliest clicked_at time. I would run the query below.

SELECT min(clicked_at) FROM read_parquet('s3://work-project/*.parquet');

Fortunately, DuckDB uses the column statistics stored in the parquet files to compute the answer without having to download the whole dataset from S3. Simon Willison shows a more extreme example of this feature.

DuckDB is a great tool for quick investigations of remotely hosted files. Especially after having configured automatic handling of AWS credentials I can spin up a duckdb shell with one command. I hope it becomes a time saver for you too.

A great weekend

Sun, 25 Feb 2024 00:00:00 +0000

After a stressful work week I sometimes fall into the trap of trying to maximize how I use my days off. This makes me restless. I sometimes need to remind myself that the weekend is not about squeezing as many productively used hours into the day, but rather about just living, relaxing, and embracing a little bit of boredom and randomness in your life.

So here is a list of activities that make a weekend great:

walking in nature, e.g. hiking on a local trail or going to the nearest park or forest
exploring a new neighborhood in your city without a particular goal in mind
cooking a meal for your partner or friends
completing a household chore at the beginning of your day
meeting a friend for a coffee/museum walk
small talk with someone in your local community, e.g. one of your neighbors
messaging or calling a friend who moved far away
taking time for an activity (running, cycling, swimming, gym, local club meetup)
playing a board game
learning a new board game (e.g. Hive, Robinson Crusoe, Wingspan)
watching a documentary with full attention
fixing something broken at home, e.g. your bicycle
reading a fiction or history book
reading the weekly edition of a newspaper (try a section of the paper you would normally skip)
reading or deleting interesting blog posts that you have saved on your phone for later
listening to a music genre that you would not normally listen to
following a yoga / stretching / meditation routine on youtube
learning and taking notes about a topic that is not relevant for your work or career
practicing a different language
talking to your parents
volunteering at a local event

How to display jupyter notebooks on your hugo blog

Sat, 20 Jan 2024 00:00:00 +0000

I like the simplicity and ease of use of Hugo, the static site generator that powers this blog. However, as a data scientist I want to be able to make an argument using code, data, and graphs in a Jupyter notebook. This post explains my setup how to convert a Jupyter notebook into a blog post for this website, such as this . Before I started, my requirements were:

I wanted to be able to easily share both the original notebook (and accompanying data) as well as the website version.
I wanted to regenerate the website version from the notebook file with at most one command and without any post-processing. This is important because I usually work iteratively on notebooks.

Writing the notebook

The source code for this blog is available in this Gitlab repository . I created a new top-level folder notebooks/ and a subfolder for each notebook project I want to post about. This allows me to easily link to the Jupyter notebook and data to make it reproducible.

It is important that you start the notebook with a cell that contains the front matter of the blog post, e.g.

# My notebook title

Date: 2018-06-01  
Author: firstname lastname  
Categories: category1, category2  
Tags: tag1, tag2, tag3  
<!--eofm-->

The  is important to divide the front matter from the rest of the notebook.

Converting to markdown

To align with the markdown-based workflow of Hugo we need to convert the notebook to markdown. The script nb2hugo does exactly that. We can install it for example via: pip install nb2hugo.

To convert the notebook to markdown we navigate into the notebook folder and run

nb2hugo covid_bipartisan_bayesian.ipynb --site-dir /Users/rob/hugoblog/ --section posts

This command writes the output markdown file covid_bipartisan_bayesian.md into the specified folder posts/.

Adjusting the website CSS

I made two more adjustment to the style of this blog to better display the “markdown-ified” notebook. Both pandas code and displayed dataframes are quite wide so I:

increased the content width slightly to 800px,
decreased the font-size for the text in the code cells to 0.75rem.

While still not perfect, I think this gives a decent result.