Posts on staticnotes.org

UK house price changes in real terms

Sun, 11 Jan 2026 00:00:00 +0000

This post is a response to a newspaper article I read this morning. Every month the FT is reporting on how the housing market in the UK is performing as people love to talk about it:

UK house prices rise less than expected in 2025 as growth slows

UK house prices rose by 0.6 per cent in 2025 after a slowdown at the end of the year, according to lender Nationwide. […] Prices fell 0.4 per cent between November and December to an average of £271,068. Both figures were below analysts’ expectations of a 1.2 per cent annual rise and a 0.1 per cent month-on-month expansion. FT Weekend (02.01.2026)

Building a local data warehouse with DuckDB, dbt, and Superset

Sun, 09 Mar 2025 15:00:00 +0000

In previous blog posts, I described two DuckDB use cases for data scientists and data engineers: Querying remote parquet files and processing larger-than memory datasets .

Now I want to explore if I can set up a local and open-source version of the analytics data stack that we use at my company. At work, we are using Snowflake, dbt cloud, and Google’s Looker which cost us several thousand EUR per month. I am going to use the following open-source tools in my local setup:

Book notes: Poor Charlie’s Almanack

Fri, 21 Feb 2025 00:00:00 +0000

The book is a collection of speeches that Charlie Munger, partner of Warren Buffett at Berkshire Hathaway, gave over the years (1986 - 2007) at universities and institutions.

I found four interesting themes across the speeches:

Advocating interdisciplinary collaboration between university departments

He believes that the social sciences, especially economics and psychology, are focusing on too narrow theoretical problems. Academics suffer from a man-with-a-hammer-syndrome. He suggests that they collaborate more across departments, e.g. economics borrowing from psychology, and within departments, e.g. macroeconomists shouldn’t avoid microeconomic explanations. Moreover, the social science should try to incorporate findings / models from the hard sciences (physics, chemistry, etc.) and attribute properly.

How related posts are computed

Sun, 09 Feb 2025 00:00:00 +0000

The number next to related posts at the bottom of each page is the advertised post’s “similarity” to the currently viewed page (from 1.0 to -1.0).

I am using the following process to compute related posts locally:

Summarize every post and TIL using local Llama 3.1 (llama3.1:8b-instruct-q5_0 via Ollama) with the following prompt:

Prompt

You are an analyst and editor with many years of experience in reading and synthesizing content.

Book notes: How Big Things Get Done by Bent Flyvbjerg and Dan Gardner

Sun, 02 Feb 2025 00:00:00 +0000

Bent Flyvbjerg is an economics professor at IT University of Copenhagen. He maintains a database of megaprojects (power plants, opera halls, tunnels, airports) and their planned and realized timelines and budgets. He researches the reasons why modern megaprojects often fail to deliver on time and on budget. It has recently become quite popular to discuss the apparent decrease in speed with which large-scale projects are realized. Examples are easy to find, e.g. the delayed High Speed Rail 2 (HS2) project or the time and cost overruns of the Berlin Brandenburg Airport (BER) . Counter-examples are less common (or less newsworthy), e.g. the 2023 Notre-Dame Reconstruction.

LLM prompt heuristics that definitely maybe work

Fri, 11 Oct 2024 14:56:00 +0000

Effective prompt writing for large language models continues to be a dark art. Having read the prompt engineering blogs from Meta, Anthropic, and OpenAI and watched some of Anthropic’s prompt discussions online, it does feel more like design than engineering. Even employees from the same lab don’t agree what tricks actually work. If you have time watch this video of one of Anthropic’s prompt engineers running a prompting masterclass. While giving good heuristics to the audience he is careful not to make any definite statements.

Processing 112M rows of steam reviews locally with DuckDB

Sat, 28 Sep 2024 13:43:40 +0000

In DuckDB use cases for data scientists: Querying remote S3 files I wrote how I use DuckDB as a convenient way to query data from CSV or Parquet files in S3. Another use case for data scientists and data engineers is DuckDB’s ability to processing larger-than-memory data on your local machine. For us data scientists this fills the gap between

small data that you can transform with pandas
big data that typically requires a multi-node processing engine like PySpark.

In this post I am going to run some exploratory queries against this kaggle Steam review dataset . It contains 112M rows of Steam game reviews and comes as an uncompressed 46GB CSV file (17GB compressed). Let’s find out how my MacBook copes with it. Later I am going to run a small comparison against polars , a DataFrame library, which is often mentioned for out-of-memory processing.I am not comparing against pandas because its inability to work with large datasets was one of the reasons why I explored this topic. However, you could do this with chunking .

Interest rate expectations

Sat, 14 Sep 2024 00:00:00 +0000

This morning, the top headline on the front page of my weekly newspaper was:

Bets rise on bumper rate cut by Fed

Investors have sharply increased their bets on half percentage point interest rate cut by the Federal reserve next week as the US central bank prepares to lower borrowing cost for the first time in more than four years. Traders in swaps markets are pricing in a 43 percent chance the Fed will opt for a bumper cut in a bid to prevent high rates damaging the economy. - FT Weekend (14.9.2024)

Clearing up confusion around IPython, ipykernel, and Jupyter notebooks

Mon, 26 Aug 2024 00:00:00 +0000

One of my big recurring time sinks while doing data science work used to be trying to get my colleagues’ jupyter notebooks to run on my machine. The main contributing factors:

My team generally uses poetry environments to improve reproducibility, but sometimes dependencies aren’t specified
I use VS code and the jupyter extension to edit notebooks in VS code which requires more configuration then running the Jupyter web UI.
I lacked a clear understanding of the differences of IPython, ipykernel, jupyter and which python environments are being used when running a notebook.

So this is my attempt at a friendcatcher . I hope this saves you a few minutes the next time you run into similar issues.

Data analysis: Vätternrundan 2024 results

Sat, 03 Aug 2024 00:00:00 +0000

In 2024, I was foolish enough to participate in the Vätternrundan , which is a 315km distance bike sportive around Sweden’s second biggest lake. It took me and my friends a little less than 11.5 hours of cycling (15h including breaks).

Since this event is a sportive , not a race, the organizers only publish finishing times via a bib number search on their website . However, I was curious to understand how well my group did overall. To get my hands on the data I wrote a small scraping script to collect all finish times from the results page.

How to better remember books

Sun, 26 May 2024 00:00:00 +0000

In this article I am going to describe the system that I use to retain more information from the non-fiction books that I read. There seem to be two schools of thought about how to best retain the content of non-fiction books. The first approach is best described by the quote:

“I cannot remember the books I’ve read any more than the meals I have eaten; even so, they have made me.”

A great weekend

Sun, 25 Feb 2024 00:00:00 +0000

After a stressful work week I sometimes fall into the trap of trying to maximize how I use my days off. This makes me restless. I sometimes need to remind myself that the weekend is not about squeezing as many productively used hours into the day, but rather about just living, relaxing, and embracing a little bit of boredom and randomness in your life.

So here is a list of activities that make a weekend great:

DuckDB use cases for data scientists: Querying remote S3 files

Sun, 25 Feb 2024 00:00:00 +0000

DuckDB is a pretty cool in-process OLAP analytical database that I started to spin up on the fly for quick data analysis. What SQLite is to Postgres, DuckDB is to Snowflake. It is a single executable without dependencies and stores databases in local files.

I can think of four use cases for data science work:

DuckDB supports larger-than-memory workloads by loading data sequentially. You can use it to analysis datasets that are too large for Pandas (and too small to justify PySpark).
I can query CSV, parquet, and JSON files directly from remote endpoints, e.g. S3, using SQL.
I can replace Snowflake queries with DuckDB queries in unit / integration tests.
I can set up a (DuckDB + dbt) data warehouse for local development.

I want to share here my workflow for the second use case. Inspecting parquet files in AWS S3 is a pain because I can’t easily inspect them in the AWS console. Since a few months I use DuckDB to load, inspect, and analyse parquet files from the command line. I found this reduced my cognitive load in situations where I quickly want to check a remote file, because I don’t have to download the parquet file and write a python script to inspect it.

How to display jupyter notebooks on your hugo blog

Sat, 20 Jan 2024 00:00:00 +0000

I like the simplicity and ease of use of Hugo, the static site generator that powers this blog. However, as a data scientist I want to be able to make an argument using code, data, and graphs in a Jupyter notebook. This post explains my setup how to convert a Jupyter notebook into a blog post for this website, such as this . Before I started, my requirements were:

I wanted to be able to easily share both the original notebook (and accompanying data) as well as the website version.
I wanted to regenerate the website version from the notebook file with at most one command and without any post-processing. This is important because I usually work iteratively on notebooks.

Writing the notebook

The source code for this blog is available in this Gitlab repository . I created a new top-level folder notebooks/ and a subfolder for each notebook project I want to post about. This allows me to easily link to the Jupyter notebook and data to make it reproducible.

Reproducing Nate Silver's regression analysis on COVID death rates

Mon, 20 Nov 2023 00:00:00 +0000

I spent some time recently studying causal inference methods. Two great resources for this are:

Causal Inference for The Brave and True (frequentist view)
Statistical Rethinking (Bayesian view).

When I learn about an unfamiliar topic, I will always try to look for real-world examples. In this case I keep my eyes open about articles sharing statistical findings, where I can attempt to reproduce the results.

On Writing Well - 80/20 Checklist

Mon, 14 Aug 2023 00:00:00 +0000

This is a checklist that I use at work to quickly sense check my drafts of strategy documents, tech proposals, PR reviews, project feedback, and other places where effective writing is helpful. It’s listing suggestions from the book On Writing Well that I covered in this blog post . As I mentioned in the blog post I am not striving for perfection. Instead I want to be able to identify the main parts of my first draft that I can improve within 5–20 min of editing time.

Writing well as a skill for working well remotely

Sat, 12 Aug 2023 00:00:00 +0000

Writing a good blog post that summarizes William Zinsser’s classic book On Writing Well is a stressful activity. You don’t want to break his rules while writing about them. I will attempt this anyway since I learned a lot from the book that I can use while working with my team.

Book notes: How to Win Friends and Influence People by Dale Carnegie

Mon, 29 May 2023 00:00:00 +0000

I remember listening to the audiobook version of Dale Carnegie’s “How to win Friends and Influence People” in 2015 and finding his advice intuitive and applicable to real-life situations. However, as time passed I seem to have largely forgotten about the content. Coincidentally in the past months, this book has been mentioned to me again by friends and by guests of podcasts that I listened to. So I decided to give it another read, refresh my memory, and summarize the main points.

Book notes: Four Thousand Weeks by Oliver Burkeman

Tue, 10 Jan 2023 00:00:00 +0000

I read this book in December 2022 at which point I have lived ~1700/4000 weeks. Here are my summary notes for every chapter as well as some overall takeaways and questions for the author.

Introduction: In the long run, we’re all dead

Most people live only 4000 - 4700 weeks. This seems short or limited during one’s own lifetime. It is almost negligible at the cosmic timescale.
This means that proper time management should be on our mind and one of our major concerns.
Modern time management advice focuses on how to get more done in the same time. The author thinks this is a flawed approach as he describes time as an unstoppable conveyor belt (Edward T. Hall).
- It brings new tasks as quickly as we can dispatch old ones
- Becoming more productive just causes the belt to speed up
- New work appears magically as we tick things of the to-do list. Someone who appears to answer quick to email requests at work, will get a reputation and receive more requests.

Part 1: Choosing to Choose

1. The limit embracing life

This chapter shows us that our time and what we can do with it is very limited. Instead of pretending that we can have everything we want (in the future) through ever more work, we should embrace the limitedness and dedicate more time thinking about what matters to us and how to spend our time wisely and in the moment. We should not live for a future version of our life that might not materialize because we will always shift the goal post further.