staticnotes.org

TIL how to set up multiple SSH keys (for multiple Github accounts)

Fri, 28 Nov 2025 16:20:00 +0000

Setting up SSH keys is one of these things I need to do every couple of months. Inevitably, I don’t remember how to do it and then follow some tutorial, e.g. the Github Docs . It kind of always works but I haven’t fully internalized how to properly manage those different SSH keys.

Disclaimer: This learning was provoked by setting up multiple SSH keys for different Github accounts (I know how to do that) and then being able to git clone {private-repository} a private repository with the correct ssh key (I didn’t know how to do that). An example use case could be a personal and professional Github account that don’t have access to the same private repositories. So ideally when we git clone we want to automatically pick the right ssh key.

1. Generate the SSH keys for each account

Let’s assume we want to generate new separate ssh keys for each account. I like to have them all in my ~/.ssh folder. We can use the key generation command

$ cd ~/.ssh
$ ssh-keygen -t ed25519 -C "your_email or your_name"

where we specify the encryption type (-t) as ed25519 which is Edwards-curve Digital Signature Algorithm and generally recommended over RSA-based keys . The comment (-C) argument allows us to optionally add a comment that is added at the back of the key so that you remember who it created. People often use their email address. However, this is not used anywhere to select the key.

$ Generating public/private ed25519 key pair.
$ Enter file in which to save the key (/Users/youruser/.ssh/id_ed25519): /Users/youruser/.ssh/github_personal

$ Enter passphrase for "github_personal" (empty for no passphrase):

This generates both the private and public key github_personal (private) and github_personal.pub (public) in the specified path. If we entered a passphrase the private key will be encrypted as well using 3DES . We can check out what the keys look like:

$ cat github_personal.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDycaSFXnekEe32AZFhhDp3Q14JRXF7bu8CNfGf7Z6hN your_email

and

$ cat github_personal

-----BEGIN OPENSSH PRIVATE KEY-----
somegibberylongkeyperhapsfurtherencryptedusingpassphrase
-----END OPENSSH PRIVATE KEY-----

Let’s also use the same method to create a keypair for work (github_work, github_work.pub). Once we have these keys, we can add them to our two Github accounts.

2. Associate the public keys with the Github accounts

In both Github accounts, navigate to Access:SSH and GPG Keys -> New SSH key and add the public key as an Authentication Key to the account. Github now has the ability to verify messages that were signed by the corresponding private key of the pair.

3. Set up OpenSSH client config file

The file ~/.ssh/config allows us to configure “Host” specifications that we can refer to when using SSH (e.g. when git clone uses the SSH protocol for data transfer). The file looks like this:

Host hostname1
    SSH_OPTION value
    SSH_OPTION value

Host hostname2
    SSH_OPTION value

In our case, we define the two different github identities that we want to be able to select:

Host github-personal
    User git
    PreferredAuthentications publickey
    Hostname github.com
    IdentityFile ~/.ssh/github_personal

Host github-work
    User git
    PreferredAuthentications publickey
    Hostname github.com
    IdentityFile ~/.ssh/github_work

This defines which keys to use when we connect using SSH via user@host. So when we the test the config by running ssh -T git@github-personal SSH will use the SSH key ~/.ssh/github_personal as specified in the file. (Note: We can name the host and user whatever we want.)

4. Use the right hostname when cloning repositories

Let’s say we want to git clone a repository that only our work account has access to, then I can use the git clone command git clone {user}/{host}:{repo_path}.git:

$ git clone git@github-work:/org1/project1.git

or more explicitely:

$ git clone ssh://git@github-work/org1/project.git

git/ssh will look into the config file and select the configured private key for the provided host.

GitHub will send a challenge (message) to our machine for authentication. The private key will be used to sign this challenge and our machine then sends the signed message back to the Github server. The Github server will use the stored public key (from step 2) to verify the signature and allow a connection.

Note: If in step 1) we decided to encrypt the private key with a passphrase we will be prompted to provide it. Normally, at that point MacOS will offer us to add this passphrase to our Keychain .

TIL how to re-run only failed tests

Fri, 21 Nov 2025 13:00:00 +0000

This is a super small thing I stumbled over. When I implement a new feature, there is inevitably a step where I go through the failing pytest tests and fix or clarify them.

Unless I use an IDE, my work loop is like this:

Make code change
Run python -m pytest
Check which tests are failing.
Go back to 1.

Now, 2. runs all tests which can take quite some time. There is always the option to run a specific test only, but multiple failing tests can be scattered across different files.

So instead, what you can do is to only run the tests that failed at your last test run. Simply use:

python -m pytest --last-failed

Once you have fixed those tests, it’s worth running all tests one more time to check that nothing else was broken by the new changes.

Another option is to use the test view of your IDE, if you have access to one in that environment.

TIL how to investigate missing data easily with DuckDB

Fri, 16 May 2025 14:00:00 +0000

In data engineering pipelines we commonly store output data in S3 buckets at various stages. In this particular case we stored the input data received from customers as csv files in the S3 bucket s3://customer-ingested/customerA.csv and after some transformations we stored the transformed data as csv files in another S3 bucket s3://customer-transformed/customerA.csv.

Problem

For some of the customers the csv file was missing in the first bucket customer-ingested while we were sure that all customers had a file in customer-transformed. However, there was a legit reason why the file could be missing which was that in the application the customer had a state of is_deactivated = True.

The goal of this investigation was to find all the active customers for which we were missing data in the bucket customer-ingested.

Solution

The nice thing about DuckDB in this scenario is that I can combine filesystem data with information from an application table and work in (familiar) SQL land.

Here is the query:

with transformed as (
	select regexp_extract(file, '/([^/]+)\.csv$', 1) AS filenames_transformed from glob('s3://customer-transformed/*.csv')
	),
ingested as (
	select regexp_extract(file, '/([^/]+)\.csv$', 1) AS filenames_ingested from glob('s3://customer-ingested/*.csv')
	),
customer_status as (
	select customer_name, is_deactivated from read_parquet('s3://application_table_snapshot/customers.parquet')
)

select * from transformed
left join ingested on transformed.filenames_transformed = ingested.filenames_ingested 
left join customer_status as cs on transformed.filenames_transformed = cs.customer_name
where is_deactivated = False and filenames_ingested is null;

┌───────────────────────┬────────────────────┬────────────────────┬────────────────┐
│ filenames_transformed │ filenames_ingested │   customer_name    │ is_deactivated │
│        varchar        │      varchar       │      varchar       │    varchar     │
├───────────────────────┼────────────────────┼────────────────────┼────────────────┤
│ customerT             │ NULL               │ customerT          │ False          │
│ customerW             │ NULL               │ customerW          │ False          │
│ customerZ             │ NULL               │ customerZ          │ False          │
└───────────────────────┴────────────────────┴────────────────────┴────────────────┘

In the CTE, I am creating a table transformed with all the filenames in the S3 bucket customer-transformed. I am using DuckDB’s utility function glob(search_path) to retrieve the filenames. I do the same for the bucket customer-ingested.

To get the customer status I read from a recent replication snapshot of the application database to get customer_status.

Finally, I join the three expressions to identify missing files (missing values) in the column filenames_ingested for active customers.

TIL that DuckDB now supports Avro files

Wed, 12 Mar 2025 16:00:00 +0000

In the data engineering pipelines of my current company we use two main file formats:

Parquet files for analytical workloads (columnar-storage)
Avro files for transactional / event-based messages (row-storage)

In Querying remote S3 files I wrote about how I use DuckDB to query parquet files stored in S3. Recently, I noticed that DuckDB started supporting reading of Avro files in the same way.

Install extension

To install the extension open DuckDB and run:

INSTALL avro FROM community;
LOAD avro;

Read local files

Local files can be queried with the read_avro function:

select * FROM read_avro('my_dataset.avro');

Read files stored in S3

Remote files can be queried by providing the S3 URI (after logging into the AWS account ):

select * FROM read_avro('s3://my_bucket/my_remote_dataset.avro');

A few limitations of the extension are listed here .

TIL how to fix a broken Joplin sync

Mon, 17 Feb 2025 00:00:00 +0000

I use Joplin for all my note-taking as it’s open-source, markdown-based, and has decent Desktop and Android apps. I use note encryption and sync my notes via WebDAV with my remote Nextcloud instance.

Problem

After not syncing my notes for several weeks, I encountered a couple of errors both on my Desktop app and on my Android app. I don’t have access to the error message anymore, but the errors referred to files and file IDs that caused inconsistencies on the remote server, e.g. broken references:

f1231gdffgasd1asd2.md TypeError: Cannot read property 'type_ of null.

Solution

To solve the issue, I had to delete the offending file on the remote server. However, to not lose it, it’s best to make a local copy of the file’s content if it is present on one of your local devices.

First, I had to determine which file caused the issue, which is a bit harder here as filenames are encrypted. The easiest way to identify the note from the filename is to paste a local markdown reference into a new Joplin note like this:

[Note I am looking for](:/f1231gdffgasd1asd2)

and then click the link in the rendered note. This took me to the offending note.

Second, copy the content into a new note.

Third, I had to delete the file on the remote. I connected on my Mac to the remote WebDAV server via Finder. In Finder, Click on Go -> Connect to Server -> Add address: https://yourserveraddress.org/remote.php/webdav/joplin (look up the link in your Joplin settings), and authenticated with my Nextcloud credentials. Then I was able to search for the file f1231gdffgasd1asd2 and delete it.

The sync now succeeds.

TIL how to debug a broken nextcloud login (again)

Sun, 16 Feb 2025 00:00:00 +0000

The symptoms of this problem were the same as the ones I described in TIL how to debug a broken nextcloud login .

Problem

Phone and Mac apps stopped syncing data with the Nextcloud instance. In my browser I could access the login page, but after entering the details I was being sent back to the same login page without any error message.

Investigation

Last time the issue was caused by excessive logging. However, I reduced the loglevel in the nextcloud config, so this was not the issue here. I am no expert in web admin, PHP, or nextcloud, so I used Claude 3.5 to pair on debugging:

Prompt

I am self-hosting a nextcloud instance on a Hetzner VPC. I currently have the following problem. I can’t log in to nextcloud via my account. The screen just reloads and shows me the login page again. It is not a login detail issue. They are correct. Moreover, other nextcloud related syncs (files, calendar) stopped working. Walk me through how I can debug this issue and get nextcloud up and running again.

I can’t say that the suggestions worked right away, but they put me on the right path and helped me with some useful debugging commands.

The command

df -h

showed me that I didn’t have any free space. While the command

du -sh /var/www/nextcloud/data/*

indicated that in nextcloud a lot of space was occupied by old file versions and deleted files from the nextcloud file sync.

Solution

You can use the nextcloud console occ (ownCloud Console) tool to run maintenance tasks on the nextcloud instance.

The command

sudo -u www-data php /var/www/nextcloud/occ list

lists all available commands.

A couple of clean up commands to remove old file versions and files in the trashbin worked for me:

# Check Nextcloud trashbin size
sudo -u www-data php /var/www/nextcloud/occ trashbin:cleanup --all-users

# Clean up file versions
sudo -u www-data php /var/www/nextcloud/occ versions:cleanup

# Clear Nextcloud cache:
sudo -u www-data php /var/www/nextcloud/occ cache:clear

This freed up enough space for nextcloud to work again.

Unfortunately, the occ commands didn’t work right away because there was apparently not enough space to run any of them. Each one returned the same error message:

fwrite(): write of 4963 bytes failed with errno=28 No space left on device

So I first had to free up a few hundred megabytes:

Clear local repository of retrieved package files:
```
sudo apt-get clean
```

Remove old log files:

sudo find /var/log -type f -name "*.gz" -delete
sudo find /var/log -type f -name "*.1" -delete
sudo truncate -s 0 /var/log/*.log

Remove temporary files:
```
sudo rm -rf /tmp/*
```

This cleared enough space to run the occ commands.

TIL (from Claude) how to create separate RSS feeds for my hugo blog

Thu, 03 Oct 2024 00:00:00 +0000

I really like the idea of providing different content types of my blog in separate RSS feeds. For example a blog could have:

essays
today-I-learned posts
link-list or bloglink posts
short social media style posts
etc.

Providing different RSS feeds for them allows my readers to subscribe to all or a subset of these, depending on their interest. I am using hugo, as the static site generator for this website. Setting up these different feeds for a framework that I am only partially familar with is a great use-case to work with an LLM.

I knew that I would need to:

Distinguish different content types with a flag
Specify different RSS templates that filter on these flags
Tell hugo to generate separate xml files for each content type and use the right templates.

I just didn’t know how to piece these parts together. I first tried googling the answer and read the typical mix of

stackoverflow posts that were not solving the exact same problem
hugo forum discussions that did not have a clear solution or where the solution wasn’t transferable to my unique setup.

I think with those first pointers, spending some time on the hugo documentation, and trial-and-erroring, I could have worked out how to do it. I think that these shallow but unknown problems are the ideal use case for an LLM.

I think of using LLMs as an interface for search + context, where I don’t have to take the laborious work of pre-filtering and connecting the sources. Instead, it helps me get a quick overview and a potential draft of a solution. I can then go deeper and understand the problem better by asking about parts of the solution that I don’t understand. I can then mix this with more googling and get to a solution.

My Goal:

Prompt

I have a hugo blog with the following folder structure:

/content/posts/post1.md
/content/posts/post2.md
/content/posts/_index.md
/content/til/_index.md
/content/til/til1.md
/content/til/til2.md

Markdown pages written in the posts subfolder have type: post in the frontmatter and pages written in the til subfolder have type: til in the submatter.

Explain to me step by step how I can create three separate rss feeds on my website.

Only showing content coming from the til subfolder on til.xml
Only showing content coming from the post subfolder posts.xml
One showing everything on index.xml

Solution

Model: claude-3-5-sonnet-20241022 - response

I’d be happy to explain how you can create three separate RSS feeds for your Hugo blog with the structure you’ve described. Let’s go through this step-by-step:

Create RSS template files: First, you’ll need to create three separate RSS template files in your Hugo project’s layouts folder.

Now, let me explain each step in more detail:

We create three separate RSS template files in the layouts/_default/ directory. Each file corresponds to one of the RSS feeds you want to create.
In each RSS template, we use Hugo’s templating language to generate the RSS feed. The main difference between the templates is in the range statement:
- For til.xml, we use {{ range where .Site.RegularPages "Type" "til" }}
- For posts.xml, we use {{ range where .Site.RegularPages "Type" "post" }}
- For index.xml, we use {{ range .Site.RegularPages }} to include all content
We update the Hugo configuration file to define new output formats for TIL and Posts, and to include these new formats in the home page outputs.
We add links to the new RSS feeds in the site’s HTML, typically in the <head> section or footer.
Finally, we build the site to generate the new RSS feed files.

After following these steps, your Hugo site will generate three separate RSS feeds:

til.xml will only include content from the til subfolder
posts.xml will only include content from the posts subfolder
index.xml will include all content

Would you like me to explain any part of this process in more detail?

I followed these steps, except for using my already existing xml template for til.xml, posts.xml, index.xml. I already read in some hugo discussions that I need to add [outputs] and [outputFormats] to my hugo config.toml file. However, I didn’t really understand the difference between home, page, section and what exactly home = ["HTML", "RSS", "JSON", "TIL", "POSTS"] does. So I asked for clarification:

Prompt

Can you explain to me the config [outputs] and its purpose as well as [outputFormats]?

Model: claude-3-5-sonnet-20241022 - response

Certainly! I’d be happy to explain the [outputs] and [outputFormats] configurations in Hugo and their purposes.

Now, let me provide a more detailed explanation of these configurations:

[outputs]: This configuration section determines what types of files Hugo will generate for different types of pages on your site. In our case, we’re telling Hugo to generate additional output formats for the home page. The line home = ["HTML", "RSS", "TIL", "POSTS"] means:

For the home page, generate the standard HTML and RSS outputs
Also generate our custom TIL and POSTS outputs This is crucial for creating our multiple RSS feeds, as it instructs Hugo to actually generate these files when building the site.

[outputFormats]: This section allows you to define new output formats or modify existing ones. In our case, we’re defining two new output formats: TIL and POSTS. For each new format, we specify:

mediaType: This tells Hugo what kind of content this is. We’re using “application/rss+xml” because we’re creating RSS feeds.
baseName: This determines the filename of the output file. For example, baseName = "til" will result in a file named til.xml.

By defining these custom output formats, we’re giving Hugo the information it needs to generate our specialized RSS feeds.

The interaction between these two configuration sections is key:

[outputFormats] defines what the TIL and POSTS formats are
[outputs] tells Hugo to actually generate these formats for the home page

This setup allows you to have fine-grained control over what files Hugo generates and how they’re named, which is exactly what we need to create multiple, specialized RSS feeds. Is there any part of this you’d like me to elaborate on further?

This helped but I still didn’t really get why I would have to add this to home and not page because I thought that I want to generate rss for page-types.

Prompt

What’s the difference between adding TIL and POSTS to home instead of page?

Model: claude-3-5-sonnet-20241022 - response

That’s an excellent question! The difference between adding output formats to home versus page in Hugo’s configuration is quite significant and affects how and where these outputs are generated. Let me explain:

Now, let me elaborate on the key points:

Scope and Purpose:
- Adding TIL and POSTS to home creates site-wide RSS feeds that include all relevant content from across your site. This is ideal for your use case of creating separate feeds for TIL entries and blog posts.
- If you added these to page, it would create individual RSS feeds for each content page, which is not what you’re aiming for.
Number of Files Generated:
- With home, you get one til.xml and one posts.xml file for your entire site.
- If you used page, you’d get a til.xml and posts.xml for every single page on your site, which would be unnecessary and potentially confusing.
Content Aggregation:
- The home outputs can access and aggregate content from across your entire site, which is perfect for creating comprehensive RSS feeds.
- page outputs typically only contain information from that specific page.
Location of Generated Files:
- home outputs are usually placed at the root of your site, making them easy to find and link to (e.g., https://yoursite.com/til.xml).
- page outputs would be scattered throughout your site structure, mirroring your content organization.
Practical Implementation:
- For your RSS feeds, using home allows you to create the three distinct feeds you want (TIL, posts, and everything) in a clean, organized manner.
- Using page for this purpose would be impractical and wouldn’t achieve your goal of separate, comprehensive feeds.

In summary, for your specific need of creating site-wide RSS feeds for different content types, adding TIL and POSTS to the home outputs is the correct approach. It generates the feeds at the site level, includes all relevant content, and produces a single file for each feed type, which is exactly what you’re looking to achieve.

I followed the suggested approach and added to my config.toml:

# I want to top-level output html, rss, and my two custom outputs
[outputs]
  home = ["html", "rss", "til", "post"]

# Here I define how to process these custom feeds
[outputFormats.til]
  mediaType = "application/rss+xml"
  baseName = "til"

[outputFormats.post]
  mediaType = "application/rss+xml"
  baseName = "posts"

Then I built the page and hugo would build three different xml files:

public/index.xml
public/posts.xml
public/til.xml

However, the files all had the same content as if they were built from the same template instead of the individual templates that filter by page type.

Prompt

This generates the three xml files but they all seem to use the same template file

Model: claude-3-5-sonnet-20241022 - response

I apologize for the confusion in my previous explanation. You’re right to point out that issue. Let’s correct this and explain how to properly set up distinct templates for each RSS feed.

Now, let me explain the key changes and why they solve the problem:

Template File Naming: The crucial change is in how we name and place the template files. Instead of putting them in layouts/_default/, we’re now creating:
- layouts/index.til.xml
- layouts/index.posts.xml
- layouts/index.xml
This naming convention is important because Hugo uses it to match the output formats we defined in the configuration.
Template Content: The content of each template remains largely the same, but we’ve made sure that each one is filtering for the correct content type:
- index.til.xml uses {{ range where .Site.RegularPages "Type" "til" }}
- index.posts.xml uses {{ range where .Site.RegularPages "Type" "post" }}
- index.xml continues to use {{ range .Site.RegularPages }} for all content
Configuration: The configuration in config.toml remains the same as before. The [outputFormats] and [outputs] sections are correctly set up to generate these files.
How It Works:
- When Hugo sees TIL in the [outputs] section, it looks for a template named index.til.xml
- Similarly, for POSTS, it looks for index.posts.xml
- The default RSS output uses index.xml

This setup ensures that each RSS feed uses its own template, allowing you to customize the content for each feed independently.

The reason your previous attempt generated three XML files with the same content was likely because Hugo was falling back to a default RSS template when it couldn’t find specific templates for TIL and POSTS.

With these changes, you should now have three distinct RSS feeds, each with its appropriate content. Does this explanation help clarify the issue? Would you like me to elaborate on any part of this solution?

So I moved the template layout files as suggested to:

layouts/index.til.xml
layouts/index.posts.xml
layouts/index.xml

and they were correctly selected. You can find the correct feeds here:

https://staticnotes.org/index.xml

https://staticnotes.org/posts.xml

https://staticnotes.org/til.xml

Conclusion

A common criticism is that LLMs will confidently hallucinate incorrect information. I think this will likely be true for the next couple of years. However, when I use the old-school google + stackoverflow + documentation + online discussion workflow I also might generate an incorrect approach. This can happen because I misunderstand the content or because I can’t piece the information from different sources correctly together for my use case.

I think viewing an LLM as an interactive interface to search and combine information and context is the right mental model to use them productively.

TIL about the QUALIFY statement in SQL

Mon, 16 Sep 2024 00:00:00 +0000

Today I came across the QUALIFY clause which is supported in some SQL dialects. It’s not part of the SQL standard but supported by main analytical databases like BigQuery, Snowflake, Oracle, Databricks, DuckDB, etc. It’s part of Snowflake’s dialect which is the data warehouse that I use at work. The QUALIFY statement lets me filter the result of a query based on the result of a window function.

The order of execution is the following:

-- SELECT <columns>
--   FROM <table>
--   [GROUP BY ...]
--   [HAVING ...]
--   QUALIFY <window function predicate>

QUALIFY is for window functions what HAVING is for group-by statements. It allows to filter based on the outputs of these operations without having to use a subquery or CTE. Therefore I can write more concise SQL.

I often deduplicate data using the row_number() window function. Sometimes I just want the latest event for a particular user in a dataset.

An example would be:

-- logs with some user events and timestamps
with logs_annotated as (
    select
        logs.*,
        row_number() over (partition by user_id order by event_timestamp desc) as row_number
    from logs
)
select * from logs_annotated where row_number = 1

With QUALIFY I can write one statement instead:

select * from logs
qualify row_number() over (partition by user_id order by event_timestamp desc) = 1

This works because the QUALIFY statement is evaluated after the window function result has been computed and can therefore be used in filtering.

TIL about just and justfiles

Fri, 13 Sep 2024 00:00:00 +0000

just is just a command runner that I recently started using to define aliases for commands in my projects, e.g. build, test, or deploy commands.

Installation

just can be installed via homebrew:

brew install just

Example

I want to create alias commands for my dbt project to build my dbt models in production vs. development environments. At the moment, I need to run the commands:

# prod warehouse
poetry run dbt run --target prod
# dev warehouse
poetry run dbt run --target dev

To define shorter just commands for this project I add a justfile into the project folder.

# justfile
run-prod *ARGS:
    poetry run dbt run --target prod {{ARGS}}

run-dev *ARGS:
    poetry run dbt run --target dev {{ARGS}}

If I am inside the project folder I can now run just run-prod and just run-dev. Moreover, I can pass in any dbt keyword arguments, e.g.

just run-dev --select mymodel

will resolve to

poetry run dbt run --target dev --select mymodel

Is this not just `make`?

just is inspired by the Unix build tool make from 1978. However, just is not a build system and focuses solely on command running. This avoids having to deal with things like .PHONY that can can catch you out in make recipes. I also like two additional features of just:

I can run my just commands from any subfolder of the project folder that contains the justfile. This is useful when I work on this blog. I have to run hugo serve from the top-level project folder and jupyter-to-markdown converter nb2hugo from the notebooks/project_x/ subfolder. With just command aliases it doesn’t matter from which folder I run the command.

just allows you to load environmental variables from an .env file.

Example .env:

DATABASE=localhost:1111
USERACCOUNT=rob1234

Add set load-dotenv to your justfile and the environment variables will be available in your shell:

set dotenv-load

check-vars:
    @echo "Env vars are DATABASE:$DATABASE and USER:$USERACCOUNT"

Running just check-vars will output:

Env vars are DATABASE:localhost:1111 and USER:rob1234

TIL to create continuously numbered lists with content between items

Wed, 21 Aug 2024 00:00:00 +0000

When writing notes in markdown I commonly encounter the problem that I want to create an automatically numbered list with content between the items. However, this messes up the numbering as it creates two separate numbered lists, e.g.

My first point
My second point

code block or image

My third point
My fourth point

The solution to this is embarrassingly obvious once you know it. Just move the image or code block one level down by indenting it with four spaces. This keeps the list intact.

My first point
My second point

code block or image
My third point
My fourth point

TIL how to set a lower threshold on a column in SQL

Wed, 07 Aug 2024 00:00:00 +0000

For the last couple of months I came across multiple instances where I wanted to write an SQL query to lower threshold all the values in a particular column in one of our Snowflake tables.

I used quite clunky workarounds to do that. It turns out you can use the SQL functions GREATEST or LEAST for it.

Example

I am creating a temporary table in the first CTE with a column DATA_COLUMN that has positive and negative numbers.

with example_table as (
SELECT * 
FROM (
    VALUES 
        (1, -100.423), 
        (2, 5.0), 
        (3, -0.001), 
        (4, 100.0), 
        (5, 12.0)
) AS t (row_nr, data_column)
)

select GREATEST(DATA_COLUMN, 0.) as thresholded 
from example_table

Output:

The function’s primary purpose is to take the greatest value among a number of referenced columns, e.g. GREATEST(column1, column2, column3).

TIL how to set up a local LLM as an assistant in VSCode

Wed, 10 Jan 2024 00:00:00 +0000

I am still trying to get used to the fact that the product of state-of-the-art AI companies is essentially a GB-sized CSV file and some instructions about the network architecture to perform the inference. Recently released “open source” models of medium-size are showing quite decent performance on benchmarks , so I was interested how easy it would be to set them up locally to help me with coding tasks in VSCode. Short answer: very easy.

The bad news first. Those local small-to-medium sized models won’t reach the performance of remotely hosted GPT models. I also think the local inference will likely influence battery life of your machine, but there are also some benefits of local LLMs:

free, local inference from “open-source” models (no company lock-in for inference)
works without internet access which can also substitute a quick google search if one is in a place with bad reception
no sharing of sensitive information with 3rd parties
customizability. You can pick a model fine-tuned for your use specific use-case

Figure 1. Using codellama for work on a low-cost carrier flight without internet access.

Hardware requirements

I am using a 2018 MacBook Pro with MacOS and VSCode. For 7B parameter models you need 8GB+ of RAM, for 13B parameter models you need 16GB+ of RAM. Smaller models also return the response quicker. So far a 7B parameter model gives me a response within 3-5s.

Downloading local models

To manage models and handle the inference interface, we use Ollama as our model provider:

brew install ollama

Then, we use the ollama pull [model_name] command to download one of the models from Ollama’s model library . I am using Meta’s open source Unfortunately, in 2023 open source in the context of LLMs for most people with small budgets means: open to use, not open to reproduce.Llama2 model fine-tuned for coding tasks and a “moderate size” of 7b parameters. To get the familiar chat-like experience, use an instruct model that was trained to output human-like answers to questions. More details about the different model variations and the intended use cases can be found here .

In the CLI run: ollama pull codellama:7b-instruct This will download the 3.8GB model from the internet and save the model parameters to your hard drive.

Prompting the model from CLI

After downloading the model, we can prompt it from the CLI with ollama run codellama:7b-instruct and then type in our prompt.

Using the model as a coding assistant inside VS Code

To prompt the model from within VSCode I am using the extension Continue. By default Continue uses GPT-4 in trial mode. However, you can point Continue to any local or remote model by specifying local paths, API keys, etc. in the config.json. In our case we want to point Continue to our codellama model, so we add the following to the file:

// config.json
"models": [
    {
      "title": "GPT-4",
      "provider": "openai-free-trial",
      "model": "gpt-4"
    },
    {
      "title": "GPT-3.5-Turbo",
      "provider": "openai-free-trial",
      "model": "gpt-3.5-turbo"
    },
// add the local model here
    {
      "title": "codellama:7b-instruct",
      "model": "codellama:7b-instruct",
      "contextLength": 4096,
      "provider": "ollama"
    }
  ],

The nice thing about Continue is that it allows you to easily jump between local and remote models. This is useful if you are unhappy with a response and want to try a different model.

TIL how to generate a dbt staging model description from snowflake table

Wed, 04 Oct 2023 00:00:00 +0000

In my team we use dbt to create, document, and test data models in our data warehouse, snowflake . We use different layers of models that start from the raw data and in each layer increase complexity and specialization for the target use case:

source data in a raw table
thin staging models (standardization of columns, no joins, no aggregations)
intermediate models (reoccurring building blocks that are defined once and used across models in the next layer)
mart models (most detailed models organised by department / business context / application)

Problem

The creation of staging models can sometimes be a bit mind-numbing, especially when the raw table has many columns. Let’s take a small example table USERS:

id	first_name	last_name	country	is_active
1	Peter	Clark	US	True
2	Sarah	Svenson	Sweden	False
3	Roberto	Gonzales	Spain	True

and dbt staging model stg_users.sql:

select 
id::number as id, 
first_name::varchar as first_name, 
last_name::varchar as last_name, 
country::varchar as country, 
is_active::boolean as is_active
from {{ source('RAW_DATA', 'USERS') }}

The staging model definition typically follows the pattern: raw-table-column-name::type as staging-model-column-name. I have used vim macros in the past to quickly write the staging model definition, but often end up copy-pasting and looking up the inferred types in snowflake for ten minutes.

Solution

We can run two SQL queries in snowflake to generate most of the dbt model defintion:

desc table "RAW_DATA"."DATA_SCHEMA"."USERS";
select CONCAT("name", '::', "type", ' as ', "name", ',') as dbt_model_def from table(result_scan(LAST_QUERY_ID()))

The first line (DESCRIBE TABLE) outputs the table definition. The second command takes the column name and type information about the table and concatenates them into the right format:

LAST_QUERY_ID(): returns the id of the desc table query
RESULT_SCAN(): returns the result of the last command
TABLE(): converts the output to a queryable table

The output of the query is a table with each row formatted with the above pattern. We can copy-paste the output in one step into our staging model file.

TIL how to use snowflake's VALUES sub-clause

Tue, 03 Oct 2023 00:00:00 +0000

Today I found out about the useful VALUES() sub-clause in snowflake that allows you to dynamically generate a fixed, known set of rows inside your query.

Problem

I had a table exchange_rates of EUR-currency pairs with their corresponding exchange rates

date	base_currency	target_currency	exchange_rate
2023-10-03	‘EUR’	‘GBP’	0.87
2023-10-03	‘EUR’	‘USD’	1.05
2023-10-03	‘EUR’	‘JPY’	155.95

that I wanted to join with a table transactions which contained transaction prices in different currencies:

id	price	price_currency
1	100.53	‘GBP’
2	1021.68	‘EUR’
3	7.99	‘JPY’

My goal was to standardize all prices to EUR.

The SQL query to join the two tables would be:

select 
id,
price,
price_currency,
exchange_rate
from transactions as t
inner join exchange_rates as er on er.target_currency = t.price_currency

The issue was that exchange_rates did not have a EUR-EUR currency pair and I didn’t have access to the data source. So, the inner join would filter out all EUR transactions (or leave NULLs in case of a left join). This is a bit of a constructed problem to show an application of VALUES(). One could alternatively fill the NULLs with 1.0s after a left join.

Solution

I used snowflake’s VALUES() to dynamically generate another row in the query. I use this with a UNION statement to complete the exchange_rates table before joining:

with exchange_rates_complete as (
  select * from exchange_rates
  union
  select * from (values( '2023-10-03', 'EUR', 'EUR', 1.0, 1))
)

select 
id,
price,
price_currency,
exchange_rate
from transactions as t
inner join exchange_rates_complete as er on er.target_currency = t.price_currency

The joined table looks like this:

id	price	price_currency	exchange_rate
1	100.53	‘GBP’	0.87
2	1021.68	‘EUR’	1.0
3	7.99	‘JPY’	155.95

TIL how to debug a broken nextcloud login

Sat, 19 Aug 2023 00:00:00 +0000

For the past month I was confronted with the downside of self-hosting. I use Nextcloud for me and my partner’s calendar, online file storage, and notes syncing. I host Nextcloud on a small 7EUR/month Hetzner VPC. Unfortunately, about a month ago it stopped working. Phone apps would not sync the calendar, Joplin notes were not synced across devices and I could not log into my Nextcloud account. I avoided trying to fix the issue quickly, because I know from experience that a short bug investigation in a self-hosting project can quickly spiral into a weekend long endeavour. Fortunately, I had some time this morning and I could fix the issue quickly. Leaving this here for future reference.

Problem

Figure 1. Login window in my browser that I was being redirected to.

Investigation

It is quite difficult to understand the issue without an error message. The Nextcloud docs suggest to look at the server logs to get a better idea.

To access the logs I use ssh to log into the VPC and opened the nextcloud.log log file.

$ ssh username@[VPC-IP] 
$ vim /var/www/nextcloud/data/nextcloud.log

Trying to open the file in vim showed me the likely issue. The log file was so large that it would take minutes to open. From the error message I could see that nextcloud ran out of space during previous requests.

Solution

Truncate the big log file using the following steps:

Put nextcloud in maintenance mode.

Set 'maintenance' => true in the config file /var/www/nextcloud/config/config.php

# config.php
<?php
[...]
  'dbtype' => 'mysql',
  'version' => '22.2.3.0',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'installed' => true,
  'maintenance' => true,
  'theme' => '',
  'loglevel' => 0,
[...]
);

Looking at the config file, I also noticed that I had loglevel set to 0 (the most verbose), which is the likely cause of the file size. Nextcloud has the following log levels:

0: DEBUG: All activity; the most detailed logging.
1: INFO: Activity such as user logins and file activities, plus warnings, errors, and fatal errors.
2: WARN: Operations succeed, but with warnings of potential problems, plus errors and fatal errors.
3: ERROR: An operation fails, but other services and operations continue, plus fatal errors.
4: FATAL: The server stops.

So I changed 'loglevel' => 2 to prevent the same issue from reappearing in a year.

Truncate the log file to 0 bytes to free space using the unix truncate command:

$sudo -u www-data truncate /var/www/nextcloud/data/nextcloud.log --size 0

Disable maintenance mode 'maintenance' => false in the config file.

This allowed me to login using my browser. Moreover, all apps started syncing again.