Sql on staticnotes.org

TIL how to investigate missing data easily with DuckDB

Fri, 16 May 2025 14:00:00 +0000

In data engineering pipelines we commonly store output data in S3 buckets at various stages. In this particular case we stored the input data received from customers as csv files in the S3 bucket s3://customer-ingested/customerA.csv and after some transformations we stored the transformed data as csv files in another S3 bucket s3://customer-transformed/customerA.csv.

Problem

For some of the customers the csv file was missing in the first bucket customer-ingested while we were sure that all customers had a file in customer-transformed. However, there was a legit reason why the file could be missing which was that in the application the customer had a state of is_deactivated = True.

TIL that DuckDB now supports Avro files

Wed, 12 Mar 2025 16:00:00 +0000

In the data engineering pipelines of my current company we use two main file formats:

Parquet files for analytical workloads (columnar-storage)
Avro files for transactional / event-based messages (row-storage)

In Querying remote S3 files I wrote about how I use DuckDB to query parquet files stored in S3. Recently, I noticed that DuckDB started supporting reading of Avro files in the same way.

TIL about the QUALIFY statement in SQL

Mon, 16 Sep 2024 00:00:00 +0000

Today I came across the QUALIFY clause which is supported in some SQL dialects. It’s not part of the SQL standard but supported by main analytical databases like BigQuery, Snowflake, Oracle, Databricks, DuckDB, etc. It’s part of Snowflake’s dialect which is the data warehouse that I use at work. The QUALIFY statement lets me filter the result of a query based on the result of a window function.

TIL how to set a lower threshold on a column in SQL

Wed, 07 Aug 2024 00:00:00 +0000

For the last couple of months I came across multiple instances where I wanted to write an SQL query to lower threshold all the values in a particular column in one of our Snowflake tables.

I used quite clunky workarounds to do that. It turns out you can use the SQL functions GREATEST or LEAST for it.

Example

I am creating a temporary table in the first CTE with a column DATA_COLUMN that has positive and negative numbers.

DuckDB use cases for data scientists: Querying remote S3 files

Sun, 25 Feb 2024 00:00:00 +0000

DuckDB is a pretty cool in-process OLAP analytical database that I started to spin up on the fly for quick data analysis. What SQLite is to Postgres, DuckDB is to Snowflake. It is a single executable without dependencies and stores databases in local files.

I can think of four use cases for data science work:

DuckDB supports larger-than-memory workloads by loading data sequentially. You can use it to analysis datasets that are too large for Pandas (and too small to justify PySpark).
I can query CSV, parquet, and JSON files directly from remote endpoints, e.g. S3, using SQL.
I can replace Snowflake queries with DuckDB queries in unit / integration tests.
I can set up a (DuckDB + dbt) data warehouse for local development.

I want to share here my workflow for the second use case. Inspecting parquet files in AWS S3 is a pain because I can’t easily inspect them in the AWS console. Since a few months I use DuckDB to load, inspect, and analyse parquet files from the command line. I found this reduced my cognitive load in situations where I quickly want to check a remote file, because I don’t have to download the parquet file and write a python script to inspect it.

TIL how to generate a dbt staging model description from snowflake table

Wed, 04 Oct 2023 00:00:00 +0000

In my team we use dbt to create, document, and test data models in our data warehouse, snowflake . We use different layers of models that start from the raw data and in each layer increase complexity and specialization for the target use case:

TIL how to use snowflake's VALUES sub-clause

Tue, 03 Oct 2023 00:00:00 +0000

Today I found out about the useful VALUES() sub-clause in snowflake that allows you to dynamically generate a fixed, known set of rows inside your query.

Problem

I had a table exchange_rates of EUR-currency pairs with their corresponding exchange rates

date	base_currency	target_currency	exchange_rate
2023-10-03	‘EUR’	‘GBP’	0.87
2023-10-03	‘EUR’	‘USD’	1.05
2023-10-03	‘EUR’	‘JPY’	155.95

that I wanted to join with a table transactions which contained transaction prices in different currencies: