Goal

Give these concerns, my objectives are clear: I want to use tools that can genuinely improve my coding productivity. Here are my criteria:

1. Open Source: If the creator abandons the project, I can still use and potentially maintain my own version.
2. Customizable: I should be able to disable features I don’t need.
3. Familiar: I should still be able to use my preferred tools to reduce the stress of migration.
4. Keyboard-centric: I want to use the keyboard as much as possible for efficiency.

Following these criteria, I chose NeoVim to replace other IDEs, When selecting a terminal emulator among options like Kitty, WezTerm, iTerm2, and the system default terminal. I opted for WezTerm. This choice was influenced by the fact that I can use Lua to configure both NeoVim and WezTerm.

How

Now, let me introduce you to my terminal configuration step. My toolkit consists of WezTerm, NeoVim, Sketchybar, yabai and skhd. Let’s explore each of these components in detail.

Conclusion

In this post, I’ve explained my motivations for building a custom terminal configuration and provided an overview of the tools I’ve chosen. While I haven’t delved into the specifics of each configuration. I’ve shared links to my setup files for those interested in replicating or adapting my approach.

This journey has not only improve my productivity but also deepened my understand of the tools I use daily. As I continue to use and refine this setup, I look forward to further optimizations and discoveries. If you have any questions or suggestions about my configuration, please let me know.

Use Case: Analyzing ETH Address Transactions

Let’s consider a common scenario: analyzing the transaction information of a specific Ethereum (ETH) address.

The Traditional Process vs DuckDB

Previously, this process involved several steps:

1. Fetching data using requests
2. Using pandas to preview and clean the data
3. Finally, analyzing the data using DuckDB

The data preparation stage typically involves three essential steps:

1. Handing pagination for third-party APIs
2. Extracting valid fields from the returned data
3. Cleaning the data, including null values and ata types

In this article, we’ll demonstrate how to streamline this process using the blockscout API¹, DuckDB UDF², and DuckDB Macro³.

Demonstration

We’ll explore two solutions for processing data using DuckDB, First let’s import related dependencies

import duckdb
import requests
from duckdb.typing import VARCHAR, INTEGER, DuckDBPyType
import json

def blockscout_api(module: str, action: str, address: str, start_block: int, end_block: int, page: int, offset: int) -> list[str]:
    url_prefix = f'https://eth.blockscout.com/api?module={module}&action={action}'
    
    result = []
    while True:
        url = f'{url_prefix}&address={address}&startblock={start_block}&endblock={end_block}&page={page}&offset={offset}&sort=asc'
        print(f'query page {page}')
        data = requests.get(url).json()
        if data['message'] == 'OK':
            items = data['result']
            result.extend(map(json.dumps,items))
        else:
            break
        if len(items) < offset:
            break
        page += 1
    return result

conn = duckdb.connect()
conn = conn.create_function('blockscout_api', blockscout_api)

conn.execute("""
CREATE OR REPLACE MACRO blockscout_trxs(address, start_block, end_block) as table 
    select blockscout_api('account', 'txlist', address, start_block, end_block, 1, 2) as data
""")

conn.execute("""
with raw_transactions as (
    select unnest(data) as trx from blockscout_trxs('0x603602E9A2ac7f1E26717C2b2193Fd68f5fafFf6', 20485198, 20490674)
), decode_transactions as (
select 
    trx->'$.blockHash' as block_hash,
    (trx->'$.blockNumber')::integer as block_number,
    (trx->'$.timeStamp')::integer as timestamp,
    to_timestamp(timestamp) as datetime,
    trx->'$.hash' as hash,
    (trx->'$.transactionIndex')::integer as transaction_index,
    trx->'$.from' as 'from',
    trx->'$.to' as 'to',
    trx->'$.value' as value,
    trx->'$.contractAddress' as contract_address,
    (trx->'$.gas')::integer as gas,
    (trx->'$.gasPrice')::bigint as gas_price,
    (trx->'$.gasUsed')::integer as gas_used,
    trx->'$.isError' as is_error,
    trx->'$.txreceipt_status' as txreceipt_status,
    trx->'input' as 'input'
from raw_transactions
)
select 
  block_number,
  datetime,
  hash,
  'from',
  'to',
  value,
from decode_transactions
""").df()

query page 1
query page 2

fields = {
    'blockHash': str,
    'blockNumber': int,
    'timeStamp': int,
    'hash': str,
    'transactionIndex': int,
    'from': str,
    'to': str,
    'value': str,
    'contractAddress': str,
    'gas': int,
    'gasPrice': int,
    'gasUsed': int,
    'isError': int,
    'txreceipt_status': int,
    'input': str,
}

field_keys = fields.keys()

def blockscout_api_with_fields(module: str, action: str, address: str, start_block: int, end_block: int, page: int, offset: int):
    url_prefix = f'https://eth.blockscout.com/api?module={module}&action={action}'
    result = []
    while True:
        url = f'{url_prefix}&address={address}&startblock={start_block}&endblock={end_block}&page={page}&offset={offset}&sort=asc'
        print(f'query page {page} -> {url}')
        resp = requests.get(url).json()
        if resp['message'] == 'OK':
            items = resp['result']
            result.extend([{f: i[f] for f in field_keys} for i in items])
            if len(items) < offset:
                break
        else:
            break
        page += 1
    return result

conn = duckdb.connect()
conn = conn.create_function(blockscout_api_with_fields.__name__, blockscout_api_with_fields, [VARCHAR, VARCHAR, VARCHAR, INTEGER, INTEGER, INTEGER, INTEGER], DuckDBPyType(list[fields]))
conn.execute("""
CREATE OR REPLACE MACRO blockscout_trxs_with_fields(address, start_block, end_block) as table 
    select blockscout_api_with_fields('account', 'txlist', address, start_block, end_block, 1, 5) as data
""")

conn.execute("""
with raw_transactions as (
    select unnest(data) as trx from blockscout_trxs_with_fields('0x603602E9A2ac7f1E26717C2b2193Fd68f5fafFf6', 20485198, 20490674)
), flatten_transactions as (
  select unnest(trx) from raw_transactions
)
select 
  blockNumber as block_number,
  to_timestamp(timeStamp) as datetime,
  hash,
  'from',
  'to',
  value
from flatten_transactions
""").df()

query page 1 -> https://eth.blockscout.com/api?module=account&action=txlist&address=0x603602E9A2ac7f1E26717C2b2193Fd68f5fafFf6&startblock=20485198&endblock=20490674&page=1&offset=5&sort=asc

Summary

By leveraging UDF⁶ and Macro⁷ of DuckDB, we can significantly simplify the data processing workflow. This approach makes data analysis more efficient and results in cleaner, more maintainable code.

For day-to-day use, Solution 1 is recommended for quick analysis and verification. However, for production environments, Solution 2 is preferred due to its stricter field constraints, which help prevent issues during data processing.

In addition, you can also use with recursive⁸ to achieve pagination query, but this process is more complicated, more SQL writing, not recommended for use. If you are interested, I will write an article to share it later.

references

DuckDB

Considering that the task data already exists in Postgres, and visualization is needed, DuckDB has a Postgres plugin that can directly obtain data Therefore, choose to use DuckDB with Jupyter Notebook to quickly verify the logic.

Actual operation

The part of DuckDB operating Postgres can refer to PostgreSQL Extension

Omit the irrelevant code, the code logic is divided into several steps

import duckdb

conn = duckdb.connect()

mock_data_sql = """
SELECT * FROM (
    VALUES 
      (1, DATE '2024-07-01'),
      (1, DATE '2024-07-02'),
      (1, DATE '2024-07-03'),
      (1, DATE '2024-07-05'),
      (1, DATE '2024-07-06'),
      (2, DATE '2024-07-01'),
      (2, DATE '2024-07-02'),
      (2, DATE '2024-07-03'),
      (3, DATE '2024-07-01'),
      (3, DATE '2024-07-02'),
      (3, DATE '2024-07-03'),
      (3, DATE '2024-07-04'),
      (3, DATE '2024-07-05'),
      (3, DATE '2024-07-06'),
      (3, DATE '2024-07-07'),
      (3, DATE '2024-07-08'),
      (3, DATE '2024-07-09'),
      (4, DATE '2024-07-05'),
      (4, DATE '2024-07-06'),
      (4, DATE '2024-07-07'),
      (4, DATE '2024-07-08'),
      (4, DATE '2024-07-09'),
  ) AS t(source_id, end_date)
"""
df = conn.execute(mock_data_sql).df()
df

group_date_sql = """
 SELECT 
    source_id,
    end_date,
    end_date - INTERVAL (ROW_NUMBER() OVER (PARTITION BY source_id ORDER BY end_date) - 1) DAY AS group_date
  FROM df
  order by source_id, end_date
"""
grouped_date_df = conn.execute(group_date_sql).df()
grouped_date_df

group_sql = """
SELECT 
    source_id,
    MIN(end_date) AS start_date,
    MAX(end_date) AS end_date,
  FROM grouped_date_df
  GROUP BY source_id, group_date
  ORDER BY source_id, group_date
"""
grouped_df = conn.execute(group_sql).df()
grouped_df

import plotly.express as px
fig = px.timeline(grouped_df, x_start='start_date', x_end='end_date', y='source_id')
fig.update_yaxes(autorange="reversed")
fig.show()

sql = """
with raw_data as (
  SELECT * FROM (
    VALUES 
      (1, DATE '2024-07-01'),
      (1, DATE '2024-07-02'),
      (1, DATE '2024-07-03'),
      (1, DATE '2024-07-05'),
      (1, DATE '2024-07-06'),
      (2, DATE '2024-07-01'),
      (2, DATE '2024-07-02'),
      (2, DATE '2024-07-03'),
      (3, DATE '2024-07-01'),
      (3, DATE '2024-07-02'),
      (3, DATE '2024-07-03'),
      (3, DATE '2024-07-04'),
      (3, DATE '2024-07-05'),
      (3, DATE '2024-07-06'),
      (3, DATE '2024-07-07'),
      (3, DATE '2024-07-08'),
      (3, DATE '2024-07-09'),
      (4, DATE '2024-07-05'),
      (4, DATE '2024-07-06'),
      (4, DATE '2024-07-07'),
      (4, DATE '2024-07-08'),
      (4, DATE '2024-07-09'),
  ) AS t(source_id, end_date)
), group_date as (
  SELECT 
    source_id,
    end_date,
    end_date - INTERVAL (ROW_NUMBER() OVER (PARTITION BY source_id ORDER BY end_date) - 1) DAY AS group_date
  FROM raw_data
  order by source_id, end_date
), final as (
  SELECT 
    source_id,
    MIN(end_date) AS start_date,
    MAX(end_date) AS end_date,
  FROM grouped_date_df
  GROUP BY source_id, group_date
  ORDER BY source_id, group_date
)
from final
"""
date_df = conn.execute(sql).df()
gap_fig = px.timeline(date_df, x_start='start_date', x_end='end_date', y='source_id')
gap_fig.update_yaxes(autorange="reversed")
gap_fig.show()

Summary

This article mainly introduces how to use DuckDB to find missing dates. Through window functions and grouping functions, missing dates can be quickly located.

PS: Recently, the frequency of using DuckDB in work is getting higher and higher. The reason is that DuckDB is lightweight, fast, and the efficiency is surprisingly high when combined with Jupyter Notebook.

Install dependencies

!pip install duckdb jupysql --quiet

Basic configuration

import duckdb

conn = duckdb.connect()

jupysql configuration

%load_ext sql

%config SqlMagic.autopandas = True
%config SqlMagic.feedback = 0
%config SqlMagic.displaycon = True
%config SqlMagic.displaylimit = 10

%sql conn --alias duckdb-native

Demo data

Refer to Jupyter Notebooks

%%sql --save short_trips --no-execute
SELECT *
FROM 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet'
WHERE trip_distance < 6.3

%sqlplot histogram --table short_trips --column trip_distance --bins 10 --with short_trips