For a while now, the DuckDB Iceberg extension has had great support for AWS S3 Tables (AWS’s managed Iceberg table experience); you can perform basic CRUD operations such as create a table, insert, update, and delete. I wrote about this a few months ago here; but for many that have been on the AWS stack for a while, you all know well that S3 Tables are still relatively new and arrived years after AWS started supporting Iceberg tables in their regular Glue catalogs. But DuckDB and their Iceberg extension had not been focused on this “legacy” glue stuff…until now :).

The DuckDB 1.5 update brought a significant update to standard AWS glue catalogs, which this article will cover. We will demonstrate how to connect to a regular glue catalog, create new Iceberg tables, run inserts, updates, and deletes on them.

Connecting to the Catalog

To connect to a standard glue iceberg catalog with DuckDB, this is all it takes:

I’m using the credential chain to authenticate to AWS, which IMO is best practice here vs. using an IAM user with persistent access keys.

Creating the Table

The major update that DuckDB V1.5 and iceberg brought was the ability to specify a location for an Iceberg table in the SQL using the “WITH” predicate. This feature was not necessary with S3 Tables, since the location was built into the backend ARN. Also, DuckDB enabled CTAS operations, which is rather interesting, given that AWS’s official docs located here indicate that their Iceberg rest endpoint does not support this (see section on creating a table and the remark on “stage create”). Gotta give it to the Duck and its parsing gymnastics to get the job done:

Basic CRUD Operations

As I mentioned before, we can also perform INSERT/UPDATE/DELETE operations on these tables like so:

And Now…Let’s See The Results

Voila…

A Gotcha

One thing I noticed is that when I attempted to drop the table before creating it via a “DROP TABLE IF EXISTS”, it kind of worked if the table was already there. But if it was missing, I got some weird HTTP error. Thus, as a workaround, I instead wrote 2 functions to use the GLUE Api to nuke the table and the S3 API to delete all its supporting files. Running that prior to creating the table makes the process repeatable with no errors. Below is the code that does that:

Additional Remarks

I also tested these 2 CRUD operations, which did not work:

• CREATE OR REPLACE TABLE AS…

• MERGE INTO

The good new is that after testing those 2 operations, I filed these issues here and here on the official duckdb-iceberg extension repo and got a quick response that these were in the works, which is great to hear. Like I’ve said many times, I think with just a little more time, the DuckDB Iceberg extension will be nearly feature complete. The 1.5 release was a giant leap forward IMO.

THE CODE

Thanks for reading,

Matt

I think it’s no secret at this point, but if you saw my post the other day or the LinkedIn announcement, you’d know that Alex and I are hard at work on the O’Reilly book titled “DuckLake - The Definitive Guide”. With that being said, when Ducklake first came out in 2025, I was amazed at how easy it was to get up and running; but like most data warehousing/“lake-housing” I’ve done in the past, I always am curious on how these implementations look on downstream BI tools. Because let’s be honest folks:

Some might scoff at such a statement and say “how dare you”, but I would respond with “why go through the effort to curate this data if no one is going to use it to make informed decisions?”

Anyways, today’s article shows how to get DuckLake to talk to Tableau Desktop.

Side Note - Out of all the BI tools I’ve used (I’ve used most of them), Tableau is by far the easiest and most intuitive. That’s just my opinion though

Step 1: Let’s Build a DuckLake

For today’s article, I will use a python script and duckDB to build out some fake datasets - a classic order header and detail, with the theme of this one being a store that sells flashlights. Honestly though, thanks to the advancements in LLM’s over the last year, I just gave copilot and GPT 5.3 codex this prompt and it generated the python script in less than 10 seconds:

Once the script was ready, all I had to do was run this in terminal:

And voila - 2 csv’s were generated. Now onto the funner part - let’s load this data to some ducklake tables in AWS. To do that, all you need is a simple script like this:

I ran that in the duckdb CLI and within a couple seconds, my ducklake catalog was created and the CSV data was slingshotted up to S3 from my laptop. Simply amazing at how brain-dead easy this is.

Ok - We Have Our Lakehouse - But What About Tableau?

Tableau desktop can be downloaded as a free trial, which is what I did. And DuckDB has good documentation that can be found here that shows how to connect to Tableau. So for my first attempt, I clicked on the “Other (JDBC)” connector for tableau, set the dialect to Postgres on the drop down and then connected to a duckdb database like this:

The connection worked fine, but then here was the fundamental problem:

After some snooping around on Google and consulting some experts, the only real solution would be to use a Tableau connector that would support an “initial SQL” statement e.g. when you connect, run this command first before connecting to the actual database.

But then lo and behold, if you read further down on DuckDB’s Tableau documentation, MotherDuck has put out their own custom connector for Tableau that does allow an init sql statement. So I figured, is this all I need? Let’s go try:

1. Download this “taco” file from here

   1. a taco file is a custom connector file for Tableau

2. Once its downloaded, copy it into this path (if on a Mac): ~/Documents/My Tableau Repository/Connectors

3. restart Tableau

You should now see in your connectors menu a new one called “DuckDB by MotherDuck”

Click on that and you will see this pop-up that has a general and an “Initial SQL” tab:

Click on the Initial SQL tab and add in your secrets command as well as attaching the ducklake catalog that you had previously generated in the earlier part of this article:

After that, hit sign-in and you should be at the landing page that shows your available schemas and tables:

Under database, we should see our “aws_wh” we created with ducklake and attached. Then under main, we should see our tables. Let’s drag those on the canvas and start slicing:

Boom - it worked. Tableau is using DuckDB and connecting to our ducklake tables that live and breath in AWS S3. Winning!

Here’s another slice I did for fun:

Conclusion

This article showed us how to create a simple ducklake lakehouse from scratch, inject some dummy data, send it up to S3, and then have Tableau connect and visualize it. In regards to other BI tools such as PowerBI, I have not done much research; but the kicker is that as long as the connector supports an “initial sql” block, you should be able to make it work.

Hope this helped you learn something!

THE CODE

Thanks for reading,
Matt

I haven’t written on Substack in over a month, and if you saw the announcement this morning via LinkedIn, you’d understand why. I’m currently working on a book with Alex Monahan that we hope to roll out by the end of the year. I’ve never written a book before so this will be a major shift in how I approach projects. I do plan to still write here from time to time, but don’t expect an article every couple weeks.

Stay tuned though: “DuckLake - The Definitive Guide” is going to be awesome, and Alex and I look forward to getting it out the door so you all can learn about the newest offering of a lakehouse and where it fits in.

If this is your first time hearing about it, here’s a primer that I wrote back in June of last year: Ducklake - The Easiest Way to Lakehouse

Thanks,
Matt

I’ve been on a roll lately with MotherDuck, who is the sponsor of today’s article:

Read more

We live in a world where nothing is ever perfect; especially in the data engineering space, we are tasked quite often with solving integration problems where we have multiple catalogs and multiple data stores floating around the ether; to further complicate things, these catalogs can also be in different clouds or managed services. I know most of you re…

Read more

We live in a world these days where we have many choices abound for data services; the 3 predominant cloud providers are no doubt AWS, GCP and Azure; and what’s really interesting about the current landscape is that many orgs find themselves in situations where they have built up apps and data stores that span across more than just one cloud; this is us…

Read more

If you have readily followed me on LinkedIn, you will notice from time to time, I drop some nuggets on my rowing journey. I would not consider myself an “elite” rower by any means, but I have logged well over 1M meters now on my Concept 2 rower and have a pretty good clue of what direction is up. Plus, the few times that I have ranked my workouts in the…

Read more

It’s been over a month since I last wrote an article. I’ve been on a bit of a hiatus, but I have some good things coming down the pipeline. Today though, we will be looking at the current state of AWS S3 Tables and Duckdb.

If you rewind the clock a few months back, Duckdb had a monumental release (1.4) which came with the ability to write to Iceberg tabl…

Read more

I’ve been on a spark config kick lately. A week or so ago, I published a post showing how to config Spark for the AWS Glue Iceberg Rest endpoint. Overall, the config is not too complex, but does require a ton of inputs. Comparing that to the duck, you see a stark contrast.

However, after working through that article, I was somewhat disappointed that the …

Read more

Duckdb has been making some major inroads over the past few months to advance their integrations with Iceberg and AWS. One can simply run the following now to map any iceberg tables from an AWS account into duckdb (no fancy s3 tables required):

Read more

Read more

Read more

I first learned about window functions back in my SQL Server days; to be exact, it was on the SQL Server 2012 edition when I was an analyst in finance; up until that day, whenever I had to build forecasts and/or budgets, I’d query the data from SQL Server, dump it in Excel, and then build a bunch of excel calcs there to figure out moving averages and Yo…

Read more

I think I’ve finally cracked a part of the LLM secret sauce; it took me several months of trial/error to get here, but now that I have gotten over this hard paradigm shift, I wanted to share my success with you all so that you can learn and build on it.

What the gist of this really boils down to is the following:

Read more

We’ve all been there before. We inherit someone’s sql query, we crack it open, and we see a long winded case when statement that uses the like operator to categorize data. Something ugly like this:

Read more

Throughout my career, I’ve had to at times, pull government and/or public datasets for analytics. As an example, when I worked in transportation and logistics as an analyst, each month we would go to the department of energy website and pull in fuel forecasts to help us understand where our budget would potentially land in case the price of fuel went u…

Read more

I’ve shared a few times on LinkedIn throughout the past several months that I’m an avid rower on my concept 2 rowerg; I recently (May 2025) broke a major milestone of 1M meters. This took about 13 months to get to, but throughout the journey, I learned a ton about rowing and how to hone in my technique. With that said, the Concept 2 logbook is a free se…

Read more

Throughout my career, I’ve seen many evolutions of how we, as data engineers, “ship” pipelines to production. Some of the common methods are through:

• CICD Pipelines in Gitlab/Github

• Emailing the script to the DBA to run (No one will ever admit this though 😆)

• Saving bash jobs to a shared drive and using cron to run them (or windows task scheduler)

• Airflow+S…

Read more

Full Disclosure - I was able to “vibe” roughly 95% of this entire article and script via ChatGPT with some careful prompting. I did this more as an experiment and don’t plan on having chatgpt draft my substack articles for the foreseeable future. But I was also getting bored with my writing style and wanted some inspiration. If chat gippity’s overblown …

Read more

I truly sorely miss the days I was a cowboy running wild on SQL Servers. A little bit of sp_executesql here, a little bit of xp_cmdshell there 😆…Needless to say, I owe a lot of my early career and resume building to Microsoft in this regard. I started off with SQL Server 2005 running under a desk at my office, but had over time graduated to IT supporte…

Read more