Datasette enrichments

Datasette Enrichments: a new plugin framework for augmenting your data introduces a new enrichments framework for Datasette.

An enrichment is code that can be executed against every row (or a filtered subset of rows) in a table, modifying or enhancing that data in some way - including importing new data from external APIs.

One example is geocoding: populating latitude and longitude columns based on the contents of an address column, using an external geocoder such as OpenCage:

Enrichments are provided by plugins. There are four plugins available today:

• datasette-enrichments-opencage - geocoding (and reverse-geocoding) using OpenCage

• datasette-enrichments-gpt - run prompts against text and images through OpenAI's GPT-3.5 and GPT-4 APIs, writing the results back to a column

• datasette-enrichments-jinja - use sandboxed Jinja templates to populate new columns based on existing data

• datasette-enrichments-re2 - execute regular expressions for search-and-replace or to extract data from text

Here's a video demonstrating the new feature, including how to use GPT-4 to extract structured data from unstructured text and how GPT-4 Vision can be used to provide detailed descriptions of linked images.

There are lots more details in the blog post announcement, including how to write your own enrichments.

Datasette comments

Alex Garcia has been working with me on Datasette Cloud, generously sponsered by Fly.io.

Alex’s latest project is datasette-comments, an open source plugin that enabling a new way of collaborating around data analysis. He introduces the new plugin in Annotate and explore your data with datasette-comments on the Datasette Cloud blog.

Authenticated users can attach and reply to comments on any row of data in any table. Comments support @-mentions, reaction emoji and hashtags, and a comment thread can be marked as resolved to move it to an archive page.

Datasette Cloud now has enrichments and comments

We’ve enabled both of these plugins on Datasette Cloud. If your team would benefit from collaboratively analyzing data in this way, please drop us a line by replying to this message!

Datasette Cloud

Datasette Cloud is the hosted SaaS version of Datasette. It exists to solve several problems:

• I want teams to be able to use Datasette to collaborate on finding stories in their data - initially targeting newsrooms and data journalism, but the software is very clearly applicable to many other fields as well. Asking people to install software on their own server is a major barrier to entry, and I want to remove that barrier.

• I'd like the project to be financially sustainable - not just to cover my own costs, but I want to grow a team of people to work full-time on Datasette and associated projects.

• I want to make it as easy as possible for people to publish their data online. Datasette is great for this already... if you know how to use it (with Fly, Vercel, Cloud Run or similar). Datasette Cloud aims to make that even easier.

I wrote more about the goals of Datasette Cloud in the first post on the new Datasette Cloud blog: Welcome to Datasette Cloud.

The Datasette Cloud effort is partially sponsored by Fly.io, who are funding Alex Garcia to work with me on the project. Alex's first piece of work for that is the new datasette-write-ui, built initially for Datasette Cloud but available as an open source plugin for anyone to use. Alex introduced that in datasette-write-ui: a Datasette plugin for editing, inserting, and deleting rows on the Datasette Cloud blog.

If you want to try out Datasette Cloud you can request access to the preview today.

Datasette 1.0 alphas

Datasette 1.0 will be the version of Datasette that promises stability, in terms of the JSON API, the template context (so you can build custom templates without fear that they will break in future minor releases) and the API for plugins.

There have been four releases in the 1.0 alpha series so far:

• 1.0a0 introduced signed API tokens and the Datasette write JSON API.

• 1.0a1 expanded the write API with CORS support and other small features, and was accompanied by this blog entry showing detailed examples of what the write API can do.

• 1.0a2 added upsert support, and a mechanism for creating finely-grained access tokens. More in this blog post.

• 1.0a3 features a new default JSON output, one of the most significant milestones on the way to 1.0 final.

• 1.0a4 fixes a security vulnerability in the 1.0 alpha series, described next.

A vulnerability in the 1.0 alphas

I released 1.0a4 this morning, which fixes a security vulnerability in the 1.0 alpha series.

This affects you if you running a Datasette 1.0 alpha instance on a public server.

The bug is that an API explorer interface within Datasette (added in 1.0a0) could be visited by unauthenticated users and would reveal the names of the databases and tables in that instance - though not any of the actual table content.

You should upgrade as quickly as possible if:

• You are running a Datasette 1.0 alpha instance on the public internet

• That Datasette instance is authenticated using a plugin such as datasette-auth-passwords

• There are private databases and tables in that instance where the names of those databases and tables should be considered sensitive information.

For more information read this GitHub security advisory: Datasette 1.0 alpha series leaks names of databases and tables to unauthenticated users.

The vulnerability was present on Datasette Cloud but has now been patched there, and a review of our logs showed that no unauthorized users had accessed that page across any of our hosted instances.

LLM: a new Datasette project

I've spent a lot of time over the past year immersed in research and exploration of the weird new world of LLMs - Large Language Models, the technology behind ChatGPT and GPT-4 and Bard and Bing and Claude and Llama and other similar projects.

Catching up on the weird world of LLMs is a 40m talk (plus detailed transcript, notes and links) I gave at North Bay Python in August which attempts to summarize everything I've learned about LLMs so far.

I'm increasingly optimistic about the role LLMs can play in the Datasette world, especially when combined with Datasette's plugin architecture.

You can read about some of my early explorations of LLMs and Datasette here:

• How to implement Q&A against your documentation with GPT3, embeddings and Datasette describes my earliest attempt at building retrieval-augmented generation (RAG) against data in Datasette, where you take a user's question, find relevant content, then feed that to the LLM with "Given the above context, answer the following question: ..." appended to it.

• I built a ChatGPT plugin to answer questions about data hosted in Datasette talks about my implementation of a plugin for ChatGPT that turns English questions into SQL queries and lets ChatGPT run those against Datasette and use the results to generate an answer.

• Enriching data with GPT3.5 and SQLite SQL functions shows how my openai-to-sqlite CLI tool can run enrichments such as sentiment analysis against data in a SQLite database and store the results.

• Storing and serving related documents with openai-to-sqlite and embeddings shows how to use the same tool to store embeddings for documents in a SQLite database, then use those embeddings to identify related documents, save those to a table and serve them up as a "Related" section on my TIL site.

These explorations have added up to a new project called LLM, a new command-line utility and Python library for working with LLMs.

The LLM project site is llm.datasette.io.

LLM is heavily inspired by both Datasette and sqlite-utils. It stores prompts and their responses to SQLite, providing you with a permanent structured log of your LLM interactions. It also provides a plugin architecture for adding new functionality.

Most importantly, the plugin architecture lets you add support for alternative models - including models like Llama 2 which you can run on your own machine!

The simplest version of LLM usage looks like this.

First, install it:

pip install llm

Or use pipx or Homebrew:

brew install llm

If you have an OpenAI API key you can configure that and start using LLM immediately:

llm keys set openai
<paste key here>

Then run a prompt:

llm 'Five nautical names for a pet pelican'

But where things get really fun is when you start adding LLM plugins to install and run alternative models on your own machine.

llm install llm-gpt4all
llm -m orca-mini-7b '3 names for a pet cow'

The first time you run this it will download the orca-mini-7b model from the GPT4All project and use it to run the prompt.

LLM has eight plugins already. My current favorite is llm-mlc, which uses the MLC framework to run models like Llama 2 using GPU acceleration on my M2 Mac laptop.

For more about LLM, read these blog entries:

• llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs

• The LLM CLI tool now supports self-hosted language models via plugins

• Accessing Llama 2 from the command-line with the llm-replicate plugin

• Run Llama 2 on your own Mac using LLM and Homebrew

Other Datasette news

The Datasette News page has short-form news about the project. The datasette tag on my blog is a lot noisier and more frequently updated.

I plan to send this newsletter out more often, especially given the increased pace of development towards 1.0.

Datasette Lite provides a new way to run Datasette: entirely in your browser, thanks to WebAssembly and the Pyodide project.

Visit https://lite.datasette.io/ to try it out.

You can also pass it the URL to a SQLite database, CSV file or SQL initialization script. Here are some examples:

• https://lite.datasette.io/?url=https://congress-legislators.datasettes.com/legislators.db loads a database with historic Members of the United States Congress.

• https://lite.datasette.io/?csv=https%3A%2F%2Fraw.githubusercontent.com%2Ffivethirtyeight%2Fdata%2Fmaster%2Ffight-songs%2Ffight-songs.csv loads US college fight songs from this CSV file in the FiveThirtyEight GitHub repository.

Read more about Datasette Lite in this series of articles on my blog:

• Datasette Lite: a server-side Python web application running in a browser

• Joining CSV files in your browser using Datasette Lite

• Plugin support for Datasette Lite

• Sort by number of JSON intersections, which shows how Datasette Lite can be used as a tool to demonstrate the answer to questions about how to do something using SQL

Datasette Tutorials

The official Datasette website now features tutorials to help people get started using Datasette:

• Exploring a database with Datasette shows how to use the Datasette web interface to explore a new database.

• Learn SQL with Datasette introduces SQL, and shows how to use that query language to ask questions of your data.

• Cleaning data with sqlite-utils and Datasette shows how to use sqlite-utils to clean data, import it into SQLite and then explore it with Datasette. Includes a ten minute accompanying video.

Sign up for the preview of Datasette Cloud

Datasette Cloud is the new hosted SaaS service I am building to help teams run a private Datasette instance to share and collaborate on data analysis. It's nearly ready for preview users - if you'd like to try it out, please add yourself to the waiting list using the form on the homepage.

Datasette Cloud features include:

• Upload CSVs and import data from external sources to populate your own, private Datasette instance

• Invite coworkers to collaborate in your team's own dedicated space

• Build search engines against your data

• All of Datasette's standard features: filter and facet your data, execute SQL queries, generate visualizations and more

Community highlights

Philip James has been using Datasette to publish searchable versions of the minutes of city meetings for the cities of Alameda and Oakland:

• https://data.oakland.works/oakland_minutes/pages lists 37,000 pages of Oakland civic meeting minutes dating back to 2003 - see Philip’s Twitter thread for details

• https://data.alameda.one/alameda_minutes/pages provides 18,000 pages of Alameda city minutes - thread here

Alex Garcia has published a series of interactive posts about writing SQLite extensions:

• Introducing sqlite-lines - a SQLite extension for reading files line-by-line

• Introducing sqlite-html: query, parse, and generate HTML in SQLite

• Introducing sqlite-http: A SQLite extension for making HTTP requests

Each post includes interactive demos powered Observable notebooks, some of which work by talking to a Datasette backend using Alex’s Datasette Client Observable integration.

Jim Crist-Harif released Ibis-Datasette, a Python library that lets you build SQL queries using a SQLAlchemy style chained query builder syntax, run them against a Datasette instance and get back the results in a DataFrame-like object.

In [1]: import ibis
In [2]: ibis.options.interactive = True
In [3]: con = ibis.datasette.connect(
   ...:    "https://congress-legislators.datasettes.com/legislators"
   ...: )
In [4]: legislators = con.tables["legislators"]
In [5]: legislators.groupby("bio_gender").count()
Out[5]:
  bio_gender  count
0          F    399
1          M  12195

Datasette Discord

The Datasette project now has an official presence on Discord! Join our Discord server here to talk about Datasette and the other projects that make up the wider Datasette ecosystem.

Join Datasette Discord

An ongoing goal I have for Datasette is to make it available for as many people to use as possible.

I've tried a bunch of different strategies for this in the past:

• Datasette is built using Python, so if you have a working Python environment you can install it using pip install datasette

• I packaged it for Homebrew, so brew install datasette on macOS should work too

• I have documentation for running it online using Glitch, where it can be installed using just your web browser

• The datasette publish set of commands are designed to help deploy Datasette to Cloud Run, Heroku, Vercel or fly.io as easily as possible

Unfortunately, each of these options require some degree of deep technical knowledge before you can use them - familiarity with the command-line, or knowledge of how to deploy websites. These things are a significant barrier to entry!

My goal with Datasette Desktop is to make installing Datasette as easy as downloading and installing any other Mac application.

You can try that out here: click the Download link, open the zip file and drag the Datasette application to your /Applications folder.

The application comes with its own copy of Python 3.9 tucked away inside it, which means that even if you don’t have Python installed it will still be able to run just fine. It also pulls off some tricks to ensure that existing Datasett plugins can be installed using the “Plugins → Install and Manage Plugins…” menu option.

Once installed, the application can be used to open existing CSV or SQLite files on your computer. It can also open CSV files by URL, and uses that capability to offer examples that you can open to try it out (currently using data from the Central Park Squirrel Census and the London Fire Brigade’s list of animal rescue incidents).

How I built it

Like Datasette itself, Datasette Desktop is open source. I’ve been building it entirely in the open in the simonw/datasette-app GitHub repository.

I’ve been writing up details of how it works as I’ve gone along:

• Building a desktop application for Datasette describes the initial research into the project, and talks about how I’m building it using Electron.

• Datasette Desktop—a macOS desktop application for Datasette was the initial launch announcement for the first installable version. It describes how the application works in some detail, including how I bundled Python inside the application and how I figured out signing and distribution of macOS apps using Electron.

• Datasette Desktop 0.2.0: The annotated release notes describes the second release, which introduced some significant new features including a plugin management interface for installing, upgrading and uninstalling plugins and the ability to open SQLite and CSV files directly using the application.

I’ve also been collecting numerous TILs (Today I Learned) about Electron.

Feedback welcome

The application so far is just a starting point: I’m very keen to hear from people who have tried it out. Please leave questions, suggestions and feedback on the GitHub Discussions forum for the project.

What’s new in Datasette

Datasette 0.55 added support for cross-database SQL queries: you can now start Datasette with the new “--crossdb” option which enables cross-database joins! More details in the documentation or you can try out an example cross-database query here.

0.56 was mainly small bug fixes and documentation improvements, though it did also introduce a handle you can drag to resize the SQL editor.

0.57 brought a critical security fix - if you have any Datasette instances running on the public web you should upgrade to a version higher than this (or 0.56.1 which applies the same fix to 0.56) as soon as possible.

It also fixed a long-standing issue where SQL errors were displayed without letting you edit your original SQL, and added mechanisms for selectively showing and hiding columns on a table page (“?_col=” and “?_nocol”).

0.58 added Unix domain socket support, useful for people running Datasette behind an Apache or NGINX proxy server. It also added two new plugin hooks and a significant performance boost for faceting - all of which are covered in detail in the Datasette 0.58 annotated release notes.

What’s new in sqlite-utils

Datasette’s close companion is sqlite-utils - a combination CLI tool and Python library for creating and manipulating SQLite databases.

sqlite-utils gained two key new features this year.

sqlite-utils memory is a new command that lets you pipe JSON and CSV data directly into a temporary, in-memory SQLite database, execute a SQL query that filters or joins that data, then returns the results as tabular, CSV or JSON output.

It turns sqlite-utils into an ad-hoc querying tool that works against all sorts of data, and can be used to combine data from different sources - effectively running joins between JSON, CSV and other data sources.

I recorded a YouTube video showing the new tool in action:

Here’s an example that fetches JSON from the GitHub API, executes a SQL query to select three columns and sort by the number of stars, then outputs the result to the console as CSV:

$ curl -s 'https://api.github.com/users/dogsheep/repos' \
  | sqlite-utils memory - '
      select full_name, forks_count, stargazers_count as stars
      from stdin order by stars desc limit 3
    ' --csv
full_name,forks_count,stars
dogsheep/twitter-to-sqlite,12,225
dogsheep/github-to-sqlite,14,139
dogsheep/dogsheep-photos,5,116

I wrote more about this in Joining CSV and JSON data with an in-memory SQLite database.

The sqlite-utils convert command is the other big new feature: it lets you apply a conversion function written in Python to every value in a column in your SQLite database.

Here’s how to remove commas from a column with 123,456 style numbers:

sqlite-utils convert states.db states count \
    'value.replace(",", "")'

And this will split a “location” column into two separate “latitude” and “longitude” columns (using the --multi option):

sqlite-utils convert data.db places location '
latitude, longitude = value.split(",")
return {
    "latitude": float(latitude),
    "longitude": float(longitude),
}' --multi

Read Apply conversion functions to data in SQLite columns with the sqlite-utils CLI tool for more.

Django SQL Dashboard

I spent the first few months of the year mainly not working on Datasette at all: I joined the team running VaccinateCA / Vaccinate The States to build a Django application to help crowdsource information about places that people could go to get vaccinated against COVID-19 in the USA.

My first order of business was to help replace their Airtable infrastructure - which was rapidly reaching the size limits of what could be kept in Airtable - with a Django equivalent.

Since a major benefit of building on Airtable was its ad-hoc reporting abilities, I found myself really needing something similar to Datasette that could work against a PostgreSQL database.

So I built exactly that: Django SQL Dashboard - an open source package that adds a Datasette-like querying interface to a Django+PostgreSQL project, protected by the Django authentication layer.

I recorded this video with a demo of the project:

Lots more detail in the Django SQL Dashboard launch announcement.

I’m still deciding if and when to bring the lessons from that project back into Datasette itself - having Datasette work against PostgreSQL in addition to SQLite would open up a huge new set of possibilities for the project, and my work on Django SQL Dashboard has helped me start to scope out how much work it would be to make that a reality.

If you’re interested in reading more about my work at VaccinateCA I’ve imported my writing from my internal blog at the organization into my regular blog - you can find that series of posts here: simonwillison.net/series/vaccinateca

New Datasette Plugins

I decided to check how many new Datasette plugins I’ve released since the last newsletter on January 26th. The answer turned out to be seven! Here’s the query I used:

select
  repos.full_name,
  tag_name,
  min(releases.created_at) as launched_at
from
  releases
  join repos on releases.repo = repos.id
where
  repos.full_name like 'simonw/datasette-%'
  and repos.full_name not in (
  -- These are older but the first tagged release was this year
    'simonw/datasette-jellyfish',
    'simonw/datasette-haversine'
  )
group by
  repo
having
  launched_at > '2021-01-26'
order by
  launched_at

Run that here.

datasette-tiles is a plugin for serving map tiles directly out of a SQLite database, using the MBTiles standard for bundling tiles in a DB file.

datasette-basemap is a related plugin that bundles the first four layers of tiles from OpenStreetMap in a SQLite database file which can be installed in a place where Datasette can find it using “pip install datasette-basemap”.

Here’s a demo of the plugin, or you can browse the underlying database of tile images.

I wrote more about these plugins in Serving map tiles from SQLite with MBTiles and datasette-tiles.

datasette-block is a plugin that lets you block all access to specific path prefixes within Datasette. You probably don’t need this one!

datasette-placekey adds SQL functions for working with the placekey mechanism for creating identifiers for locations. I built this for some analysis at VaccinateCA.

datasette-remote-metadata lets you define your Datasette metadata in an external file and have Datasette fetch it on-demand while it is running. This is useful for if you have a large (1GB+) database running somewhere like Cloud Run where deploying new metadata can take several minutes - with the plugin you can edit the remote file and have the changes go live a few seconds later.

I built this as part of working with Stanford’s Big Local News to help launch the Stanford School Enrollment Project, which makes available detailed school enrollment figures for schools and districts across the USA from this year back to 2015.

datasette-pyinstrument adds the ability to add ?_pyinstrument=1 to any Datasette URL in order to see the output of a profile run while constructing the page.

datasette-query-links is a highly experimental plugin which looks out for SQL queries that return strings that are themselves valid SQL queries, and turns those strings into links that will execute the SQL. I thought this was a neat and unique idea, until I found out that PostgreSQL already has a version of this as a built in feature called \gexec!

What’s next

I’ve decided to try and get this newsletter back onto a roughly monthly cadence. In the meantime, I suggest keeping an eye on the official Datasette website at datasette.io - you can even subscribe to its Atom feeds!

And a reminder: I’m still running Datasette Office Hours every Friday, so if you’d like to have a 25 minute video chat with me about Datasette to talk about things you’re working on or provide feedback on the project, sign up for a slot!

datasette-leaflet-freedraw

Datasette supports parameterized SQL queries: if you compose a SQL query with a placeholder such as “select * from counties where population > :number” Datasette will spot the :number parameter and turn it into a form field on a page:

I’ve been planning to add custom form field widgets for a while. datasette-leaflet-freedraw is a new plugin which starts exploring this concept, by enhancing any fields ending in “freedraw” with a map widget that lets users draw a shape, which will be converted to a GeoJSON MultiPolygon ready to be sent back to Datasette.

Here’s an interactive demo:

You can read more about this plugin in Drawing shapes on a map to query a SpatiaLite database.

Datasette 0.54

I released Datasette 0.54 yesterday. You can read the annotated release notes on my blog, but the two two big new features in this release are the _internal SQLite in-memory database storing details of all connected databases and tables, and support for JavaScript modules in plugins and additional scripts.

The _internal database

I want Database to be able to handle dozens of attached database files containing potentially thousands of tables. To support this, the Datasette homepage will need to get a lot smarter: it needs to provide pagination and search and filtering against the list of tables themselves.

Datasette is already pretty good at providing paginated search and filtering! The new _internal database is the first step to providing this feature: it's an in-memory database which is automatically updated with details of all of the attached databases, plus their columns, foreign keys and indexes.

As part of building this I added support for shared, named in-memory databases - so plugins can now create an in-memory database that will persist for the lifetime of the Datasette process and can be accessed from multiple connections.

You can browse a demo of the _internal database by signing into the demo as root and then accessing latest.datasette.io/_internal

JavaScript modules

JavaScript modules are supported by modern browsers and allow you to use the “import” and “export” keywords from ECMAScript 2015 without needing a build step.

They only work in <script> blocks with the type="module" attribute, so Datasette now has the ability to include scripts with that attribute, for both plugins and extra scripts loaded through configuration.

A big benefit of native modules is that they help avoid accidentally loading the same script twice. There are three Datasette plugins that use Leaflet maps now - datasette-cluster-map, datasette-leaflet-geojson and the new datasette-leaflet-freedraw - so I've started working on a new base plugin called datasette-leaflet which makes the latest Leaflet available for all of them to use via the modules system.

datasette-css-properties and datasette-export-notebook

Two more new plugins: these use Datasette's register_output_renderer hook to add extra options for getting data back out of Datasette.

I wrote about datasette-css-properties here - it's a very weird plugin! It was inspired by Custom Properties as State by Chris Coyier, which proposed server APIs that return CSS files defining custom properties that can be used to style pages.

I'm not sure if this is a good idea, but the joy of plugins is that you can try out ideas like this without risking damage to the integrity of the core project!

datasette-export-notebook is more obviously useful: it adds instructions for exporting data from Datasette into Jupyter and Observable notebooks. You can try that out here against the Covid-19 Datasette.

Datasette Office Hours

I've now had 17 conversations with people about Datasette in office hours sessions. It's been absolutely wonderful! Talking to people about what they're building, or what they might build, is incredibly valuable for helping me decide what to work on next, and it's also just really fun talking to people about the project.

I plan to continue running office hours sessions on Friday for the forseeable future. If you'd like to chat about the project you can grab a slot with me here.

datasette.io is the new home for Datasette. The site features news about the project, explains the problems that Datasette and its ecosystem of tools try to solve and gathers together a complete directory of plugins and tools for using with Datasette.

The site itself is built using Datasette - it’s actually a heavily themed installation of Datasette, making extensive use of the datasette-template-sql plugin and custom pages to implement the different sections of the site. The code for the site is available on GitHub at simonw/datasette.io.

I wrote more about how the site works in datasette.io, an official project website for Datasette.

Adding search to datasette.io

datasette.io also includes a search engine for the project.

It provides faceted search across:

• Every section of the latest documentation (416 total)

• 48 plugin READMEs

• 22 tool READMEs

• 64 news items posted on the Datasette website

• 212 items from my blog

• Release notes from 557 package releases

That’s 1,319 items total! The overall Datasette project continues to grow in all kinds of directions, so having a single search engine to tie it all together felt like a useful addition.

I built the search engine using Dogsheep Beta, the cross-table SQLite search engine I originally built for my Dogsheep Personal Analytics project.

You can read more about how the search engine works, including a breakdown of the YAML configuration used to create the unified search index, in Building a search engine for datasette.io.

sqlite-utils analyze-tables

I released sqlite-utils 3.1 with a useful feature for getting to grips with new data: the “sqlite-utils analyze-tables” command.

Run this against any SQLite database and it will spit out a summary of every column in every table, showing you common and least-common values, the number of distinct items and how many rows contain nulls or blanks.

The analyze-tables documentation has more. In addition to outputting the summary to the command-line you can add the “--save” option to store the generated summary in a SQLite table called _analyze_tables_ - which means you can then further examine it using Datasette.

Here’s an example of the output when running analyze-tables against a database created using my github-to-sqlite tool.

Office hours for 2021

Every Friday I run Datasette Office Hours sessions, where people who use Datasette (or who are interested in using it) can grab a 20 minute video chat with me to talk about the project. I’ve done a couple of weeks of these now and they’ve been fantastic - it’s so interesting hearing how people are using or plan-to-use the tool, and I’ve already had a bunch of great feedback to help me plan the next steps for the project.

If you’re interested in talking to me, slots in January 2021 are available now!

Datasette Office Hours

One of the toughest things about running open source projects is the challenge of getting feedback: users often assume they are doing you a favor by leaving you alone, so generally you only hear from people if they find a bug.

I’m always really keen to talk to people who are using Datasette - or who aren’t using it yet, since I want to understand what it’s missing.

So I’m going to try a new approach: I’m setting aside time every Friday for Datasette Office Hours, where anyone can book a 20 minute Zoom call with me to talk about the project.

I’d love to hear from you if:

• You’re solving problems with Datasette

• You have a problem you think Datasette might be able to help with

• You’ve run into problems using Datasette

• You have thoughts on the direction you think the project should go

• You’d like to see some demos of things I’m working on

• You’re just interested in having a chat!

You can sign up for office hours at calendly.com/swillison/datasette-office-hours

Personal Data Warehouses

A gave a talk for the GitHub OCTO Speaker Series a couple of weeks ago about my Datasette and Dogsheep projects called Personal Data Warehouses: Reclaiming Your Data.

GitHub shared the video on their YouTube channel, and I’ve prepared an extended, annotated version of the talk with additional screenshots, links and notes.

The talk shows how I built my own personal data warehouse on top of Datasette that imports, analyzes and visualizes my data from Twitter, Swarm, GitHub, 23AndMe, Apple HealthKit, Apple Photos and more.

datasette-ripgrep

datasette-ripgrep is a web application I built for running regular expression searches against source code, built on top of the amazing ripgrep command-line tool.

Here are some example searches, running across around 100 Datasette and Datasette- adjacent GitHub repositories:

• with.*AsyncClient - regular expression search for with.*AsyncClient

• .plugin_config, literal=on - a non-regular expression search for .plugin_config(

• with.*AsyncClient glob=datasette/** - search for that pattern only within the datasette/ top folder

• "sqlite-utils[">] glob=setup.py - a regular expression search for packages that depend on either sqlite-utils or sqlite-utils>=some-version

• test glob=!*.html - search for the string test but exclude results in HTML files

I wrote about the project on my blog, in datasette-ripgrep: deploy a regular expression search engine for your source code

It’s an interesting use-case for Datasette in that it doesn’t use SQLite at all - the tool works by running the “rg” executable against a folder full of source code. It does benefit from Datasette’s “datasette publish” mechanism - the following one-liner will deploy a Datasette instance to Google Cloud Run pre-configured to run searches against everything in the “all/“ directory, which is uploaded as part of the deployment:

datasette publish cloudrun \
    --metadata metadata.json \
    --static all:all \
    --install=datasette-ripgrep \
    --service datasette-ripgrep \
    --apt-get-install ripgrep

The official demo is deployed by this GitHub Actions workflow which pulls a list of repos using github-to-sqlite, filters down to just the ones I want to include in the demo, then deploys them using the above pattern.

datasette-indieauth

My other big plugin project this month was datasette-indieauth, an authentication plugin which adds support for the emerging IndieAuth single sign-on standard.

You can read more about this project in Implementing IndieAuth for Datasette on my blog. IndieAuth is a spiritual successor to OpenID which allows users to sign-in using a website address that they control. It’s a particularly good fit for Datasette as it allows you to deploy single sign-on without first registering your site with a central authority.

Here’s an animation showing what the experience looks like signing in to the official demo at datasette-indieauth-demo.datasette.io/-/indieauth:

Datasette Client for Observable

Alex Garcia built a beautiful JavaScript client library for interacting with data hosted by Datasette from Observable notebooks. His demo at observablehq.com/@asg017/datasette-client shows how to use the client and demonstrates it integrating with Observable’s form elements and visualizing data as a stacked area chart using D3.

Other significant releases this month

Datasette 0.52 (and 0.52.1) - a relatively small release, this adds a new database_actions(datasette, actor, database) plugin hook and renames the --config option to --setting - --config still works but shows a deprecation message, and will be removed in Datasette 1.0.

github-to-sqlite 2.8 adds a new “github-to-sqlite workflows” command which imports GitHub Actions workflow YAML flies and uses them to populate new workflows, jobs and steps tables.

datasette-graphql 1.2 and 1.3 add support for the Datasette view-instance permission and view-database permissions, and use the new table actions plugin hook to add example GraphQL queries to the cog action menu on every table page. You can try that out against this commits table created using github-to-sqlite.

sqlite-utils 3.0 adds a new command-line tool and Python method for executing full-text searches against a table, and returning the results ordered by relevance. It also adds a new --tsv output option and makes some small changes to other command-line options, hence the 3.0 major version bump.

(You may have seen these annotated release notes on my blog - if so, feel free to skip straight to plugin of the week.)

Datasette 0.51, the annotated release notes

I shipped Datasette 0.51 at the weekend, with a new visual design, plugin hooks for adding navigation options, better handling of binary data, URL building utility methods and better support for running Datasette behind a proxy. It’s a lot of stuff! Here are the annotated release notes.

New visual design

Datasette is no longer white and grey with blue and purple links! Natalie Downe has been working on a visual refresh, the first iteration of which is included in this release. (#1056)

It’s about time Datasette grew beyond its clearly-designed-by-a-mostly-backend-engineer roots. Natalie has been helping me start adding some visual polish: we’ve started with an update to the colour scheme and will be continuing to iterate on the visual design as the project evolves towards the 1.0 release.

The new design makes the navigation bar much more obvious, which is important for this release since the new navigation menu (tucked away behind a three-bar icon) is a key new feature.

Plugins can now add links within Datasette

A number of existing Datasette plugins add new pages to the Datasette interface, providig tools for things like uploading CSVs, editing table schemas or configuring full-text search.

Plugins like this can now link to themselves from other parts of Datasette interface. The menu_links(datasette, actor) hook (#1064) lets plugins add links to Datasette’s new top-right application menu, and the table_actions(datasette, actor, database, table) hook (#1066) adds links to a new “table actions” menu on the table page.

This feature has been a long time coming. I’ve been writing an increasing number of plugins that add new pages to Datasette, and so far the main way of using them has been to memorise and type in their URLs!

The new navigation menu (which only displays if it has something in it) provides a global location to add new links. I’ve already released several plugin updates that take advantage of this.

The new “table actions” menu imitates Datasette’s existing column header menu icon—it’s a cog. Clicking it opens a menu of actions relating to the current table.

Want to see a demo?

The demo at latest.datasette.io now includes some example plugins. To see the new table actions menu first sign into that demo as root and then visit the facetable table to see the new cog icon menu at the top of the page.

Here’s an animated GIF demo showing the new menus in action.

Binary data

SQLite tables can contain binary data in BLOB columns. Datasette now provides links for users to download this data directly from Datasette, and uses those links to make binary data available from CSV exports. See Binary data for more details. (#1036 and #1034).

I spent a ton of time on this over the past few weeks. The initial impetus was a realization that Datasette CSV exports included ugly Python b'\x15\x1c\x02\xc7\xad\x05\xfe' strings, which felt like the worst possible way to display binary in a CSV file, out of universally bad options.

Datasette’s main interface punted on binary entirely—it would show a <Binary data: 7 bytes> label which didn’t help much either.

The only way to get at binary data stored in a Datasette instance was to request the JSON version and then manually decode the Base-64 value within it!

This is now fixed: binary columns can be downloaded directly to your computer, using a new .blob output renderer. The approach is described on this new page in the documentation.

Security was a major consideration when building this feature. Allowing the download of arbitrary byte payloads from a web server is dangerous business: it can easily result in XSS holes where HTML with dangerous <script> content can end up hosted on the primary domain.

After some research, I decided to serve up binary content for download using the following headings:

content-type: application/binary x-content-type-options: nosniff content-disposition: attachment; filename="data-f30889.blob" 

application/binary is a safer Content-Type option than the more common application/octet-stream, according to Michal Zalewski’s renowned web application security book The Tangled Web (quoted here)

x-content-type-options: nosniff disables the XSS-tastic content sniffing feature in older versions of Internet Explorer, where IE would helpfully guess that you intended to serve HTML based on the first few bytes of the response.

The content-disposition: attachment header causes the browser to show a “download this file” dialog, using the suggested filename.

If you know of a reason that this isn’t secure enough, please let me know!

URL building

The new datasette.urls family of methods can be used to generate URLs to key pages within the Datasette interface, both within custom templates and Datasette plugins. See Building URLs within plugins for more details. (#904)

Datasette’s base_url configuration setting was the forcing factor around this piece of work.

It allows you to configure Datasette to serve content starting at a path other than /—for example:

datasette --config base_url:/path-to-datasette/ 

This will serve all Datasette pages at locations starting with /path-to-datasette/.

Why would you want to do this? It’s useful if you are proxying traffic to Datasette from within the URL hierarchy of an existing website.

The feature didn’t work properly, and enough people care about it that I had a steady stream of bug reports. For 0.51 I gathered them all into a single giant tracking issue and worked through them all one by one.

It quickly became apparent that the key challenge was building URLs within Datasette—not just within HTML template pages, but also for things like HTTP redirects.

Datasette itself needed to generate URLs that took the base_url setting into account, but so do Datasette plugins. So I built a new datasette.urls collection of helper methods and made them part of the documented internals API for plugins. The Building URLs within plugins documentation shows how these should be used.

I also added documentation on Running Datasette behind a proxy with example configs (tested on my laptop) for both nginx and Apache.

The datasette.client mechanism from Datasette 0.50 allows plugins to make calls to Datasette’s internal JSON API without the overhead of an HTTP request. This is another place where plugins need to be able to construct valid URLs to internal Datasette pages.

I added this example to the documentation showing how the two features can work together:

table_json = (     await datasette.client.get(         datasette.urls.table("fixtures", "facetable", format="json")     ) ).json()

One final weird detail on this: Datasette now has various methods that automatically add the base_url prefix to a URL. I got worried about what would happen if these were applied more than once (as above, where datasette.urls.table() applies the prefix so does datasette.client.get()).

I fixed this using the same trick that Django and Jinja use to avoid appliying auto-escaping twice to content that will be displayed in HTML: the datasette.urls methods actually return a PrefixedUrlString object which is a subclass of str that knows that the prefix has been applied! Code for that lives here.

Smaller changes

A few highlights from the “smaller changes” in Datasette 0.51:

• Wide tables shown within Datasette now scroll horizontally (#998). This is achieved using a new <div class="table-wrapper"> element which may impact the implementation of some plugins (for example this change to datasette-cluster-map).

I think this is a big improvement: if your database table is too wide, it now scrolls horizontally on the page (rather than blowing the entire page out to a wider width). You can see that in action on the global-power-plants demo.

• New debug-menu permission. (#1068)

If you are signed in as root the new navigation menu links to a whole plethora of previously-undiscoverable Datasette debugging tools. This new permission controls the display of those items.

• Link: HTTP header pagination. (#1014)

Inspired by GitHub and WordPress, which both use the HTTP Link header in this way. It’s an optional extra though: Datasette will always offer in-JSON pagination information.

• Edit SQL button on canned queries, (#1019)

Suggested by Jacob Fenton in this issue. The implementation had quite a few edge cases since there are certain categories of canned query that can’t be executed as custom SQL by the user. See the issue comments for details and a demo.

• --load-extension=spatialite shortcut. (#1028)

Inspired by a similar feature in sqlite-utils.

• datasette -o option now opens the most relevant page. (#976)

This is a fun little feature. If your Datasette only loads a single database, and that database only has a single table (common if you’ve just run a single CSV import) then running this will open your browser directly to that table page:

datasette data.db -o 

• datasette --cors option now enables access to /database.db downloads. (#1057)

This was inspired by Mike Bostock’s Observable Notebook that uses the Emscripten-compiled JavaScript version of SQLite to run queries against SQLite database files.

It turned out you couldn’t use that notebook against SQLite files hosted in Datasette because they weren’t covered by Datasette’s CORS option. Now they are!

• New documentation on Designing URLs for your plugin. (#1053)

Recommendations for plugin authors, inspired by a question from David Kane on Twitter. David has been building datasette-reconcile, a Datasette plugin that offers a reconciliation API endpoint that can be used with OpenRefine. What a brilliant idea!

Plugin of the week: datasette-copyable

One of my goals with Datasette is to make getting data out of it as easy as possible. I want Datasette to be the obvious place to store your data because you’ll be able to transform it (either manually or automatically) into any other format you could possible want, often by taking advantage of plugins.

datasette-copyable is an acknowledgement that often the easiest way to export data is using copy and paste! It adds the ability to export Datasette tables in 21 different formats - from TSV to Jira table markup to LaTeX, Mediawiki markup or HTML.

The most useful of these is the first one: TSV. The great thing about tab-separated data is that you can paste it directly into Microsoft Excel, Apple Numbers or Google Sheets and it will be automatically arranged into cells that correspond to the original table. This makes copy-as-TSV by far the easiest way to get small amounts of data out of Datasette and into a spreadsheet.

The other flavours of export are provided by the excellent Tabulate Python library, which is also used for the different command-line export options provided by sqlite-utils.

You can try out datasette-copyable on fivethirtyeight.datasettes.com - this linked example provides exports of airline safety data.

Since copy-and-paste is such a useful way of moving data around, I’m starting to percolate the idea of supporting copy-and-paste of data into Datasette as well.

Dogsheep

The name requires some explanation. In February 2019 Stephen Wolfram published Seeking the Productive Life: Some Details of My Personal Infrastructure - an astonishingly detailed essay about the extreme lengths he has gone to over the years to optimize his personal productivity. Stephen runs a 1,000+ person company as a remote CEO, so he’s been figuring out productive remote working since long before Covid forced the rest of us to all learn to work from home.

There’s a lot of stuff in there, but one thing that stood out to me in particular was Stephen’s description of his “metasearcher”:

A critical part of my personal infrastructure is something that in effect dramatically extends my personal memory: my “metasearcher”. At the top of my personal homepage is a search box. Type in something like “rhinoceros elephant” and I’ll immediately find every email I’ve sent or received in the past 30 years in which that’s appeared, as well as every file on my machine, and every paper document in my archives.

That sounded pretty great to me, so I asked myself if I could bring all of my data from different sources together into a single personal search engine. And since I wanted to build something that was inspired by Wolfram but not nearly as impressive, I decided to call it Dogsheep.

Stephen has a search engine called Wolfram Alpha, so obviously my search engine would be called Dogsheep Beta. And I admit that this pun amused me so much I felt obliged to build the software!

Dogsheep is a collection of open source tools designed to convert personal data from different sources into SQLite databases, so that I can explore them with Datasette. The tools I’ve written so far are:

• twitter-to-sqlite - your tweets, followers, people you follow, favourited tweets.

• healthkit-to-sqlite - converts Apple HealthKit into tables about heart rate, workouts, step counts, running routes.

• swarm-to-sqlite - checkin history from Foursquare Swarm.

• inaturalist-to-sqlite - wildlife and plants you have spotted on iNaturalist

• google-takeout-to-sqlite imports data from Google Takeout, which can include your Google Maps location history, your search history on Google and more.

• genome-to-sqlite imports your genome from 23andMe into a SQLite database.

• github-to-sqlite imports repositories you have created or starred on GitHub, plus issues, issue comments, releases, commits, dependent repos. Live demo here.

• pocket-to-sqlite imports articles you have saved using Pocket

• hacker-news-to-sqlite imports stories and comments posted to Hacker News

• dogsheep-photos imports metadata about photos from Apple Photos, including machine learning labels. See Using SQL to find my best photo of a pelican according to Apple Photos.

• evernote-to-sqlite (new this week) converts exported notes from Evernote into a SQLite database. See Building an Evernote to SQLite exporter for details of how I built that one.

These tools have inspired some other developers to build their own. Here are the other tools in the Dogsheep family:

• beeminder-to-sqlite by Ben Congdon saves your goals and data points from Beeminder.

• goodreads-to-sqlite by Tobias Kunze imports your reading history from Goodreads.

• pinboard-to-sqlite by Jacob Kaplan-Moss saves your bookmarks from Pinboard.

• todoist-to-sqlite by Ben Congdon saves your tasks and projects from Todoist.

• parkrun-to-sqlite by Mark Woodbridge imports your parkruns.

If you want to see a demo of my own personal Dogsheep instance in action, take a look at the talk I gave about the project at PyCon AU 2020:

I used this Google Doc as a handout for the session, with links, notes and the Q&A.

Dogsheep Beta

I finally got dogsheep-beta working a few weeks ago. It’s the search engine that ties everything else together, and it works by building a single SQLite table with a search index built against copies of data pulled from all of the other sources.

It uses YAML configuration to specify queries that should be run against each data source to build the index - extracting out titles, timestamps and searchable text for each record.

The YAML can also define SQL to be used to load extra details about search results in order to feed variables to a template, and those templates can then be embedded in the YAML file as well. This means you can use Dogsheep Beta to quickly define custom multi-source search engines for any kind of content that you’ve previously pulled into a SQLite database.

It may even be useful for things other than personal search engines! I should really get a live demo up of it running somewhere that’s not full of my own personal data.

In the meantime, here’s a screenshot of my personal Dogsheep Beta instance showing a search for #dogfest that returns both tweets and Swarm checkins:

I released dogsheep-beta 0.9 at the weekend, upgraded to use the new datasette.client internal API request mechanism added in Datasette 0.50.

Datasette 1.0 API changes

I continue to work towards a Datasette 1.0 release. Datasette is pretty stable at the moment, but for 1.0 I want to be able to make a promise that plugins built against Datasette and external code written against Datasette’s JSON API will continue to work without changes until a 2.0 release that breaks backwards-compatibility: and with any luck I hope to stay in the 1.x series for as long as possible.

Datasette’s default JSON API at the moment isn’t quite right. When I work with it myself I almost always find myself opting for the ?_shape=array option - compare the following:

Default: covid-19.datasettes.com/covid/ny_times_us_counties.json

shape=array: covid-19.datasettes.com/covid/ny_times_us_counties.json?_shape=array

I’m reconsidering the default design at the moment in #782: Redesign default JSON format in preparation for Datasette 1.0. This is one of the most significant remaining tasks before a 1.0 release, so I want to be sure to get it right!

I’m experimenting with the new default shape in a plugin called datasette-json-preview, which adds a new .json-preview extension showing what I’m thinking about. You can try that out at https://latest-with-plugins.datasette.io/fixtures/facetable.json-preview - that’s a demo Datasette instance I run which installs as many plugins as possible in order to see what happens when they all run inside the same instance.

I am very keen to get feedback and suggestions on the JSON redesign. Please post any comments you might have on the issue.

Plugin of the week: datasette-cluster-map

datasette-cluster-map is the first Datasette plugin I released, and remains one of my favourites.

Once installed, it quietly checks each table to see if it has latitude and longitude columns. If it does, the plugin triggers, loads up a Leaflet map and attempts to render all of the rows with a latitude and longitude on a map.

It’s using Leaflet.markercluster under the hood, which in my experienc will quite happily display hundreds of thousands of clustered markers on a single page.

Here’s my favorite demo: a table of 33,000 power plants around the world using data maintained by the World Resources Institute. You can show them all on one map, or you can use Datasette facets to filter down to e.g. the 429 hydro plants in France:

I find myself using this plugin constantly - any time I come across data with latitude and longitude columns I’ll render it on a map, and often I’ll spot interesting anomolies - like the fact that the island of Corsica in the Mediterranean remains a region of France and has 7 Hydro power plants, as shown in the image above.

This also illustrates a pattern that I’m keen on exploring more: Datasette plugins which detect something interesting about the data that is being displayed and automatically offer additional ways to explore and visualize it. datasette-cluster-map is still the best example of this, but other examples include datasette-render-binary (tries to detect the format of binary data) and datasette-leaflet-geojson (spots GeoJSON values and renders them as a map).

Join the conversation

More newsletter next week, but in the meantime I’d love to hear from you on Twitter (I’m @simonw) or in the Datasette GitHub Discussions. Thanks for reading!

Every week I’ll be sharing news and tips from the ecosystem of tools that is growing around Datasette and SQLite.

Datasette 0.50

Datasette 0.50 came out this week, closely followed by 0.50.1 and 0.50.2 to squash some tiny bugs that sneaked into the release.

The biggest new feature in this release is something I’m calling the column actions menu. A little cog icon in the header of each table column provides options for sorting or faceting by that column - and there’s space for new features to be added later, including potentially features defined by plugins.

You can try the menu out at https://latest.datasette.io/fixtures/facetable - latest.datasette.io is a Datasette instance which is deployed on every commit to the main branch in the repository, so it’s always available to try out new features.

The other big feature in 0.50 is aimed at plugin authors. datasette.client is a new internal API for making requests to Datasette’s JSON HTTP API from within plugins themselves. This means that anything exposed in the external API can now be used by plugins as well. datasette.client is powered by the excellent HTTPX async Python HTTP library, taking advantage of it’s ability to simulate requests against an ASGI application without making a real network call. The new internal API is documented here.

Deploying Datasette

The datasette publish command has long been the recommended way of publishing a Datasette instance on the internet - just one command-line command can upload a database to Heroku, Cloud Run, Vercel or fly.io and configure it to be served by Datasette.

Sometimes this isn’t enough. The providers supported by datasette publish are all read-only: great for hosting static data but no use if you want to make changes to it without redeploying (e.g. using the datasette-upload-csvs plugin). Datasette 0.50 includes a new documentation section on Deploying Datasette with notes on deploying on Linux using systemd - I’ve used this myself for a few projects but I’d love to hear from other people who have tried this out and have suggestions for how I can improve the recipe.

Last week DigitalOcean announced App Platform, their new Heroku-style PaaS hosting platform. I figured out how to run Datasette on it and used that to inform a new section of the documentation on Deploying using buildpacks - the mechanism originally invented by Heroku which has now been adopted by various other providers. My simonw/buildpack-datasette-demo GitHub repo shows a simple example of how this style of deployment can be used.

Git scraping

Git scraping is a Datasette-adjacent technique I’ve been experimenting with over the past few years that involves scraping data sources into a Git repository, in order to track changes to those sources over time.

I finally wrote an explanation of it on my blog: Git scraping: track changes over time by scraping to a Git repository. This lead to an interesting discussion on Hacker News where a number of people shared their own experiences using this pattern.

Here’s a diff I captured illustrating the ongoing Zogg Fires in Northern California (scraped from www.fire.ca.gov/incidents into my simonw/ca-fires-history repo):

Having captured data in this way, the next challenge is to visualize and analyze that. In the past I’ve done this using SQLite and Datasette, by writing a script to walk through the repository history and create a SQLite database of the various changes. My best example of doing that is described in Tracking PG&E outages by scraping to a git repo from October last year.

Extracting columns with sqlite-utils extract

sqlite-utils is my combination Python library and CLI utility for creating and manipulating SQLite databases. It’s my attempt at answering the question “how do I get this into SQLite?” for as many different shapes of data as possible.

sqlite-utils 2.20, released last month, added a new tool for a common problem I face when working with data that started out as a CSV file: column extraction.

It’s common for data to have duplicated columns - this example has Department Code: POL and Department: Police for example. In an ideal, normalized world we would extract these columns out into a separate database table.

That’s what sqlite-utils extract lets you do. I wrote a tutorial showing how to use it to refactor the Employee Compensation CSV file for the City of San Francisco into something a bit more relational:

Refactoring databases with sqlite-utils extract - 23rd September 2020

See also the project documentation on sqlite-utils extract.

Plugin of the week: datasette-graphql

Datasette has more than 50 plugins now, so I’m planning to highlight a plugin in the newsletter every week. This week: datasette-graphql.

I started working on this plugin partly to learn more about GraphQL and partly to demonstrate that it wasn’t a great fit for Datasette - I’ve long thought that SQL makes a more interesting API language than GraphQL, and Datasette supports SQL out of the box.

I’ve since changed my mind: GraphQL running on top of Datasette is actually a really pleasant way of interacting with data, thanks mainly to the brilliant GraphiQL API explorer tool which adds auto-complete and introspected documentation.

Here’s an example GraphQL query running against a demo Datasette instance with data I pulled in from GitHub using my github-to-sqlite tool. The query searches for repos matching “datasette” with at least 10 stargazers, then for each repository returns the first 5 issue titles supported by most recently created.

An interesting challenge with GraphQL is that, because it supports nested queries, it can lead to an enormous number of underlying SQL queries - a classic case of the N+1 queries problem.

Because SQLite runs as an in-process database library without needing the overhead of network requests to a server it’s actually an extremely good fit for implementing GraphQL - as explained by the classic essay Many Small Queries Are Efficient In SQLite.

I’ve started deploying datasette-graphql on all sorts of projects now - a few examples:

• til.simonwillison.net/graphql

• covid-19.datasettes.com/graphql

I also used it to help Natalie Downe build Rocky Beaches, a new website providing tips on when and where to go tidepooling around the SF Bay Area. That site takes advantage of the graphql() template function exposed by the plugin, which allows you to use GraphQL server-side to make data available to custom Datasette templates, as seen in the row-data-places.html template.

You can learn more about the plugin in the datasette-graphql README. Installation is as easy as running “datasette install datasette-graphql”.

Feedback, suggestions and more

This is my first time sending out this newsletter, so I’m really keen to hear your feedback. You can contact me at swillison@gmail.com, @simonw on Twitter or by posting on Datasette’s GitHub Discussions forum. I’d love to hear from you!

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!