death and gravity

Learn Python object-oriented programming with Raymond Hettinger

2026-04-16T09:00:00+00:00

💢 There must be a better way.

Raymond Hettinger is a Python core developer. Even if you haven't heard of him, you've definitely used his work, bangers such as sorted(), enumerate(), collections, itertools, @lru_cache, and many others. Over the years, he's held lots of great talks, some of them on effective object-oriented programming in Python.

The talks in this article had a huge impact on my development as a software engineer, are some of the best I've heard, and are the single most important reason you should not be afraid of inheritance anymore; don't trust me, look at the YouTube comments!

Note

This list is only to whet your appetite – to get the most out of it, watch the talks in full; besides being a great teacher, Raymond is quite the entertainer, too.

Contents

The Art of Subclassing
Python's Class Development Toolkit
Super considered super!
Object Oriented Programming from scratch (four times)
Beyond PEP 8 – Best practices for beautiful intelligible code
Bonus: The Mental Game of Python

The Art of Subclassing #

Subclassing can just be viewed as a technique for code reuse.

The Art of Subclassing (2012) is about use cases, principles, and design patterns for inheritance, with examples from the standard library.

The point is to unlearn the animal examples – instead of the classic hierachical view where subclasses are specializations of the parent, there's an operational view where:

classes are dicts of functions
subclasses point to other dicts to reuse their code
subclasses decide when to delegate

This view brings clarity to other related topics:

Liskov substitution principle: allow existing code to work with your subclass¹
circle–ellipse problem: the class with the most reusable code should be the parent
open–closed principle: subclasses should not break base class invariants

Finally, a quote about the standard library:

The best way to become a better Python programmer is to spend some time reading the source code written by great Python programmers.

Sounds familiar?

Learn by reading code: Python standard library design decisions explained

Python's Class Development Toolkit #

Each user will stretch your code in different ways.

Python's Class Development Toolkit (2013) is a hands-on exercise: build a single class, encounter common problems users have with it, come up with solutions, repeat.

Use the lean startup methodology to build an advanced circle analytic toolkit with:

instance and class variables
instance methods (self refers to you or your children²)
class methods (use cls for alternative contructors)
static methods (attach functions to classes for discoverability)
properties (transparent getters and setters)
slots (when you have many, many instances)

Super considered super! #

Super considered super! (2015) goes deep into cooperative multiple inheritance, problems you might encounter, and how to fix them.

The main point is that just like how self refers not to you, but to you or your children, super() does not call your ancestors, but your children's ancestors – it may even call a class that isn't defined yet.

This allows you to change the inheritance chain after the fact; examples include a form of dependency injection, overriding parent behavior without changing its code, and an OrderedCounter class based on Counter and OrderedDict.

The article by the same name is worth a read too, and has different examples.

Object Oriented Programming from scratch (four times) #

Object Oriented Programming from scratch (four times) (2020) does exactly that, each time giving a new insight into what, how, and why we use objects in Python.

The first part shows OOP emerge naturally from the need for more namespaces, by iteratively improving a script that emulates dictionaries.

The second one covers the history of moving from a huge pile of data and functions to:

data associated with functions (objects)
groups of related functions (classes)
related groups of functions (inheritance)
using the same name for similar functions (polymorphism)

Sounds familiar?

When to use classes in Python? When your functions take the same arguments
When to use classes in Python? When you repeat similar sets of functions

The third part explains the mechanics of objects via ChainMap, tl;dr:

ChainMap(instance_dict, class_dict, parent_class_dict, ...)

The fourth part highlights how OOP naturally expresses entities and relationships by looking the data model of a Twitter clone and the syntax tree of a compiler.

Beyond PEP 8 – Best practices for beautiful intelligible code #

Well factored code looks like business logic.

Beyond PEP 8 – Best practices for beautiful intelligible code (2015) (code) is about how excessive focus on following PEP 8 can lead to code that is beautiful, but bad, since it distracts from the beauty that really matters:

Pythonic

coding beautifully in harmony with the language to get the maximum benefits from Python

Transform a bad API into a good one using an adapter class and stuff like:

context managers for setup / teardown
flat modules for simpler imports
magic methods to make things iterable
properties instead of getter methods
custom exceptions for clearer business logic
a good __repr__() for better debuggability

And remember:

If you don't transform bad APIs into Pythonic APIs, you're a fool!

Bonus: The Mental Game of Python #

The computer gives us words that do things; what daddy does is make new words to make computers easier to use.³

The Mental Game of Python (2019) is not just about programming, but about problem solving strategies. The most relevant one for OOP is: build classes independently and let inheritance discover itself; this is because:

A lot of real world problems aren't tic-tac-toe problems, where you can see to the end; they are chess problems, where you can't.

For a more meta discussion of this idea, check out Repeat yourself, do more than one thing, and rewrite everything by tef; it should be considered a classic at this point.

Parting Raymond Hettinger quote:

I came to here you to show you how to chunk, and how to stack one chunk on top of the other, and this is a way to reduce your cognitive load and manage complexity; it is the core of our craft; it's what we're here to do.

Anyway, that's it for now. :)

Who learned something new today? Share it with others! PyCoder's Weekly HN Bluesky linkedin Twitter

But remember it's a principle, not a law, violations are fine – you may only want substitutability in some places (e.g. contructors are often not substitutable). ^[return]
...unless you use double underscores. ^[return]
In my opinion, this quote alone is reason enough to watch the talk. ^[return]

reader 3.22 released – new web app

2026-04-02T18:00:00+00:00

Hi there!

I'm happy to announce version 3.22 of reader, a Python feed reader library.

What's new? #

Here are the highlights since reader 3.20.

New feed reader web app #

The new web application is done! Features include:

add / list / filter / delete feeds
list / filter articles
mark articles as read / (un)important
article view
custom feed titles
dark mode

In the next releases, I'll be adding back features already present in the legacy web app, stuff like full-text search, tags, read time, MP3 tag fixing, and more.

This is building towards a hosted version of reader, which should take the pain out of self-hosting, while still leaving it as an option; more to follow soon™. (Meanwhile, if this sounds like something you'd like to use, get in touch.)

For now, here are some screenshots:

main page (dark mode)

more filters (light mode)

feed page (dark mode)

feeds page (dark mode)

article view (light mode)

article view (dark mode)

Config and plugin loading #

Part of the hosted reader work, I've unified how configuration and plugins are loaded across make_reader(), the command-line interface, and the web app, by using Click to parse and validate configuration (it's not as wrong as it sounds, I promise).

As a consequence, the config file format changed from YAML to TOML and follows the shape of the CLI, and a few commands and environment variables were renamed; no other breaking changes are expected in the foreseeable future.

Scheduled updates by default #

Both update_feeds() and the CLI now limit how often feeds get updated by default; while this is a minor compatibility break, the previous behavior was arguably a bug – doing the right thing should not be opt-in.

Also, reader now honors the Cache-Control max-age and Expires HTTP headers when updating feeds, in addition to Retry-After.

AI contributions #

Finally, reader now has an AI contributions policy; tl;dr: they are banned, for now.

The reasoning is two-fold. First, after a few low-effort contributions, I decided I don't have time for this. Second, there are various issues surrounding LLMs I don't want to bother with; for more details, see the Servo, CPython, and LLVM policies.

I am open to revisiting this later (I'll do so on my own, though, thank you).

That's it for now. For more details, see the full changelog.

Want to contribute? Check out the docs and the roadmap.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

reader is a library
reader is for the long term
reader is extensible
reader is stable (within reason)
reader is simple to use; API matters
reader features work well together
reader is tested
reader is documented
reader has minimal dependencies

DynamoDB crash course: part 2 – data model

2026-02-10T10:35:00+00:00

This is part two of a series covering core DynamoDB concepts, from philosophy all the way to single-table design. The goal is to get you to understand idiomatic usage and trade-offs in under an hour.

Today, we're looking at the DynamoDB data model – what the main abstractions are, what you can do with them, and how they scale.

(While the AWS documentation is mostly comprehensive, it's also all over the place, including some other places that aren't the documentation at all, like the AWS blog. This series brings the important stuff in one place, so you can get a mental model of how it all ties together without having to read the entire documentation twice).

Contents

Core components
Limits
Indexes
- Global secondary indexes
- Local secondary indexes
Features

Core components #

According to the documentation, the core components of DynamoDB are tables, items, and attributes. This is accurate in the sense of what you can act on through the API, but can be deceptively simple, and leaves out two other equally important aspects: what you can do with it (the logical model) and how it scales (the physical model).

Let's put it all together, starting from the top.

API model: tables, items, attributes #

As far as the API is concerned, "a table is a collection of items, and each item is a collection of attributes".¹

An item is uniquely identified by two attributes, the partition key and the sort key,² which together compose its primary key.³ A group of items with the same partition key value is called an item collection,⁴ but this is more of a logical grouping, and does not exist as a distinct entity in the API.

An attribute is a named data element, with its value either a scalar (number, string, binary, boolean, null), a set of scalars, or a document (a list or map of possibly nested attributes, similar to JSON).

There are no limits on table size or number of items, nor on those of an item collection. Items do have a size limit of 400 KB / item, which indirectly limits attribute size.

As we've seen in the previous article, the core DynamoDB data operations are:

PutItem, GetItem, UpdateItem, DeleteItem
Query items with the same partition key, sorted by sort key, and optionally narrowed down to a specific a range of sort keys
Scan all the items in the table, possibly in parallel

Besides whole items, the API allows getting and updating specific attributes, as well as filtering query and scan results by expressions using them.

See also

Logical model: hash table of B‍-‍trees #

The operations above may seem arbitrarily restrictive – for example, why can't I query items by sort key alone? It might make more sense to think about it like this:

Conceptually, a DynamoDB table is a hash table of B‍-‍trees, with partition keys being hash table keys, and sort keys being B‍-‍tree keys (making item collections B‍-‍trees). The hash table allows efficient find collection by partition key operations; within each collection, the B‍-‍tree keeps the items sorted, and allows efficient find item by sort key and find items by sort key range operations.

As a consequence, any access not based on partition and sort key is expensive, since instead of taking advantage of the underlying data structure, you have to go through all the items in the table to find anything (aka a full table scan), and at the scales you'd use DynamoDB at, this can mean billions of items.⁵

Example

(from here) Take a Music table where items correspond to songs, with Artist as primary key and Song as sort key:

# table Music (partition key: Artist, sort key: Song)
1000mods: !btree
  Claws: { Album: Vultures }
  Vidage: { Year: 2011 }
Kyuss: !btree
  Space Cadet: { }

You can efficiently:

query songs by artist (sorted by song title)
get the song by artist and song title

...and that's it, anything else requires a full table scan.

To a first approximation, this is also a decent model of how DynamoDB scales – you could imagine that each collection has its own dedicated computer, which in theory would account for the unlimited number of collections.

Physical model: partitions #

Of course, there are not infinitely many computers, and that would be wildly inefficient anyway. Instead, collections are packed together into a smaller number of partitions, each a few gigabytes in size. To figure out which partition an item should go on, DynamoDB hashes its partition key (also called a hash key, for obvious reasons).

This is similar to hash table buckets,⁶ except there's one more level of indirection – instead of mapping to a single number, each partition maps to a range of numbers, which allows splitting a partition into two new ones by splitting its range. Furthermore, an item collection can be split on multiple partitions too, by using the sort key.⁷

And that is how the scaling magic happens:

When you increase provisioned capacity, partitions are split as needed.
If a partition or collection becomes too big, it gets split.
If the throughput to a partition or collection is high enough for long enough, it also gets split,⁸ possibly with a bias towards keys with higher utilization; this is a feature of adaptive capacity.

Partition management is handled entirely by DynamoDB and is transparent to the user, but it doesn't happen instantly – it takes several minutes to allocate new partitions and shuffle things around.

Since partitions are backed by real computers, they do have a throughput limit.

See also

Limits #

Part of DynamoDB's appeal is that it scales "infinitely" for specific dimensions: there are no limits on table size or number of items. However, there are some hard, non-adjustable limits you will have to take into account when designing your application.

See also

Cheat sheet # Service quota basics
Quotas and Constraints
(unofficial) The Three DynamoDB Limits You Need to Know

Partition throughput #

The most important limit is that on partition throughput (aka capacity) – how much data DynamoDB can read from or write to a partition in a given amount of time:⁹

1 MB/s for writes
24 MB/s for reads, eventually consistent
12 MB/s for reads, strongly consistent

Throughput measures whole items DynamoDB has to access, not the data that goes through the API. While you can touch single attributes and filter query results, the consumed capacity is always that of the whole items DynamoDB had to read or write.¹⁰

Once you reach the limit, the operation is throttled, and you can try again later, ideally with exponential backoff (the AWS SDK usually takes care of this for you).

The best way to avoid throttling is to distribute the load uniformly across partitions by using a high-cardinality partition key.¹¹ Uneven key distribution can create hot partitions that suffer from persistent throttling.

Nowadays, this is less of a problem. For long-term imbalances, partition splitting should rebalance things over time; you "might" even end up with a single popular item per partition. For short-term ones like traffic spikes, burst and adaptive capacity will, on a best effort basis, "borrow" capacity above the table limit and between partitions. However, AWS is very non-committal about their behavior, and there's nothing you can do besides increasing traffic gradually, so good partition key design remains key.

Of note, while the throughput is fixed, the other dimensions are not; this means that you have a trade-off between how often you access items, the number of items, and item size; for example, you can split items into smaller ones based on how attributes are accessed, aka vertical partitioning.

See also

Read and write operations (capacity unit consumption)
Partition key design and Distributing workloads
Sort key design
(blog) Choosing the Right DynamoDB Partition Key

Item size #

Second, the maximum item size is 400 KB, which ought to be enough for anybody. You can work around this limit either by splitting items into parts, or by putting the data somewhere else entirely, like S3, and keeping only a reference in DynamoDB.

See also

Page size #

Finally, the maximum response size for query and scan operations is 1 MB (a page). You can continue from the end of the previous page by passing the LastEvaluatedKey response element to subsequent calls, which is essentially keyset pagination. One consequence of this is that, throughput limit aside, there's an implicit limit on how fast you can query the items in a collection, since the calls are sequentiall.¹²

See also

Indexes #

As discussed in the logical model, access not based on primary key is very inefficient.

Secondary indexes allow queries and scans that use alternative primary keys, ones composed of different attributes than that of the base table. Unlike tables, index sort keys do not have to be unique for a given partition key. An item that is missing one of the index primary key attributes will not appear in the index.

Changes to the table are automatically propagated to any secondary indexes. Aside from the index and table primary key attributes, an index can include copies of other attributes (aka attribute projection), which allows the index to answer queries alone, without extra reads to the base table.¹³

See also

Global secondary indexes #

A global secondary index allows using different partition and sort key attributes.

Conceptually, a global secondary index is just a table: it has its own separate capacity, no limits on size or number of items, and the same partition throughput limits apply.

Despite GSIs being updated asynchronously, an index without enough capacity to process the updates will cause write throttling. To retrieve attributes not in the index, you have to get them yourself from the table (batch operations can speed this up).

Example

(from here) Continuing with the music example, a GSI with Genre and Album as partition and sort keys would allow you to also efficiently:

query songs by genre (and, with additional processing, albums by genre)
query songs by genre and album (but, since two albums can have the same genre and title, you might want to group by artist in application code)

# table Music (partition key: Artist, sort key: Song)
Kyuss: !btree
  Demon Cleaner: { Album: Welcome To Sky Valley, Genre: Rock }
  Space Cadet: { Album: Welcome To Sky Valley, Genre: Rock }
1000mods: !btree
  Claws: { Genre: Rock }  # has no Album!
  Vidage: { Album: Super Van Vacation, Genre: Rock }
Solar Fields: !btree
  Air Song: { Album: Leaving Home, Genre: Electronic }

# GSI Genres (partition key: Genre, sort key: Album)
Rock: !btree
  Super Van Vacation: { Artist: 1000mods, Song: Vidage}
  Welcome To Sky Valley: { Artist: Kyuss, Song: Space Cadet }
  Welcome To Sky Valley: { Artist: Kyuss, Song: Demon Cleaner }
Electronic: !btree
  Leaving Home: { Artist: Solar Fields, Song: Air Song }

See also

Local secondary indexes #

A local secondary index allows using a different sort key attribute.

LSI data is stored together with partition data (the index is local to the partition), so besides the table B‍-‍tree, each collection has one B‍-‍tree per LSI.

This allows strongly consistent reads and fetching non-projected attributes, but also limits collection size to 10 GB and collection throughput to the partition limit, since it prevents further partition splitting (as each sort key would split the items in a different way).

Example

A LSI with Year as sort key would allow you to also efficiently:

query songs by artist, in chronological order
query songs by artist and year

# table Music (partition key: Artist, sort key: Song)
1000mods: !btree
  Claws: { Year: 2014 }
  Road To Burn: { Year: 2011 }
  Vidage: { Year: 2011 }
Solar Fields: !btree
  Sombrero: { Year: 2011 }

# table Music (partition key: Artist, LSI sort key: Year)
1000mods: !btree
  2011: { Song: Vidage }
  2011: { Song: Road To Burn }
  2014: { Song: Claws }
Solar Fields: !btree
  2011: { Song: Sombrero }

See also

Features #

Let's look at some of the things DynamoDB can do besides CRUD operations.

Eventual consistency #

So, remember I said partitions are backed by real computers? I didn't say how many.

To allow the high-availability magic to happen, a partition is backed by three nodes in separate data centers:¹⁴ a leader that handles writes and two asynchronous replicas.

This explains why there are two kinds of reads:

strongly consistent reads go to the leader, so you always get the latest data
eventually consistent reads go to any node, so you may get slightly older data, but if you repeat the read later, you will eventually get the latest data; because they use all the available nodes, they are more efficient, and thus cheaper

Note that strongly consistent reads do not replace synchronization primitives like conditional writes and transactions, but they can be useful to lower the rate at which these operations fail for highly-contended items.

See also

DynamoDB read consistency

Conditional writes #

Write operations can specify a condition expression that must be true for the write to happen (e.g. an attribute has a specific value); if the expression is false, the write fails. Condition expressions can refer only to the item being modified.

Conditional writes are critical for data consistency and avoiding concurrency bugs, since they are only the way to run logic server-side, while the item is being modified. You can use conditional writes to build higher level abstractions like optimistic locking, distributed locks, and atomic counters.¹⁵

See also

Transactions #

Transactions allow performing multiple writes as a single atomic operation, isolated from other operations; if two operations attempt to change an item at the same time, one of them fails. Transactions can target up to 100 distinct items in one or more tables in the same region, and consume twice as much capacity.

You can use transactions with condition expressions – if a condition fails for one item, none of the items are modified; you can also check an item without modifying it. Like with single-item writes, an expression can refer only an individual item (you can't have a condition about another item in the transaction).

See also

Working with transactions

Batch operations #

Batch operations allow you to put/delete up to 25 items or read up to 100 items in a single request, up to 16 MB in total, more efficiently than using single-item operations. Batch writes don't support updates or condition expressions.

The operations in a batch are independent from one another – some writes may fail, or only some of the read items may be returned (e.g. if throughput limits are reached).

See also

Streams #

Streams allow you to capture changes to the items in a table in near-real time. There are two flavors of streams, DynamoDB Streams and Kinesis Data Streams, each with different features and integrations.

Notable applications of streams are Lambda triggers (similar to the ones in relational databases, except they run after the change), replication to places like S3 or Redshift via Firehose, and automatic archival.

See also

Anyway, that's it for now.

In the next article, we'll have a closer look at core DynamoDB design patterns, including the fundamental single-table design.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Here, "collection" just means a "group of things". ^[return]
Ignore the names for now, it'll make sense in a bit. ^[return]
It is also possible to have a table with only a partition key and no sort key, but you can think of that like a degenerate case where the sort key is a constant value, and thus each partition key can have only one item. ^[return]
Yes, an item collection is different from a "collection of items". Don't look at me, I didn't pick the names. ಠ_ಠ ^[return]
Indexes, which we'll discuss later, offer an escape hatch to this. ^[return]
A quick rant on naming. With a hash table, you would say "hash table key", or maybe even "item key"; you would not say "bucket key", since that's a low level detail, and also a bucket can have multiple keys. You know, like in DynamoDB.

THEN WHY IS IT CALLED A PARTITION KEY ^[return]
You have to admit this is a great explanation. Surely you'd find it in the docs, and not burried in a random blog post published four years after the feature was announced, also in a blog post, and which itself explains more than the official documentation does to this day, seven years later! ^[return]
Assuming the table has enough configured throughput. ^[return]
Converted to normal people units for your convenience. DynamoDB uses its own capacity units as a convoluted way of saying that for accounting purposes, item size is rounded up to 4 KB (1 RCU) for reads and 1 KB (1 WCU) for writes. This is presumably because the size of a capacity unit can increase over time. ^[return]
Yes, this includes just counting them. ^[return]
If there is no good natural partition key, you can make one by sharding a low-cardinality attribute, which we'll cover in the next article. ^[return]
You could make it twice as fast by querying from both ends, but probably no faster, since for binary search you'd need to jump in the middle of two sort keys. Unsurprisingly, we'll look at a potential solution in the next article. ^[return]
This is the same as a covering index in relational databases. ^[return]
Or better said, availability zones. ^[return]
Although you can also use update expressions for atomic counters. ^[return]

DynamoDB crash course: part 1 – philosophy

2026-01-27T17:00:00+00:00

This is part one of a series covering core DynamoDB concepts and patterns, from the data model and features all the way up to single-table design.

The goal is to get you to understand what idiomatic usage looks like and what the trade-offs are in under an hour, providing entry points to detailed documentation.

(Don't get me wrong, the AWS documentation is comprehensive, but can be quite complex, and DynamoDB being a relatively low level product with lots of features added over the years doesn't really help with that.)

Today, we're looking at what DynamoDB is and why it is the way it is.

What is DynamoDB? #

Quoting Wikipedia:

Amazon DynamoDB is a managed NoSQL database service provided by AWS. It supports key-value and document data structures and is designed to handle a wide range of applications requiring scalability and performance.

See also

This definition should suffice for now; for a more detailed refresher, see:

The DynamoDB data model can be summarized as follows:

A table is a collection of items, and an item is a collection of attributes. Items are uniquely identified by two attributes, the partition key and the sort key. The partition key determines where (i.e. on what computer) an item is stored. The sort key is used to get ordered ranges of items from a specific partition.

That's is, that's the whole data model. Sure, there's indexes and transactions and other features, but at its core, this is it. Put another way:

A DynamoDB table is a hash table of B-trees¹ – partition keys are hash table keys, and sort keys are B-tree keys. Because of this, any access not based on partition and sort key is expensive, since you end up doing a full table scan.

If you were to implement this model in Python, it'd look something like this:

from collections import defaultdict
from sortedcontainers import SortedDict

class Table:

    def __init__(self, pk_name, sk_name):
        self._pk_name = pk_name
        self._sk_name = sk_name
        self._partitions = defaultdict(SortedDict)

    def put_item(self, item):
        pk, sk = item[self._pk_name], item[self._sk_name]
        old_item = self._partitions[pk].setdefault(sk, {})
        old_item.clear()
        old_item.update(item)

    def get_item(self, pk, sk):
        return dict(self._partitions[pk][sk])

    def query(self, pk, min_sk=None, max_sk=None, inclusive=(True, True), reverse=False):
        # in the real DynamoDB, this operation is paginated
        partition = self._partitions[pk]
        for sk in partition.irange(min_sk, max_sk, inclusive, reverse):
            yield dict(partition[sk])

    def scan(self):
        # in the real DynamoDB, this operation is paginated
        for partition in self._partitions.values():
            for item in partition.values():
                yield dict(item)

    def update_item(self, item):
        pk, sk = item[self._pk_name], item[self._sk_name]
        old_item = self._partitions[pk].setdefault(sk, {})
        old_item.update(item)

    def delete_item(self, pk, sk):
        del self._partitions[pk][sk]

>>> table = Table('Artist', 'Song')
>>>
>>> table.put_item({'Artist': '1000mods', 'Song': 'Vidage', 'Year': 2011})
>>> table.put_item({'Artist': '1000mods', 'Song': 'Claws', 'Album': 'Vultures'})
>>> table.put_item({'Artist': 'Kyuss', 'Song': 'Space Cadet'})
>>>
>>> table.get_item('1000mods', 'Claws')
{'Artist': '1000mods', 'Song': 'Claws', 'Album': 'Vultures'}
>>> [i['Song'] for i in table.query('1000mods')]
['Claws', 'Vidage']
>>> [i['Song'] for i in table.query('1000mods', m='Loose')]
['Vidage']

Philosophy #

One can't help but feel this kind of simplicity would be severely limiting.

A consequence of DynamoDB being this low level is that, unlike with most relational databases, query planning and sometimes index management happen at the application level, i.e. you have to do them yourself in code. In turn, this means you need to have a clear, upfront understanding of your application's access patterns, and accept that changes in access patterns will require changes to the application.

In return, you get a fully managed, highly-available database that scales infinitely:² there are no servers to take care of, there's almost no downtime, and there are no limits on table size or the number of items in a table; where limits do exist, they are clearly documented, allowing for predictable performance.

This highlights an intentional design decision that is essentially DynamoDB's main proposition to you as its user: data modeling complexity is always preferable to complexity coming from infrastructure maintenance, availability, and scalability (what AWS marketing calls "undifferentiated heavy lifting").

To help manage this complexity, a number of design patterns have arisen, covered extensively by the official documentation, and which we'll discuss in a future article. Even so, the toll can be heavy – by AWS's own admission, the prime disadvantage of single table design, the fundamental design pattern, is that:

[the] learning curve can be steep due to paradoxical design compared to relational databases

As this walkthrough puts it:

a well-optimized single-table DynamoDB layout looks more like machine code than a simple spreadsheet

...which, admittedly, sounds pretty cool, but also why would I want that? After all, most useful programming most people do is one or two abstraction levels above assembly, itself one over machine code.

See also

NoSQL design
(unofficial) # The DynamoDB philosophy of limits

A bit of history #

Perhaps it's worth having a look at where DynamoDB comes from.

Amazon.com used Oracle databases for a long time. To cope with the increasing scale, they first adopted a database-per-service model, and then sharding, with all the architectural and operational overhead you would expect. At its 2017 peak (five years after DynamoDB was released in AWS, and over ten years after some version of it was available internally), they still had 75 PB of data in nearly 7500 Oracle databases, owned by 100+ teams, with thousands of applications, for OLTP workloads alone. That sounds pretty traumatic – it was definitely bad enough to allegedly ban OLTP relational databases internally, and require that teams get VP approval to use one.

Yeah, coming from that, it's hard to argue DynamoDB adds complexity.

That is not to say relational databases cannot be as scalable as DynamoDB, just that Amazon doesn't belive in them – distributed SQL databases like Google's Spanner and CockroachDB have existed for a while now, and even AWS seems to be warming up to the idea.

This might also explain why the design patterns are so slow to make their way into SDKs, or even better, into DynamoDB itself; when you have so many applications and so many experienced teams, the cost of yet another bit of code to do partition key sharding just isn't that great.

See also

(paper) Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service (2022)
(paper) Dynamo: Amazon’s Highly Available Key-value Store (2007)

Anyway, that's it for now.

In the next article, we'll have a closer look at the DynamoDB data model and features.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Or any other sorted data structure that allows fast searches, sequential access, insertions, and deletions. ^[return]
As the saying goes, the cloud is just someone else's computers. Here, "infinitely" means it scales horizontally, and you'll run out of money before AWS runs out of computers. ^[return]

reader 3.20 released – we're so back

2025-12-01T18:00:00+00:00

Hi there!

I'm happy to announce version 3.20 of reader, a Python feed reader library.

What's new? #

Here are the highlights since reader 3.16.

Web app re-design #

Earlier this year, I started a web app re-design based on htmx and Bootstrap. I only had time for the main page, but it turned out pretty nice, and I've been using it ever since. What is new is that I now have free time and some plans; watch this space.

Here are some screenshots; it's not much, but it's honest work:

main page (dark mode)

main page – more filters (light mode)

htmx? ugh, how original

Unsurprisingly, htmx is perfect for this kind of application. As CEO of htmx, I can confirm that all the allegations are true – it is indeed a pleasure to use, not to mention a huge improvement over my home-grown, half-assed JavaScript forms framework.

isn't Bootstrap from like 2010 or something? also, how original

I've tried many modern popular CSS frameworks (since forgotten, for my own sanity), and it turns out tab-navigable form controls are a hard, unsolved problem in 2025. Thankfully, if you're willing to time travel (to the future, surely!), Bootstrap comes with excellent documentation, built-in accessibility, and a comprehensive component library.

Entry deduplication #

I may have accidentally rewritten the entry_dedupe plugin. 😅

Sometimes, the id of some or all the entries in a feed changes (e.g. from example.com/123 to example.com/entry-title), causing each entry to appear twice. entry_dedupe fixes this by copying user attributes to the new entry and deleting the old one.

What started as a quick and dirty plugin now has:

lots of new, data-driven heuristics to find potential duplicates
- by title (had this one already)
- by link
- by published timestamp
- by title with common prefixes removed
better approximate content matching
an extensive test suite
extensive internal documentation (like this)

That last one is pretty important: the old logic was data-driven too, but because I didn't document my entire reasoning, I wasted more time than I'd like to admit on a useless title similarity heuristic. It's a trope that you should write code such that others can understand it, and "others" includes you six months from now; I don't know about six months, but four years will definitely do it. (If that sounds like an interesting story and you'd like to hear more, drop me a line.)

It's not all bad, though – a side-effect of better content matching is that image alt and title attributes are also included in the full-text search index now, so you can search xkcd comics by title text.

Project infrastructure #

I've taken some time to modernize the project infrastructure; stuff like:

Use GitHub Actions to publish releases to PyPI.
Use dependency groups instead of extras for development dependencies.
Rewrite contributor documentation and run.sh to optimize for readability.

(I'll admit the uv hype is quite strong, but the vanilla tooling seems fine, for now.)

Read-only Reader #

It's now possible to prevent write operations on a Reader object through the read_only make_reader() argument. Thanks to Roman Milko for the pull request!

Python versions #

reader 3.17 added support for PyPy 3.11.

reader 3.18 dropped support for Python 3.10.

reader 3.19 added support for Python 3.14.

That's it for now. For more details, see the full changelog.

Want to contribute? Check out the docs and the roadmap.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

reader is a library
reader is for the long term
reader is extensible
reader is stable (within reason)
reader is simple to use; API matters
reader features work well together
reader is tested
reader is documented
reader has minimal dependencies

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

Announcing asyncio-thread-runner: you can have a little async (as a treat)

2025-08-22T12:00:00+00:00

Hi there!

Back in 2023, we made a thing for running async code from sync code.

I'm happy to announce that you can now install it from PyPI, and read the documented, tested, type-annotated code on GitHub! ⭐️

Without further ado:

asyncio-thread-runner allows you to run async code from sync code.

This is useful when you're doing some sync stuff, but:

you also need to do some async stuff, without making everything async
maybe the sync stuff is an existing application
maybe you still want to use your favorite sync library
or maybe you need just a little async, without having to pay the full price

Features:

unlike asyncio.run(), it provides a long-lived event loop
unlike asyncio.Runner, you can use it from multiple threads
it allows you to use async context managers and iterables from sync code
check out this article for why these are useful

Usage:

$ pip install asyncio-thread-runner

>>> async def double(i):
...     return i * 2
...
>>> from asyncio_thread_runner import ThreadRunner
>>> runner = ThreadRunner()
>>> runner.run(double(2))
4

Annotated example:

import aiohttp
from asyncio_thread_runner import ThreadRunner

# you can use ThreadRunner as a context manager,
# or call runner.close() when you're done with it
with ThreadRunner() as runner:

    # aiohttp.ClientSession() should be used as an async context manager,
    # enter_context() will exit the context on runner shutdown;
    # because instantiating ClientSession requires a running event loop,
    # we pass it as a factory instead of calling it in the main thread
    session = runner.enter_context(aiohttp.ClientSession)

    # session.get() returns an async context manager...
    request = session.get('https://death.andgravity.com/asyncio-bridge')
    # which we turn into a normal one with wrap_context()
    with runner.wrap_context(request) as response:

        # response.content is an async iterator;
        # we turn it into a normal iterator with wrap_iter()
        lines = list(runner.wrap_iter(response.content))

    # "got 935 lines"
    print('got', len(lines), 'lines')

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Inheritance over composition, sometimes

2025-07-15T20:00:00+00:00

In ProcessThreadPoolExecutor: when I‍/‍O becomes CPU-bound, we built a hybrid concurrent.futures executor that runs tasks in multiple threads on all available CPUs, bypassing Python's global interpreter lock.

Here's some interesting reader feedback:

Currently, the code is complex due to subclassing and many layers of delegation. Could this solution be implemented using only functions, no classes? Intuitively I feel classes would be hell to debug.

Since a lot of advanced beginners struggle with structuring code, we'll implement the same executor using inheritance, composition, and functions only, compare the solutions, and reach some interesting conclusions. Consider this a worked example.

Note

Today we're focusing on code structure. While not required, reading the original article will give you a better idea of why the code does what it does.

Contents

Requirements
concurrent.futures
Three solutions
Comparison
Try it out

Requirements #

Before we delve into the code, we should have some understanding of what we're building. The orginal article sets out the following functional requirements:

Implement the Executor interface; we want a drop-in replacement for existing concurrent.futures executors, so that user code doesn't have to change.
Spread the work to one worker process per CPU, and then further to multiple threads inside each worker, to work around CPU becoming a bottleneck for I‍/‍O.

Additionally, we have two implicit non-functional requirements:

Use the existing executors where possible (less code means fewer bugs).
Only depend on stable, documented features; we don't want our code to break when concurrent.futures internals change.

concurrent.futures #

Since we're building on top of concurrent.futures, we should also get familiar with it; the docs already provide a great introduction:

The concurrent.futures module provides a high-level interface for asynchronously executing callables. [...this] can be performed with threads, using ThreadPoolExecutor, or separate processes, using ProcessPoolExecutor. Both implement the same interface, which is defined by the abstract Executor class.

Let's look at the classes in more detail.

Executor is an abstract base class¹ defined in concurrent.futures._base. It provides dummy submit() and shutdown() methods, a concrete map() method implemented in terms of submit(), and context manager methods that shutdown() the executor on exit. Notably, the documentation does not mention the concrete methods, instead saying that the class "should not be used directly, but through its concrete subclasses".

The first subclass, ThreadPoolExecutor, is defined in concurrent.futures.thread; it implements submit() and shutdown(), inheriting map() unchanged.

The second one, ProcessPoolExecutor, is defined in concurrent.futures.process; as an optimization, it overrides map() to chop the input iterables and pass the chunks to the superclass method with super().

Three solutions #

Now we're ready for code.

Inheritance #

First, the original implementation,² arguably a textbook example of inheritance.

We override __init__, submit(), and shutdown(), and do some extra stuff on top of the inherited behavior, which we access through super(). We inherit the context manager methods, map(), and any public methods ProcessPoolExecutor may get in the future, assuming they use only other public methods (more on this below).

class ProcessThreadPoolExecutor(concurrent.futures.ProcessPoolExecutor):

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        self.__result_queue = multiprocessing.Queue()
        super().__init__(
            initializer=_init_process,
            initargs=(self.__result_queue, max_threads, initializer, initargs)
        )
        self.__tasks = {}
        self.__result_handler = threading.Thread(target=self.__handle_results)
        self.__result_handler.start()

    def submit(self, fn, *args, **kwargs):
        outer = concurrent.futures.Future()
        task_id = id(outer)
        self.__tasks[task_id] = outer

        outer.set_running_or_notify_cancel()
        inner = super().submit(_submit, task_id, fn, *args, **kwargs)

        return outer

    def __handle_results(self):
        for task_id, ok, result in iter(self.__result_queue.get, None):
            outer = self.__tasks.pop(task_id)
            if ok:
                outer.set_result(result)
            else:
                outer.set_exception(result)

    def shutdown(self, wait=True):
        super().shutdown(wait=wait)
        if self.__result_queue:
            self.__result_queue.put(None)
            if wait:
                self.__result_handler.join()
            self.__result_queue.close()
            self.__result_queue = None

Because we're subclassing a class with private, undocumented attributes, our private attributes have to start with double underscores to avoid clashes with superclass ones (such as _result_queue).

In addition to the main class, there are some global functions used in the worker processes which remain unchanged regardless of the solution:

# this code runs in each worker process

_executor = None
_result_queue = None

def _init_process(queue, max_threads, initializer, initargs):
    global _executor, _result_queue

    _executor = concurrent.futures.ThreadPoolExecutor(max_threads)
    _result_queue = queue

    if initializer:
        initializer(*initargs)

def _submit(task_id, fn, *args, **kwargs):
    task = _executor.submit(fn, *args, **kwargs)
    task.task_id = task_id
    task.add_done_callback(_put_result)

def _put_result(task):
    if exception := task.exception():
        _result_queue.put((task.task_id, False, exception))
    else:
        _result_queue.put((task.task_id, True, task.result()))

Download the entire file.

Composition #

OK, now let's use composition – instead of being a ProcessPoolExecutor, our ProcessThreadPoolExecutor has one. At a first glance, the result is the same as before, with super() changed to self._inner:

class ProcessThreadPoolExecutor:

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        self._result_queue = multiprocessing.Queue()
        self._inner = concurrent.futures.ProcessPoolExecutor(
            initializer=_init_process,
            initargs=(self._result_queue, max_threads, initializer, initargs)
        )
        self._tasks = {}
        self._result_handler = threading.Thread(target=self._handle_results)
        self._result_handler.start()

    def submit(self, fn, *args, **kwargs):
        outer = concurrent.futures.Future()
        task_id = id(outer)
        self._tasks[task_id] = outer

        outer.set_running_or_notify_cancel()
        inner = self._inner.submit(_submit, task_id, fn, *args, **kwargs)

        return outer

    def _handle_results(self):
        for task_id, ok, result in iter(self._result_queue.get, None):
            outer = self._tasks.pop(task_id)
            if ok:
                outer.set_result(result)
            else:
                outer.set_exception(result)

    def shutdown(self, wait=True):
        self._inner.shutdown(wait=wait)
        if self._result_queue:
            self._result_queue.put(None)
            if wait:
                self._result_handler.join()
            self._result_queue.close()
            self._result_queue = None

Except, we need to implement the context manager protocol ourselves:

    def __enter__(self):
        # concurrent.futures._base.Executor.__enter__
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        # concurrent.futures._base.Executor.__exit__
        self.shutdown(wait=True)
        return False

...and we need to copy map() from Executor, since it should use our submit():

    def _map(self, fn, *iterables, timeout=None, chunksize=1):
        # concurrent.futures._base.Executor.map

        if timeout is not None:
            end_time = timeout + time.monotonic()

        fs = [self.submit(fn, *args) for args in zip(*iterables)]

        def result_iterator():
            try:
                fs.reverse()
                while fs:
                    if timeout is None:
                        yield _result_or_cancel(fs.pop())
                    else:
                        yield _result_or_cancel(fs.pop(), end_time - time.monotonic())
            finally:
                for future in fs:
                    future.cancel()
        return result_iterator()

...and the chunksize optimization from its ProcessPoolExecutor version:

    def map(self, fn, *iterables, timeout=None, chunksize=1):
        # concurrent.futures.process.ProcessPoolExecutor.map

        if chunksize < 1:
            raise ValueError("chunksize must be >= 1.")

        results = self._map(partial(_process_chunk, fn),
                            itertools.batched(zip(*iterables), chunksize),
                            timeout=timeout)
        return _chain_from_iterable_of_lists(results)

...and a bunch of private functions they use.

def _result_or_cancel(fut, timeout=None):
    # concurrent.futures._base._result_or_cancel
    try:
        try:
            return fut.result(timeout)
        finally:
            fut.cancel()
    finally:
        del fut

def _process_chunk(fn, chunk):
    # concurrent.futures.process._process_chunk
    return [fn(*args) for args in chunk]

def _chain_from_iterable_of_lists(iterable):
    # concurrent.futures.process._chain_from_iterable_of_lists
    for element in iterable:
        element.reverse()
        while element:
            yield element.pop()

And, when the Executor interface gets new methods, we'll need to at least forward them to the inner executor, although we may have to copy those too.

On the upside, no base class means we can name attributes however we want.

Download the entire file.

But this is Python, why do we need to copy stuff? In Python, methods are just functions, so we could almost get away with this:

class ProcessThreadPoolExecutor:
    ... # __init__, submit(), and shutdown() just as before
    __enter__ = ProcessPoolExecutor.__enter__
    __exit__ = ProcessPoolExecutor.__exit__
    map = ProcessPoolExecutor.map

Alas, it won't work – ProcessPoolExecutor map() calls super().map(), and object, the superclass of our executor, has no such method, which is why we had to change it to self._map() in our copy in the first place.

Functions #

Can this be done using only functions, though?

Theoretically no, since we need to implement the executor interface. Practically yes, since this is Python, where an "interface" just means having specific attributes, usually functions with specific signatures. For example, a module like this:

def init(max_threads=None, initializer=None, initargs=()):
    global _result_queue, _inner, _tasks, _result_handler
    _result_queue = multiprocessing.Queue()
    _inner = concurrent.futures.ProcessPoolExecutor(
        initializer=_init_process,
        initargs=(_result_queue, max_threads, initializer, initargs)
    )
    _tasks = {}
    _result_handler = threading.Thread(target=_handle_results)
    _result_handler.start()

def submit(fn, *args, **kwargs):
    outer = concurrent.futures.Future()
    task_id = id(outer)
    _tasks[task_id] = outer

    outer.set_running_or_notify_cancel()
    inner = _inner.submit(_submit, task_id, fn, *args, **kwargs)

    return outer

def _handle_results():
    for task_id, ok, result in iter(_result_queue.get, None):
        outer = _tasks.pop(task_id)
        if ok:
            outer.set_result(result)
        else:
            outer.set_exception(result)

def shutdown(wait=True):
    global _result_queue
    _inner.shutdown(wait=wait)
    if _result_queue:
        _result_queue.put(None)
        if wait:
            _result_handler.join()
        _result_queue.close()
        _result_queue = None

Like before, we need to copy map() with minor tweaks.

def _map(fn, *iterables, timeout=None, chunksize=1):
    # concurrent.futures._base.Executor.map

    if timeout is not None:
        end_time = timeout + time.monotonic()

    fs = [submit(fn, *args) for args in zip(*iterables)]

    def result_iterator():
        try:
            fs.reverse()
            while fs:
                if timeout is None:
                    yield _result_or_cancel(fs.pop())
                else:
                    yield _result_or_cancel(fs.pop(), end_time - time.monotonic())
        finally:
            for future in fs:
                future.cancel()
    return result_iterator()

def map(fn, *iterables, timeout=None, chunksize=1):
    # concurrent.futures.process.ProcessPoolExecutor.map

    if chunksize < 1:
        raise ValueError("chunksize must be >= 1.")

    results = _map(partial(_process_chunk, fn),
                   itertools.batched(zip(*iterables), chunksize),
                   timeout=timeout)
    return _chain_from_iterable_of_lists(results)

Behold, we can use the module itself as an executor:

>>> ptpe.init()
>>> ptpe.submit(int, '1').result()
1

Of note, everything that was an instance variable before is now a global variable; as a consequence, only one executor can exist at any given time, since there's only the one module.³ But it gets worse – calling init() a second time will clobber the state of the first executor, leading to all sorts of bugs; if we were serious, we'd prevent it somehow.

Also, some interfaces are more complicated than having the right functions; defining __enter__ and __exit__ is not enough to use a module in a with statement, since the interpreter looks them up on the class of the object, not on the object itself. We can work around this with an alternate "constructor" that returns a context manager:

@contextmanager
def init_cm(*args, **kwargs):
    init(*args, **kwargs)
    try:
        yield sys.modules[__name__]
    finally:
        shutdown()

>>> with ptpe.init_cm() as executor:
...     assert executor is ptpe
...     ptpe.submit(int, '2').result()
...
2

Download the entire file.

Liking this so far? Here's another article you might like:

When to use classes in Python? When your functions take the same arguments

Comparison #

So, how do the solutions stack up? Here's a summary:

	pros	cons
inheritance	least amount of code inherits new high level methods for free	assumes inherited high level methods use only the public API attribute names have to start with double underscores (minor)
composition	attributes can have any name (minor)	copies lots of code must be kept in sync with the interface
functions	?	copies lots of code must be kept in sync with the interface only one global executor at a time state is harder to discover alternate "constructor" to use as context manager (minor)

I may be a bit biased, but inheritance looks like a clear winner.

Composition over inheritance #

Given that favoring composition over inheritance is usually a good practice, it's worth discussing why inheritance won this time. I see three reasons:

Composition helps most when you have unrelated components that need to be flexible in response to an evolving business domain; that's not the case here, so we get all the drawbacks with none of the benefits.
The existing code is designed for inheritance.
We have a true is-a relationship – ProcessThreadPoolExecutor really is a ProcessPoolExecutor with extra behavior, and not just part of an arbitrary hierarchy.

For a different line of reasoning involving subtyping, check out Hillel Wayne's When to prefer inheritance to composition; he offers this rule of thumb:

So, here's when you want to use inheritance: when you need to instantiate both the parent and child classes and pass them to the same functions.

Forward compatibility #

The inheritance solution assumes map() and any future public ProcessPoolExecutor methods are implemented only in terms of other public methods. This assumption introduces a risk that updates may break our executor; this is lowered by two things:

concurrent.futures is in the standard library, which rarely does major rewrites of existing code, and never within a minor (X.Y) version; concurrent.futures exists in its current form since Python 3.2, released in 2011.
concurrent.futures is clearly designed for inheritance, even if mainly to enable internal reuse, and not explicitly documented.

As active mitigations, we can add a basic test suite (which we should do anyway), and document the supported Python versions explicitly (which we should do anyway if we were to release this on PyPI).

If concurrent.futures were not in the standard library, I'd probably go with the composition version instead, although as already mentioned, this wouldn't be free from upkeep either. Another option would be to upstream ProcessThreadPoolExecutor, so that it is maintained together with the code it depends on.

Global state #

The functions-only solution is probably the worst of the three, since it has all the downsides of composition, and significant limitations due to its use of global state.

We could avoid using globals by passing the state (process pool executor instance, result queue, etc.) as function arguments, but this breaks the executor interface, and makes for an awful user experience. We could group common arguments into a single object so there's only one argument to pass around; if you call that argument self, it becomes obvious that's just a class instance with extra steps.

Having to keep track of a bunch of related globals has enough downsides that even if you do want a module-level API, it's still worth using a class to group them, and exposing the methods of a global instance at module-level (like so); Brandon Rhodes discusses this at length in The Prebound Method Pattern.

Complexity #

While the code is somewhat complex, that's mostly intrinsic to the problem itself (what runs in the main vs. worker processes, passing results around, error handling, and so on), rather than due to our of use classes, which only affects how we refer to ProcessPoolExecutor methods and how we store state.

One could argue that copying a bunch of code doesn't increase complexity, but if you factor in keeping it up to date and tested, it's not exactly free either.

One could also argue that building our executor on top of ProcessPoolExecutor is increasing complexity, and in a way that's true – for example, we have two result queues and had to deal with dead workers too, which wouldn't be the case if we wrote it from scratch; but in turn, that would come with having to understand, maintain, and test 800+ lines of code of low level process management code. Sometimes, complexity I have to care about is more important that total complexity.

Debugging #

I have to come clean at this point – I use print debugging a lot 🙀 (especially if there are no tests yet, and sometimes from tests too); when that doesn't cut it, IPython's embed() usually provides enough interactivity to figure out what's going on.⁴

With the minimal test at the end of the file driving the executor, I used temporary print() calls in _submit(), _put_result(), and __handle_results() to check data is making its way through properly; if I expected the code to change more often, I'd replace them with permanent logging calls.

In addition, there were two debugging scripts in the benchmark file that I didn't show, one to automate killing workers at the right time, and one to make sure shutdown() waits any pending tasks.

So, does how we wrote the code change any of this? Not really, no; all the techniques above (and using a debugger too) apply equally well. If anything, using classes makes interactive debugging easier, since it's easier to discover state via autocomplete (with functions only, you have to know to look it up on the module).

If you've made it this far, you might like:

Learn by reading code: Python standard library design decisions explained

Try it out #

As I've said before, try it out – it only took ~10 minutes to convert the initial solution to the other two. In part, the right code structure is a matter feeling and taste, and both are educated by reading and writing lots of code. If you think there's a better way to do something, do it and see how it looks; it is a sort of deliberate practice.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Executor is an abstract base class only by convention: it is a base class (other classes are supposed to subclass it), and it is abstract (other classes are supposed to provide concrete implementations for some methods).

Python also allows formalizing abstract base classes using the abc module; see When to use classes in Python? When you repeat similar sets of functions for an example of this and other ways of achieving the same goal. ^[return]
For brevity, I'm using the version before dealing with dead workers; the final code is similar, but with a more involved __handle_results. ^[return]
This is almost true – we could "this is Python" our way deeper and reload the module while still keeping a reference to the old one, but that's just a round-about, unholy way of emulating class instances. ^[return]
Pro tip: you can use embed() as a breakpoint() hook: PYTHONBREAKPOINT=IPython.embed python myscript.py. ^[return]

ProcessThreadPoolExecutor: when I‍/‍O becomes CPU-bound

2025-05-07T18:00:00+00:00

So, you're doing some I‍/‍O bound stuff, in parallel.

Maybe you're scraping some websites – a lot of websites.

Maybe you're updating or deleting millions of DynamoDB items.

You've got your ThreadPoolExecutor, you've increased the number of threads and tuned connection limits... but after some point, it's just not getting any faster. You look at your Python process, and you see CPU utilization hovers above 100%.

You could split the work into batches and have a ProcessPoolExecutor run your original code in separate processes. But that requires yet more code, and a bunch of changes, which is no fun. And maybe your input is not that easy to split into batches.

If only we had an executor that worked seamlessly across processes and threads.

Well, you're in luck, since that's exactly what we're building today!

And even better, in a couple years you won't even need it anymore.

Contents

Establishing a baseline
Why not both?
Bonus: free threading

Establishing a baseline #

To measure things, we'll use a mock that pretends to do mostly I‍/‍O, with a sprinkling of CPU-bound work thrown in – a stand-in for something like a database connection, a Requests session, or a DynamoDB client.

class Client:
    io_time = 0.02
    cpu_time = 0.0008

    def method(self, arg):
        # simulate I/O
        time.sleep(self.io_time)

        # simulate CPU-bound work
        start = time.perf_counter()
        while time.perf_counter() - start < self.cpu_time:
            for i in range(100): i ** i

        return arg

We sleep() for the I‍/‍O, and do some math in a loop for the CPU stuff; it doesn't matter exactly how long each takes, as long I‍/‍O time dominates.

Real multi-threaded clients are usually backed by a shared connection pool, which allows for connection reuse (so you don't pay the cost of a new connection on each request) and multiplexing (so you can use the same connection for multiple concurrent requests, possible with protocols like HTTP/2 or newer). We could simulate this with a semaphore, but limiting connections is not relevant here – we're assuming the connection pool is effectively unbounded.

Since we'll use our client from multiple processes, we write an initializer function to set up a global, per-process client instance (remember, we want to share potential connection pools between threads); we can then pass the initializer to the executor constructor, along with any arguments we want to pass to the client. Similarly, we do the work through a function that uses this global client.

# this code runs in each worker process

client = None

def init_client(*args):
    global client
    client = Client(*args)

def do_stuff(*args):
    return client.method(*args)

Finally, we make a simple timing context manager:

@contextmanager
def timer():
    start = time.perf_counter()
    yield
    end = time.perf_counter()
    print(f"elapsed: {end-start:1.3f}")

...and put everything together in a function that measures how long it takes to do a bunch of work using a concurrent.futures executor:

def benchmark(executor, n=10_000, timer=timer, chunksize=10):
    with executor:
        # make sure all the workers are started,
        # so we don't measure their startup time
        list(executor.map(time.sleep, [0] * 200))

        with timer():
            values = list(executor.map(do_stuff, range(n), chunksize=chunksize))

        assert values == list(range(n)), values

Threads #

So, a ThreadPoolExecutor should suffice here, since we're mostly doing I‍/‍O, right?

>>> from concurrent.futures import *
>>> from bench import *
>>> init_client()
>>> benchmark(ThreadPoolExecutor(10))
elapsed: 24.693

More threads!

>>> benchmark(ThreadPoolExecutor(20))
elapsed: 12.405

Twice the threads, twice as fast. More!

>>> benchmark(ThreadPoolExecutor(30))
elapsed: 8.718

Good, it's still scaling linearly. MORE!

>>> benchmark(ThreadPoolExecutor(40))
elapsed: 8.638

...more?

>>> benchmark(ThreadPoolExecutor(50))
elapsed: 8.458
>>> benchmark(ThreadPoolExecutor(60))
elapsed: 8.430
>>> benchmark(ThreadPoolExecutor(70))
elapsed: 8.428

Problem: CPU becomes a bottleneck #

It's time we take a closer look at what our process is doing. I'd normally use the top command for this, but since the flags and output vary with the operating system, we'll implement our own using the excellent psutil library.

@contextmanager
def top():
    """Print information about current and child processes.

    RES is the resident set size. USS is the unique set size.
    %CPU is the CPU utilization. nTH is the number of threads.

    """
    process = psutil.Process()
    processes = [process] + process.children(True)
    for p in processes: p.cpu_percent()

    yield

    print(f"{'PID':>7} {'RES':>7} {'USS':>7} {'%CPU':>7} {'nTH':>7}")
    for p in processes:
        try:
            m = p.memory_full_info()
        except psutil.AccessDenied:
            m = p.memory_info()
        rss = m.rss / 2**20
        uss = getattr(m, 'uss', 0) / 2**20
        cpu = p.cpu_percent()
        nth = p.num_threads()
        print(f"{p.pid:>7} {rss:6.1f}m {uss:6.1f}m {cpu:7.1f} {nth:>7}")

And because it's a context manager, we can use it as a timer:

>>> init_client()
>>> benchmark(ThreadPoolExecutor(10), timer=top)
    PID     RES     USS    %CPU     nTH
  51395   35.2m   28.5m    38.7      11

So, what happens if we increase the number of threads?

>>> benchmark(ThreadPoolExecutor(20), timer=top)
    PID     RES     USS    %CPU     nTH
  13912   16.8m   13.2m    70.7      21
>>> benchmark(ThreadPoolExecutor(30), timer=top)
    PID     RES     USS    %CPU     nTH
  13912   17.0m   13.4m    99.1      31
>>> benchmark(ThreadPoolExecutor(40), timer=top)
    PID     RES     USS    %CPU     nTH
  13912   17.3m   13.7m   100.9      41

With more threads, the compute part of our I‍/‍O bound workload increases, eventually becoming high enough to saturate one CPU – and due to the global interpreter lock, one CPU is all we can use, regardless of the number of threads.¹

Processes? #

I know, let's use a ProcessPoolExecutor instead!

>>> benchmark(ProcessPoolExecutor(20, initializer=init_client))
elapsed: 12.374
>>> benchmark(ProcessPoolExecutor(30, initializer=init_client))
elapsed: 8.330
>>> benchmark(ProcessPoolExecutor(40, initializer=init_client))
elapsed: 6.273

Hmmm... I guess it is a little bit better.

More? More!

>>> benchmark(ProcessPoolExecutor(60, initializer=init_client))
elapsed: 4.751
>>> benchmark(ProcessPoolExecutor(80, initializer=init_client))
elapsed: 3.785
>>> benchmark(ProcessPoolExecutor(100, initializer=init_client))
elapsed: 3.824

OK, it's better, but with diminishing returns – there's no improvement after 80 processes, and even then, it's only 2.2x faster than the best time with threads, when, in theory, it should be able to make full use of all 4 CPUs.

Also, we're not making best use of connection pools (since we now have 80 of them, one per process), nor multiplexing (since we now have 80 connections, one per pool).

Problem: more processes, more memory #

But it gets worse!

>>> benchmark(ProcessPoolExecutor(80, initializer=init_client), timer=top)
    PID     RES     USS    %CPU     nTH
   2479   21.2m   15.4m    15.0       3
   2480   11.2m    6.3m     0.0       1
   2481   13.8m    8.5m     3.4       1
  ... 78 more lines ...
   2560   13.8m    8.5m     4.4       1

13.8 MiB * 80 ~= 1 GiB ... that is a lot of memory.

Now, there's some nuance to be had here.

First, on most operating systems that have virtual memory, code segment pages are shared between processes – there's no point in having 80 copies of libc or the Python interpreter in memory.

The unique set size is probably a better measurement than the resident set size, since it excludes memory shared between processes.² So, for the macOS output above,³ the actual usage is more like 8.5 MiB * 80 = 680 MiB.

Second, if you use the fork or forkserver start methods, processes also share memory allocated before the fork() via copy-on-write; for Python, this includes module code and variables. On Linux, the actual usage is 1.7 MiB * 80 = 136 MiB:

>>> benchmark(ProcessPoolExecutor(80, initializer=init_client), timer=top)
    PID     RES     USS    %CPU     nTH
 329801   17.0m    6.6m     5.1       3
 329802   13.3m    1.6m     2.1       1
  ... 78 more lines ...
 329881   13.3m    1.7m     2.0       1

However, it's important to note that's just a lower bound; memory allocated after fork() is not shared, and most real work will unavoidably allocate more memory.

Liking this so far? Here's another article you might like:

When to use classes in Python? When your functions take the same arguments

Why not both? #

One reasonable way of dealing with this would be to split the input into batches, one per CPU, and pass them to a ProcessPoolExecutor, which in turn runs the batch items using a ThreadPoolExecutor.⁴

But that would mean we need to change our code, and that's no fun.

If only we had an executor that worked seamlessly across processes and threads.

A minimal plausible solution #

In keeping with what has become tradition by now, we'll take an iterative, problem-solution approach; since we're not sure what to do yet, we start with the simplest thing that could possibly work.

We know we want a process pool executor that starts one thread pool executor per process, so let's deal with that first.

class ProcessThreadPoolExecutor(concurrent.futures.ProcessPoolExecutor):

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        super().__init__(
            initializer=_init_process,
            initargs=(max_threads, initializer, initargs)
        )

By subclassing ProcessPoolExecutor, we get the map() implementation for free, since the original is implemented in terms of submit().⁵ By going with the default max_workers, we get one process per CPU (which is what we want); we can add more arguments later if needed.

In a custom process initializer, we set up a global thread pool executor,⁶ and then call the process initializer provided by the user:

# this code runs in each worker process

_executor = None

def _init_process(max_threads, initializer, initargs):
    global _executor

    _executor = concurrent.futures.ThreadPoolExecutor(max_threads)

    if initializer:
        initializer(*initargs)

Likewise, submit() passes the work along to the thread pool executor:

class ProcessThreadPoolExecutor(concurrent.futures.ProcessPoolExecutor):
    # ...
    def submit(self, fn, *args, **kwargs):
        return super().submit(_submit, fn, *args, **kwargs)

# this code runs in each worker process
# ...
def _submit(fn, *args, **kwargs):
    return _executor.submit(fn, *args, **kwargs).result()

OK, that looks good enough; let's use it and see if it works:

def _do_stuff(n):
    print(f"doing: {n}")
    return n ** 2

if __name__ == '__main__':
    with ProcessThreadPoolExecutor() as e:
        print(list(e.map(_do_stuff, [0, 1, 2])))

 $ python ptpe.py
doing: 0
doing: 1
doing: 2
[0, 1, 4]

Wait, we got it on the first try?!

Let's measure that:

>>> from bench import *
>>> from ptpe import *
>>> benchmark(ProcessThreadPoolExecutor(30, initializer=init_client), n=1000)
elapsed: 6.161

Hmmm... that's unexpectedly slow... almost as if:

>>> multiprocessing.cpu_count()
4
>>> benchmark(ProcessPoolExecutor(4, initializer=init_client), n=1000)
elapsed: 6.067

Ah, because _submit() waits for the result() in the main thread of the worker process, this is just a ProcessPoolExecutor with extra steps.

But what if we send back the future object instead?

    def submit(self, fn, *args, **kwargs):
        return super().submit(_submit, fn, *args, **kwargs).result()

def _submit(fn, *args, **kwargs):
    return _executor.submit(fn, *args, **kwargs)

Alas:

$ python ptpe.py
doing: 0
doing: 1
doing: 2
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "concurrent/futures/process.py", line 210, in _sendback_result
    result_queue.put(_ResultItem(work_id, result=result,
  File "multiprocessing/queues.py", line 391, in put
    obj = _ForkingPickler.dumps(obj)
  File "multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "ptpe.py", line 42, in <module>
    print(list(e.map(_do_stuff, [0, 1, 2])))
  ...
TypeError: cannot pickle '_thread.RLock' object

The immediate cause of the error is that the future has a condition that has a lock that can't be pickled, because threading locks only make sense within the same process.

The deeper cause is that the future is not just data, but encapsulates state owned by the thread pool executor, and sharing state between processes requires extra work.

It may not seem like it, but this is a partial success: the work happens, we just can't get the results back. Not surprising, to be honest, it couldn't have been that easy.

Getting results #

If you look carefully at the traceback, you'll find a hint of how ProcessPoolExecutor gets its own results back from workers – a queue; the module docstring even has a neat data-flow diagram:

|======================= In-process =====================|== Out-of-process ==|

+----------+     +----------+       +--------+     +-----------+    +---------+
|          |  => | Work Ids |       |        |     | Call Q    |    | Process |
|          |     +----------+       |        |     +-----------+    |  Pool   |
|          |     | ...      |       |        |     | ...       |    +---------+
|          |     | 6        |    => |        |  => | 5, call() | => |         |
|          |     | 7        |       |        |     | ...       |    |         |
| Process  |     | ...      |       | Local  |     +-----------+    | Process |
|  Pool    |     +----------+       | Worker |                      |  #1..n  |
| Executor |                        | Thread |                      |         |
|          |     +----------- +     |        |     +-----------+    |         |
|          | <=> | Work Items | <=> |        | <=  | Result Q  | <= |         |
|          |     +------------+     |        |     +-----------+    |         |
|          |     | 6: call()  |     |        |     | ...       |    |         |
|          |     |    future  |     |        |     | 4, result |    |         |
|          |     | ...        |     |        |     | 3, except |    |         |
+----------+     +------------+     +--------+     +-----------+    +---------+

Now, we could probably use the same queue somehow, but it would involve touching a lot of (private) internals.⁷ Instead, let's use a separate queue:

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        self.__result_queue = multiprocessing.Queue()
        super().__init__(
            initializer=_init_process,
            initargs=(self.__result_queue, max_threads, initializer, initargs)
        )

On the worker side, we make it globally accessible:

# this code runs in each worker process

_executor = None
_result_queue = None

def _init_process(queue, max_threads, initializer, initargs):
    global _executor, _result_queue

    _executor = concurrent.futures.ThreadPoolExecutor(max_threads)
    _result_queue = queue

    if initializer:
        initializer(*initargs)

...so we can use it from a task callback registered by _submit():

def _submit(fn, *args, **kwargs):
    task = _executor.submit(fn, *args, **kwargs)
    task.add_done_callback(_put_result)

def _put_result(task):
    if exception := task.exception():
        _result_queue.put((False, exception))
    else:
        _result_queue.put((True, task.result()))

Back in the main process, we handle the results in a thread:

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        # ...
        self.__result_handler = threading.Thread(target=self.__handle_results)
        self.__result_handler.start()

    def __handle_results(self):
        for ok, result in iter(self.__result_queue.get, None):
            print(f"{'ok' if ok else 'error'}: {result}")

Finally, to stop the handler, we use None as a sentinel on executor shutdown:

    def shutdown(self, wait=True):
        super().shutdown(wait=wait)
        if self.__result_queue:
            self.__result_queue.put(None)
            if wait:
                self.__result_handler.join()
            self.__result_queue.close()
            self.__result_queue = None

Let's see if it works:

$ python ptpe.py
doing: 0
ok: [0]
doing: 1
ok: [1]
doing: 2
ok: [4]
Traceback (most recent call last):
  File "concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
AttributeError: 'NoneType' object has no attribute 'result'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
AttributeError: 'NoneType' object has no attribute 'cancel'

Yay, the results are making it to the handler!

The error happens because instead of returning a Future, our submit() returns the result of _submit(), which is always None.

Fine, we'll make our own futures #

But submit() must return a future, so we make our own:

    def __init__(self, max_threads=None, initializer=None, initargs=()):
        # ...
        self.__tasks = {}
        # ...

    def submit(self, fn, *args, **kwargs):
        outer = concurrent.futures.Future()
        task_id = id(outer)
        self.__tasks[task_id] = outer

        outer.set_running_or_notify_cancel()
        inner = super().submit(_submit, task_id, fn, *args, **kwargs)

        return outer

In order to map results to their futures, we can use a unique identifier; the id() of the outer future should do, since it is unique for the object's lifetime.

We pass the id to _submit(), then to _put_result() as an attribute on the future, and finally back in the queue with the result:

def _submit(task_id, fn, *args, **kwargs):
    task = _executor.submit(fn, *args, **kwargs)
    task.task_id = task_id
    task.add_done_callback(_put_result)

def _put_result(task):
    if exception := task.exception():
        _result_queue.put((task.task_id, False, exception))
    else:
        _result_queue.put((task.task_id, True, task.result()))

Back in the result handler, we find the maching future, and set the result accordingly:

    def __handle_results(self):
        for task_id, ok, result in iter(self.__result_queue.get, None):
            outer = self.__tasks.pop(task_id)
            if ok:
                outer.set_result(result)
            else:
                outer.set_exception(result)

And it works:

$ python ptpe.py
doing: 0
doing: 1
doing: 2
[0, 1, 4]

I mean, it really works:

>>> benchmark(ProcessThreadPoolExecutor(10, initializer=init_client))
elapsed: 6.220
>>> benchmark(ProcessThreadPoolExecutor(20, initializer=init_client))
elapsed: 3.397
>>> benchmark(ProcessThreadPoolExecutor(30, initializer=init_client))
elapsed: 2.575
>>> benchmark(ProcessThreadPoolExecutor(40, initializer=init_client))
elapsed: 2.664

3.3x is not quite the 4 CPUs my laptop has, but it's pretty close, and much better than the 2.2x we got from processes alone.

The executor code so far.

Death becomes a problem #

I wonder what happens when a worker process dies.

For example, the initializer can fail:

>>> executor = ProcessPoolExecutor(initializer=divmod, initargs=(0, 0))
>>> executor.submit(int).result()
Exception in initializer:
Traceback (most recent call last):
  ...
ZeroDivisionError: integer division or modulo by zero
Traceback (most recent call last):
  ...
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

...or a worker can die some time later, which we can help along with a custom timer:⁸

@contextmanager
def terminate_child(interval=1):
    threading.Timer(interval, psutil.Process().children()[-1].terminate).start()
    yield

>>> executor = ProcessPoolExecutor(initializer=init_client)
>>> benchmark(executor, timer=terminate_child)
[ one second later ]
Traceback (most recent call last):
  ...
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Now let's see our executor:

>>> executor = ProcessThreadPoolExecutor(30, initializer=init_client)
>>> benchmark(executor, timer=terminate_child)
[ one second later ]
[ ... ]
[ still waiting ]
[ ... ]
[ hello? ]

If the dead worker is not around to send back results, its futures never get completed, and map() keeps waiting until the end of time, when the expected behavior is to detect when this happens, and fail all pending tasks with BrokenProcessPool.

Before we do that, though, let's address a more specific issue.

If map() hasn't finished submitting tasks when the worker dies, inner fails with BrokenProcessPool, which right now we're ignoring entirely. While we don't need to do anything about it in particular because it gets covered by handling the general case, we should still propagate all errors to the outer task anyway.

    def submit(self, fn, *args, **kwargs):
        # ...
        inner = super().submit(_submit, task_id, fn, *args, **kwargs)
        inner.task_id = task_id
        inner.add_done_callback(self.__handle_inner)

        return outer

    def __handle_inner(self, inner):
        task_id = inner.task_id
        if exception := inner.exception():
            if outer := self.__tasks.pop(task_id, None):
                outer.set_exception(exception)

This fixes the case where a worker dies almost instantly:

>>> executor = ProcessThreadPoolExecutor(30, initializer=init_client)
>>> benchmark(executor, timer=lambda: terminate_child(0))
Traceback (most recent call last):
  ...
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

For the general case, we need to check if the executor is broken – but how? We've already decided we don't want to depend on internals, so we can't use ProcessPoolExecutor._broken. Maybe we can submit a dummy task and see if it fails instead:

    def __check_broken(self):
        try:
            super().submit(int).cancel()
        except concurrent.futures.BrokenExecutor as e:
            return type(e)(str(e))
        except RuntimeError as e:
            if 'shutdown' not in str(e):
                raise
        return None

Using it is a bit involved, but not completely awful:

    def __handle_results(self):
        last_broken_check = time.monotonic()

        while True:
            now = time.monotonic()
            if now - last_broken_check >= .1:
                if exc := self.__check_broken():
                    break
                last_broken_check = now

            try:
                value = self.__result_queue.get(timeout=.1)
            except queue.Empty:
                continue

            if not value:
                return

            task_id, ok, result = value
            if outer := self.__tasks.pop(task_id, None):
                if ok:
                    outer.set_result(result)
                else:
                    outer.set_exception(result)

        while self.__tasks:
            try:
                _, outer = self.__tasks.popitem()
            except KeyError:
                break
            outer.set_exception(exc)

When there's a steady stream of results coming in, we don't want to check too often, so we enforce a minimum delay between checks. When there are no results coming in, we want to check regularly, so we use the Queue.get() timeout to avoid waiting forever. If the check fails, we break out of the loop and fail the pending tasks. Like so:

>>> executor = ProcessThreadPoolExecutor(30, initializer=init_client)
>>> benchmark(executor, timer=terminate_child)
Traceback (most recent call last):
  ...
concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore

So, yeah, I think we're done. Here's the final executor and benchmark code.

Some features left as an exercise for the reader:

providing a ThreadPoolExecutor initializer
using other start methods
shutdown()'s cancel_futures

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Bonus: free threading #

You may have heard people being excited about the experimental free threading support added in Python 3.13, which allows running Python code on multiple CPUs.

And for good reason:

$ python3.13t
Python 3.13.2 experimental free-threading build
>>> from concurrent.futures import *
>>> from bench import *
>>> init_client()
>>> benchmark(ThreadPoolExecutor(30))
elapsed: 8.224
>>> benchmark(ThreadPoolExecutor(40))
elapsed: 6.193
>>> benchmark(ThreadPoolExecutor(120))
elapsed: 2.323

3.6x over to the GIL version, with none of the shenanigans in this article!

Alas, packages with extensions need to be updated to support it:

>>> import psutil
zsh: segmentation fault  python3.13t

...but the ecosystem is slowly catching up.

If you've made it this far, you might like:

This is not interview advice: a priority-expiry LRU cache in Python without heaps or trees

At least, all we can use for pure-Python code. I‍/‍O always releases the global interpreter lock, and so do some extension modules. ^[return]
The psutil documentation for memory_full_info() explains the difference quite nicely and links to further resources, because good libraries educate. ^[return]
You may have to run Python as root to get the USS of child processes. ^[return]
And no, asyncio is not a solution, since the event loop runs in a single thread, so you'd still need to run one event loop per CPU in dedicated processes. ^[return]
We could have used composition instead, but then we'd have to implement the full Executor interface, defining each method explicitly to delegate to the inner process pool executor, and keep things up to date when the interface gets new methods (and we'd have no way to trick the inner executor's map() to use our submit(), so we'd have to implement it from scratch).

Yet another option would be to use both inheritance and composition – inherit the Executor base class directly for the common methods (assuming they're defined there and not in subclasses), and delegate to the inner executor only where needed (likely just map() and shutdown()). But, the only difference from the current code would be that it'd say self._inner instead of super() in a few places, so it's not really worth it, in my opinion. ^[return]
A previous version of this code attempted to shutdown() the thread pool executor using atexit, but since atexit functions run after non-daemon threads finish, it wasn't actually doing anything. Not shutting it down seems to work for now, but we may still need do it to support shutdown(cancel_futures=True) properly. ^[return]
Check out nilp0inter/threadedprocess for an idea of what that looks like. ^[return]
pkill -fn '[Pp]ython' would've done it too, but it gets tedious if you do it a lot, and it's a different command on Windows. ^[return]

reader 3.16 released – Archived feed

2024-12-09T18:00:00+00:00

Hi there!

I'm happy to announce version 3.16 of reader, a Python feed reader library.

What's new? #

Here are the highlights since reader 3.15.

Archived feed #

It is now possible to archive selected entries to a special "archived" feed, so they can be preserved once the original feed is deleted; this is similar to the Tiny Tiny RSS feature of the same name (which is where I got the idea in the first place).

At high level, this is available in the web app as an "archive all" button that archives currently-visible entries for a feed; in the re-design, the plan is to archive important entries by default as part of the "delete feed" confirmation page. It may also be a good idea to make this the default through a plugin before then, so that user interaction is not required for it to happen.

At low level, this is enabled by archive_entries() (I finally gave up and added a utils module 😅), and the copy_entry() method.

Entry source #

reader now parses and stores the entry source, containing metadata about the source feed if the entry is a copy – this can be the case when a feed aggregates articles from other feeds. The source URL can be used for filtering entries, and its title is part of the feed title during searches.

This was a side-project for the archived feed functionality, since the source of archived entries gets set to their original feed.

Bug fixes #

Also during archived feed implementation, I found out foreign keys were disabled in threads other than the one that created the reader instance, so e.g. deleting a feed from another thread would not delete its entries. This is now fixed.

The bug existed since multi-threaded use became allowed in version 2.15. Serving the web application with the serve command is known to be affected; serving it without threads (e.g. the default uWSGI configuration) should not be affected.

Attention

Your database may be in an inconsistent state. The migration to 3.16 will check for this on first use; issues will be reported as a FOREIGN KEY constraint failed storage integrity error; see the changelog for details.

That's it for now. For more details, see the full changelog.

Want to contribute? Check out the docs and the roadmap.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

reader is a library
reader is for the long term
reader is extensible
reader is stable (within reason)
reader is simple to use; API matters
reader features work well together
reader is tested
reader is documented
reader has minimal dependencies

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

reader 3.15 released – Retry-After

2024-11-13T18:00:00+00:00

Hi there!

I'm happy to announce version 3.15 of reader, a Python feed reader library.

What's new? #

Here are the highlights since reader 3.13.

Retry-After #

Now that it supports scheduled updates, reader can honor the Retry-After HTTP header sent with 429 Too Many Requests or 503 Service Unavailable responses.

Adding this required an extensive rework of the parser internal API, but I'd say it was worth it, since we're getting quite close to it becoming stable.

Next up in HTTP compliance is to do more on behalf of the user: bump the update interval on repeated throttling, and handle gone and redirected feeds accordingly.

Faster tag filters, feed slugs #

OR-only tag filters like get_feeds(tags=[['one', 'two']]) now use an index.

This is useful for maintaining a reverse mapping to feeds/entries, like the feed slugs recipe does to add support for user-defined short URLs:

>>> url = 'https://death.andgravity.com/_feed/index.xml'
>>> reader.set_feed_slug(url, 'andgravity')
>>> reader.get_feed_by_slug('andgravity')
Feed(url='https://death.andgravity.com/_feed/index.xml', ...)

(Interested in adopting this recipe as a real plugin? Submit a pull request!)

enclosure_tags improvements #

The enclosure_tags plugin fixes ID3 tags for MP3 enclosures like podcasts.

I've changed the implementation to rewrite tags on the fly, instead of downloading the entire file, rewriting tags, and then sending it to the user; this should allow browsers to display accurate download progress.

Some other, smaller improvements:

Set genre to Podcast if the feed has any tag containing "podcast".
Prefer feed user title to feed title if available.
Use feed title as artist, instead of author.

Using the installed feedparser #

Because feedparser makes PyPI releases at a lower cadence, reader has been using a vendored version of feedparser's develop branch for some time. It is now possible to opt out of this behavior and make reader use the installed feedparser package.

Python versions #

reader 3.14 (released back in July) adds support for Python 3.13.

Upcoming changes #

Replacing Requests with HTTPX #

reader relies on Requests to retrieve feeds from the internet; among others, it replaces feedparser's use of urllib to make it easier to write plugins.

However, Requests has a few issues that may never get fixed because it is in a feature-freeze – mainly the lack of default timeouts, underpowered response hooks, and no request hooks, all of which I had to work around in reader code.

So, I've been looking into using HTTPX instead.

Some reasons to use HTTPX:

largely Requests-compatible API and feature set
while the ecosystem is probably not comparable, it is actively maintained, popular enough, and the basics (mocking, auth) are there
strict timeouts by default (and more kinds than Requests)
request/response hooks
URL normalization (needed by the parser)

Bad reasons to move away from Requests:

lack of async support – I have no plan to use async in reader at this point
lack of HTTP/2 support – coming soon in urllib3 (and by extension, Requests?); also, reader makes rare requests to many different hosts, I'm not sure it would benefit all that much from HTTP/2
lack of Brotli/Zstandard compresson support – urllib3 already supports them

Reasons to not move to HTTPX:

not 1.0 yet (but coming soon)
not as battle-tested as Requests (but can use urllib3 as transport)

So, when is this happening? Nothing's actually burning, so soon™, but not that soon; watch #360 if you're interested in this.

That's it for now. For more details, see the full changelog.

Want to contribute? Check out the docs and the roadmap.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

reader is a library
reader is for the long term
reader is extensible
reader is stable (within reason)
reader is simple to use; API matters
reader features work well together
reader is tested
reader is documented
reader has minimal dependencies

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

reader 3.13 released – scheduled updates

2024-06-21T17:00:00+00:00

Hi there!

I'm happy to announce version 3.13 of reader, a Python feed reader library.

What's new? #

Here are the highlights since reader 3.12.

Scheduled updates #

reader now allows updating feeds at different rates via scheduled updates.

The way it works is quite simple: each feed has an update interval that determines when the feed should be updated next; calling update_feeds(scheduled=True) updates only feeds that should be updated at or before the current time.

The interval can be configured by the user globally or per-feed through the .reader.update tag. In addition, you can specify a jitter; for an interval of 24 hours, a jitter of 0.25 means the update will occur any time in the first 6 hours of the interval.

In the future, the same mechanism will be used to handle 429 Too Many Requests.

Improved documentation #

As part of rewriting the Updating feeds user guide section to talk about scheduled updates, I've added a new section about being polite to servers.

Also, we have a new recipe for adding custom headers when retrieving feeds.

mark_as_read reruns #

You can now re-run the mark_as_read plugin for existing entries by adding the .reader.mark-as-read.once tag to a feed. Thanks to Michael Han for the pull request!

That's it for now. For more details, see the full changelog.

Want to contribute? Check out the docs and the roadmap.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

reader is a library
reader is for the long term
reader is extensible
reader is stable (within reason)
reader is simple to use; API matters
reader features work well together
reader is tested
reader is documented
reader has minimal dependencies

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

reader 3.12 released – split search index

2024-03-07T18:00:00+00:00

Hi there!

I'm happy to announce version 3.12 of reader, a Python feed reader library.

What's new? #

Here are the highlights since reader 3.10.

Split the search index into a separate database #

The full-text search index can get almost as large as the actual data, so I've split it into a separate, attached database, which allows backing up only the main database.

(I stole this idea from One process programming notes (with Go and SQLite).)

Change tracking internal API #

To support the search index split, Storage got a change tracking API that allows search implementations to keep in sync with text content changes.

This is a first step towards search backends that aren't tightly-coupled to a storage. For example, the SQLite storage uses its FTS5 extension for search, and a PostgreSQL storage can use its own native support; the new API allows either storage to use something like Elasticsearch. (There's still no good way for search to filter/sort results without storage cooperation, so more work is needed here.)

Also, it lays some of the groundwork for searchable tag values by having tag support already built into the API.

Here's how change tracking works (long version):

Each entry has a 16 byte random sequence that changes when its text changes.
Sequence changes get recorded and are available through the API.
Search update() processes pending changes and marks them as done.

While simple on the surface, this prevents a lot of potential concurrency issues that needed special handling before. For example, what if an entry changes during pre-processing, before it is added to the search index? You could use a transaction, but this may keep the database locked for too long. Also, what about search backends where you don't have transactions?

I used Hypothesis and property-based testing to validate the model, so I'm ~99% sure it is correct. A real model checker like TLA+ or Alloy may have been a better tool for it, but I don't know how to use one at this point.

Filter by entry tags #

It is now possible to filter entries by entry tags: get_entries(tags=['tag']).

I did this to see how it would look to implement the has_enclosures get_entries() argument as a plugin (it is possible, but not really worth it).

SQLite storage improvements #

As part of a bigger storage refactoring, I made a few small improvements:

Enable write-ahead logging only once, when the database is created.
Vacuum the main database after migrations.
Require at least SQLite 3.18, since it was required by update_search() anyway.

Python versions #

reader 3.11 (released back in December) adds support for Python 3.12.

That's it for now. For more details, see the full changelog.

Want to contribute? Check out the docs and the roadmap.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

reader is a library
reader is for the long term
reader is extensible
reader is stable (within reason)
reader is simple to use; API matters
reader features work well together
reader is tested
reader is documented
reader has minimal dependencies

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

This is not interview advice: a priority-expiry LRU cache in Python without heaps or trees

2024-01-26T10:00:00+00:00

It's not your fault I got nerdsniped, but that doesn't matter.

Hi, I'm Adrian, and today we're implementing a least recently used cache with priorities and expiry, using only the Python standard library.

This is a bIG TEch CoDINg InTerVIEW problem, so we'll work hard to stay away from the correct™ data structures – no heaps, no binary search trees – but we'll end up with a decent solution anyway!

Contents

Requirements
A minimal plausible solution
Problem: expired items should go first...
- Problem: name PriorityQueue is not defined
Problem: ...low priority items second
- Problem: we're deleting items in three places
Problem: ...least recently used items last
- functools.lru_cache()
- OrderedDict
Problem: our priority queue is slow
Problem: Sorted Containers is not in stdlib
- Logarithmic time
- bisect, redux
Conclusion

Requirements #

So you're at an interview and have to implement a priority-expiry LRU cache.

Maybe you get more details, but maybe the problem is deliberately vague; either way, we can reconstruct the requirements from the name alone.

A cache is something that stores data for future reuse, usually the result of a slow computation or retrieval. Each piece of data has an associated key used to retrieve it. Most caches are bounded in some way, for example by limiting the number of items.

The other words have to do with eviction – how and when items are removed.

Each item has a maximum age – items that go past that are expired, so we don't return them. It stands to reason this does not depend on their priority or how full the cache is.
Each item has a priority – when the cache fills up, we evict items with lower priority before those with higher priority.
All other things being equal, we evict items least recently used relative to others.

The problem may specify an API; we can reconstruct that from first principles too. Since the cache is basically a key-value store, we can get away with two methods:

set(key, value, maxage, priority)
get(key) -> value or None

The problem may also suggest:

delete(key) – allows users to invalidate items for reasons external to the cache; not strictly necessary, but we'll end up with it as part of refactoring
evict(now) – not strictly necessary either, but hints eviction is a separate bit of logic, and may come in handy for testing

Types deserve their own discussion:

key – usually, the key is a string, but we can relax this to any hashable value
value – for an in-memory cache, any kind of object is fine
maxage and priority – a number should do for these; a float is more general, but an integer may allow a simpler solution; limits on these are important too, as we'll see soon enough

Tip

Your interviewer may be expecting you to uncover some of these details through clarifying questions. Be sure to think out loud and state your assumptions.

A minimal plausible solution #

I'm sure there are a lot smart people out there that can Think Really Hard™ and just come up with a solution, but I'm not one of them, so we'll take an iterative, problem-solution approach to this.

Since right now we don't have any solution, we start with the simplest thing that could possibly work¹ – a basic cache with no fancy eviction and no priorities; we can then write some tests against that, to know if we break anything going forward.

Tip

If during an interview you don't know what to do and choose to work up from the naive solution, make it very clear that's what you're doing. Your interviewer may give you hints to help you skip that.

A class holds all the things we need and gives us something to stick the API on:

class Cache:

    def __init__(self, maxsize, time=time.monotonic):
        self.maxsize = maxsize
        self.time = time

        self.cache = {}

And this is our first algorithmic choice: a dict (backed by a hash table) provides average O(1) search / insert / delete.

Tip

Given the problem we're solving, and the context we're solving it in, we have to talk about time complexity. Ned Batchelder's Big-O: How Code Slows as Data Grows provides an excellent introduction (text and video available).

set() leads to more choices:

    def set(self, key, value, *, maxage=10, priority=0):
        now = self.time()

        if key in self.cache:
            self.cache.pop(key)
        elif len(self.cache) >= self.maxsize:
            self.evict(now)

        expires = now + maxage
        item = Item(key, value, expires, priority)

        self.cache[key] = item

First, we evict items only if there's no more room left. (There are other ways of doing this; for example, evicting expired items periodically minimizes memory usage.)

Second, if the key is already in the cache, we remove and insert it again, instead of updating things in place. This way, there's only one code path for setting items, which will make it a lot easier to keep multiple data structures in sync later on.

We use a named tuple to store the parameters associated with a key:

class Item(NamedTuple):
    key: object
    value: object
    expires: int
    priority: int

For now, we just evict an arbitrary item; in a happy little coincidence, dicts preserve insertion order, so when iterating over the cache, the oldest key is first.

    def evict(self, now):
        if not self.cache:
            return

        key = next(iter(self.cache))
        del self.cache[key]

Finally, get() is trivial:

    def get(self, key):
        item = self.cache.get(key)
        if not item:
            return None

        if self.time() >= item.expires:
            return None

        return item.value

With everything in place, here's the first test:

def test_basic():
    cache = Cache(2, FakeTime())

    assert cache.get('a') == None

    cache.set('a', 'A')
    assert cache.get('a') == 'A'

    cache.set('b', 'B')
    assert cache.get('a') == 'A'
    assert cache.get('b') == 'B'

    cache.set('c', 'C')
    assert cache.get('a') == None
    assert cache.get('b') == 'B'
    assert cache.get('c') == 'C'

To make things predictable, we inject a fake time implementation:

class FakeTime:
    def __init__(self, now=0):
        self.now = now
    def __call__(self):
        return self.now

Problem: expired items should go first... #

Following from the requirements, there's an order in which items get kicked out: first expired (lowest expiry time), then lowest priority, and only then least recently used. So, we need a data structure that can efficiently remove the smallest element.

Turns out, there's an abstract data type for that called a priority queue;² for now, we'll honor its abstract nature and not bother with an implementation.

        self.cache = {}
        self.expires = PriorityQueue()

Since we need the item with the lowest expiry time, we need a way to get back to the item; an (expires, key) tuple should do fine – since tuples compare lexicographically, it'll be like comparing by expires alone, but with key along for the ride; in set(), we add:

        self.cache[key] = item
        self.expires.push((expires, key))

You may be tempted (like I was) to say "hey, the item's already a tuple, if we make expires the first field, we can use the item itself", but let's delay optimizations until we have and understand a full solution – make it work, make it right, make it fast.

Still in set(), if the key is already in the cache, we also remove and insert it from the expires queue, so it's added back with the new expiry time.

        if key in self.cache:
            item = self.cache.pop(key)
            self.expires.remove((item.expires, key))
            del item
        elif len(self.cache) >= self.maxsize:
            self.evict(now)

Moving on to evicting things; for this, we need two operations: first peek at the item that expires next to see if it's expired, then, if it is, pop it from the queue. (Another choice: we only have to evict one item, but evict all expired ones.)

    def evict(self, now):
        if not self.cache:
            return

        initial_size = len(self.cache)

        while self.cache:
            expires, key = self.expires.peek()
            if expires > now:
                break
            self.expires.pop()
            del self.cache[key]

        if len(self.cache) == initial_size:
            _, key = self.expires.pop()
            del self.cache[key]

If there are no expired items, we still have to make room for the one item; since we're not handling priorities yet, we'll evict the item that expires next a little early.

Problem: name PriorityQueue is not defined #

OK, to get the code working again, we need a PriorityQueue class. It doesn't need to be fast, we can deal with that after we finish everything else; for now, let's just keep our elements in a plain list.

class PriorityQueue:

    def __init__(self):
        self.data = []

The easiest way to get the smallest value is to keep the list sorted; the downside is that push() is now O(n log n) – although, because the list is always sorted, it can be as good as O(n) depending on the implementation.

    def push(self, item):
        self.data.append(item)
        self.data.sort()

This makes peek() and pop() trivial; still, pop() is O(n), because it shifts all the items left by one position.

    def peek(self):
        return self.data[0]

    def pop(self):
        rv = self.data[0]
        self.data[:1] = []
        return rv

remove() is just as simple, and just as O(N), because it first needs to find the item, and then shift the ones after it to cover the gap.

    def remove(self, item):
        self.data.remove(item)

We didn't use the is empty operation, but it should be O(1) regardless of implementation, so let's throw it in anyway:

    def __bool__(self):
        return bool(self.data)

OK, let's wrap up with a quick test:

def test_priority_queue():
    pq = PriorityQueue()
    pq.push(1)
    pq.push(3)
    pq.push(2)

    assert pq
    assert pq.peek() == 1
    assert pq.pop() == 1
    assert pq.peek() == 2

    assert pq.remove(3) is None

    assert pq
    assert pq.peek() == 2
    with pytest.raises(ValueError):
        pq.remove(3)

    assert pq.pop() == 2

    assert not pq
    with pytest.raises(IndexError):
        pq.peek()
    with pytest.raises(IndexError):
        pq.pop()

Now the existing tests pass, and we can add more – first, that expired items are evicted (note how we're moving the time forward):

def test_expires():
    cache = Cache(2, FakeTime())

    cache.set('a', 'A', maxage=10)
    cache.set('b', 'B', maxage=20)
    assert cache.get('a') == 'A'
    assert cache.get('b') == 'B'

    cache.time.now = 15
    assert cache.get('a') == None
    assert cache.get('b') == 'B'

    cache.set('c', 'C')
    assert cache.get('a') == None
    assert cache.get('b') == 'B'
    assert cache.get('c') == 'C'

Second, that setting an existing item changes its expire time:

def test_update_expires():
    cache = Cache(2, FakeTime())

    cache.set('a', 'A', maxage=10)
    cache.set('b', 'B', maxage=10)

    cache.time.now = 5
    cache.set('a', 'X', maxage=4)
    cache.set('b', 'Y', maxage=6)
    assert cache.get('a') == 'X'
    assert cache.get('b') == 'Y'

    cache.time.now = 10
    assert cache.get('a') == None
    assert cache.get('b') == 'Y'

Problem: ...low priority items second #

Next up, kick out items by priority – shouldn't be too hard, right?

In __init__(), add another priority queue for priorities:

        self.cache = {}
        self.expires = PriorityQueue()
        self.priorities = PriorityQueue()

In set(), add new items to the priorities queue:

        self.cache[key] = item
        self.expires.push((expires, key))
        self.priorities.push((priority, key))

...and remove already-cached items:

        if key in self.cache:
            item = self.cache.pop(key)
            self.expires.remove((item.expires, key))
            self.priorities.remove((item.priority, key))
            del item

In evict(), remove expired items from the priorities queue:

        while self.cache:
            expires, key = self.expires.peek()
            if expires > now:
                break
            self.expires.pop()
            item = self.cache.pop(key)
            self.priorities.remove((item.priority, key))

...and finally, if none are expired, remove the one with the lowest priority:

        if len(self.cache) == initial_size:
            _, key = self.priorities.pop()
            item = self.cache.pop(key)
            self.expires.remove((item.expires, key))

Add one test for eviction by priority:

def test_priorities():
    cache = Cache(2, FakeTime())

    cache.set('a', 'A', priority=1)
    cache.set('b', 'B', priority=0)
    assert cache.get('a') == 'A'
    assert cache.get('b') == 'B'

    cache.set('c', 'C')
    assert cache.get('a') == 'A'
    assert cache.get('b') == None
    assert cache.get('c') == 'C'

...and one for updating the priority of existing items:

def test_update_priorities():
    cache = Cache(2, FakeTime())

    cache.set('a', 'A', priority=1)
    cache.set('b', 'B', priority=0)
    cache.set('b', 'Y', priority=2)

    cache.set('c', 'C')
    assert cache.get('a') == None
    assert cache.get('b') == 'Y'
    assert cache.get('c') == 'C'

Problem: we're deleting items in three places #

I said we'll postpone performance optimizations until we have a complete solution, but I have a different kind of optimization in mind – for readability.

We're deleting items in three slightly different ways, careful to keep three data structures in sync each time; it would be nice to do it only once. While a bit premature, through the magic of having written the article already, I'm sure it will pay off.

    def delete(self, key):
        *_, expires, priority = self.cache.pop(key)
        self.expires.remove((expires, key))
        self.priorities.remove((priority, key))

Sure, eviction is twice as slow, but the complexity stays the the same – the constant factor in O(2n) gets removed, leaving us with O(n). If needed, we can go back to the unpacked version once we have a reasonably efficient implementation (that's what tests are for).

Deleting already-cached items is shortened to:

        if key in self.cache:
            self.delete(key)

Idem for the core eviction logic:

        while self.cache:
            expires, key = self.expires.peek()
            if expires > now:
                break
            self.delete(key)

        if len(self.cache) == initial_size:
            _, key = self.priorities.peek()
            self.delete(key)

Neat!

Problem: ...least recently used items last #

So, how does one implement a least recently used cache?

We could google it... or, we could look at an existing implementation.

functools.lru_cache() #

Standard library functools.lru_cache() comes to mind first; let's have a look.

Tip

You can read the code of stdlib modules by following the Source code: link at the top of each documentation page.

lru_cache() delegates to _lru_cache_wrapper(), which sets up a bunch of variables to be used by nested functions.³ Among the variables is a cache dict and a doubly linked list where nodes are [prev, next, key, value] lists.⁴

And that's the answer – a doubly linked list allows tracking item use in O(1): each time a node is used, remove it from its current position and plop it at the "recently used" end; whatever's at the other end will be the least recently used item.

Note that, unlike lru_cache(), we need one doubly linked list for each priority.

But, before making Item mutable and giving it prev/next links, let's dive deeper.

OrderedDict #

If you search the docs for "LRU", the next result after lru_cache() is OrderedDict.

Some differences from dict still remain: [...] The OrderedDict algorithm can handle frequent reordering operations better than dict. [...] this makes it suitable for implementing various kinds of LRU caches.

Specifically:

OrderedDict has a move_to_end() method to efficiently reposition an element to an endpoint.

Since dicts preserve insertion order, you can use d[k] = d.pop(k) to move items to the end... What makes move_to_end() better, then? This comment may shed some light:

    # The internal self.__map dict maps keys to links in a doubly linked list.

Indeed, move_to_end() does exactly what we described above – this is good news, it means we don't have to do it ourselves!

So, we need one OrderedDict (read: doubly linked list) for each priority, but still need to keep track the lowest priority:

        self.cache = {}
        self.expires = PriorityQueue()
        self.priority_buckets = {}
        self.priority_order = PriorityQueue()

Handling priorities in set() gets a bit more complicated:

        priority_bucket = self.priority_buckets.get(priority)
        if not priority_bucket:
            priority_bucket = self.priority_buckets[priority] = OrderedDict()
            self.priority_order.push(priority)
        priority_bucket[key] = None

But now we can finally evict the least recently used item:

        if len(self.cache) == initial_size:
            priority = self.priority_order.peek()
            priority_bucket = self.priority_buckets.get(priority)
            key = next(iter(priority_bucket))
            self.delete(key)

In delete(), we're careful to get rid of empty buckets:⁵

        priority_bucket = self.priority_buckets[priority]
        del priority_bucket[key]
        if not priority_bucket:
            del self.priority_buckets[priority]
            self.priority_order.remove(priority)

Existing tests pass again, and we can add a new (still failing) one:

def test_lru():
    cache = Cache(2, FakeTime())

    cache.set('a', 'A')
    cache.set('b', 'B')
    cache.get('a') == 'A'

    cache.set('c', 'C')

    assert cache.get('a') == 'A'
    assert cache.get('b') == None
    assert cache.get('c') == 'C'

All that's needed to make it pass is to call move_to_end() in get():

        self.priority_buckets[item.priority].move_to_end(key)
        return item.value

Liking this so far? Here's another article you might like:

Learn by reading code: Python standard library design decisions explained

Problem: our priority queue is slow #

OK, we have a complete solution, it's time to deal with the priority queue implementation. Let's do a quick recap of the methods we need and why:

push() – to add items
peek() – to get items / buckets with the lowest expiry time / priority
remove() – to delete items
pop() – not used, but would be without the delete() refactoring

We make two related observations: first, there's no remove operation on the priority queue Wikipedia page; second, even if we unpack delete(), we only get to pop() an item/bucket from one of the queues, and still have to remove() it from the other.

And this is what makes the problem tricky – we need to maintain not one, but two independent priority queues.

Note

We'll now go through a few data structures in quick succession, which may be a bit overwhelming without preparation. Keep in mind we don't care how they work (not yet, at least), we're just shopping around based on specs.

heapq #

If you search the docs for "priority queue", you'll find heapq, which:

[...] provides an implementation of the heap queue algorithm, also known as the priority queue algorithm.

Reading on, we find extensive notes on implementing priority queues; particularly interesting are using (priority, item) tuples (already doing this!) and removing entries:

Removing the entry or changing its priority is more difficult because it would break the heap structure invariants. So, a possible solution is to mark the entry as removed and add a new entry with the revised priority.

This workaround is needed because while removing the i-th element can be done in O(log n), finding its index is O(n). To summarize, we have:

	sort	heapq
push()	O(n)	O(log n)
peek()	O(1)	O(1)
pop()	O(n)	O(log n)
remove()	O(n)	O(n)

Still, with a few mitigating assumptions, it could work:

we can assume priorities are static, so buckets never get removed
to-be-removed expiry times will get popped sooner or later anyway; we can assume that most evictions are due to expired items, and that items being evicted due to low priority (i.e. when the cache is full) and item updates are rare (both cause to-be-removed entries to accumulate in the expiry queue)

bisect #

One way of finding an element in better than O(n) is bisect, which:

[...] provides support for maintaining a list in sorted order without having to sort the list after each insertion.

This may provide an improvement to our naive implementation; sadly, reading further to Performance Notes we find that:

The insort() functions are O(n) because the logarithmic search step is dominated by the linear time insertion step.

While in general that's better than just about any sort, we happen to be hitting the best case of our sort implementation, which has the same complexity. (Nevertheless, shifting elements is likely cheaper than the same number of comparisons.)

	sort	heapq	bisect
push()	O(n)	O(log n)	O(n)
peek()	O(1)	O(1)	O(1)
pop()	O(n)	O(log n)	O(n)
remove()	O(n)	O(n)	O(n)

Further down in the docs, there's a see also box:

Sorted Collections is a high performance module that uses bisect to managed sorted collections of data.

Not in stdlib, moving along... ¯\_(ツ)_/¯

pop() optimization #

There's an unrelated improvement that applies to both the naive solution and bisect. With a sorted list, pop() is O(n) because it shifts all elements after the first; if the order was reversed, we'd pop() from the end, which is O(1). So:

	sort	heapq	bisect
push()	O(n)	O(log n)	O(n)
peek()	O(1)	O(1)	O(1)
pop()	O(1)*	O(log n)	O(1)*
remove()	O(n)	O(n)	O(n)

Binary search trees #

OK, I'm out of ideas, and there's nothing else in stdlib that can help.

We can restate the problem as follows: we need a sorted data structure that can do better than O(n) for push() / remove().

We've already peeked at the Wikipedia priority queue page, so let's keep reading – skipping past the naive implementations, to the usual implementation, we find that:

To improve performance, priority queues are typically based on a heap, [...]

Looked into that, didn't work; next:

Alternatively, when a self-balancing binary search tree is used, insertion and removal also take O(log n) time [...]

There, that's what we're looking for! (And likely what your interviewer is, too.)

	sort	heapq	bisect	BST
push()	O(n)	O(log n)	O(n)	O(log n)
peek()	O(1)	O(1)	O(1)	O(log n)
pop()	O(1)*	O(log n)	O(1)*	O(log n)
remove()	O(n)	O(log n)	O(n)	O(log n)

But there's no self-balancing BST in the standard library, and I sure as hell am not implementing one right now – I still have flashbacks from when I tried to do a red-black tree and two hours later it still had bugs (I mean, look at the length of this explanation!).

After a bit of googling we find, among others, bintrees, a mature library that provides all sorts of binary search trees... except:

Bintrees Development Stopped

Use sortedcontainers instead: https://pypi.python.org/pypi/sortedcontainers

Sounds familiar, doesn't it?

Sorted Containers #

Let's go back to that Sorted Collections library bisect was recommending:

Depends on the Sorted Containers module.

(￢‿￢ )

I remember, I remember now... I'm not salty because the red-black tree took two hours to implement. I'm salty because after all that time, I found Sorted Containers, a pure Python library that is faster in practice than fancy self-balancing binary search trees implemented in C!

It has extensive benchmarks to prove it, and simulated workload benchmarks for our own use case, priority queues – so yeah, while the interview answer is "self-balancing BSTs", the actual answer is Sorted Containers.

How does it work? There's an extensive explanation too:⁶

The Sorted Containers internal implementation is based on a couple observations. The first is that Python’s list is fast, really fast. [...] The second is that bisect.insort⁷ is fast. This is somewhat counter-intuitive since it involves shifting a series of items in a list. But modern processors do this really well. A lot of time has been spent optimizing mem-copy/mem-move-like operations both in hardware and software.

But using only one list and bisect.insort would produce sluggish behavior for lengths exceeding ten thousand. So the implementation of Sorted List uses a list of lists to store elements. [...]

There's also a comparison with trees, which I'll summarize: fewer memory allocations, better cache locality, lower memory overhead, and faster iteration.

I think that gives you a decent idea of how and why it works, enough that with a bit of tinkering you might be able to implement it yourself.⁸

Problem: Sorted Containers is not in stdlib #

But Sorted Containers is not in the standard library either, and we don't want to implement it ourselves. We did learn something from it, though:

	sort	heapq	bisect	bisect <10k	BST
push()	O(n)	O(log n)	O(n)	O(log n)	O(log n)
peek()	O(1)	O(1)	O(1)	O(1)	O(log n)
pop()	O(1)*	O(log n)	O(1)*	O(1)*	O(log n)
remove()	O(n)	O(log n)	O(n)	O(log n)	O(log n)

We still need to make some assumptions, though:

Do we really need more than 10k priorities? Likely no, let's just cap them at 10k.
Do we really need more than 10k expiry times? Maybe? – with 1 second granularity we can represent only up to 2.7 hours; 10 seconds takes us up to 27 hours, which may just work.

OK, one more and we're done. The other issue, aside from the maximum time, is that the granularity is too low, especially for short times – rounding 1 to 10 seconds is much worse than rounding 2001 to 2010 seconds. Which begs the question –

Does it really matter if items expire in 2010 seconds instead of 2001? Likely no, but we need a way to round small values with higher granularity than big ones.

If you've made it this far, you will definitely like:

Has your password been pwned? Or, how I almost failed to search a 37 GB text file in under 1 millisecond (in Python)

Logarithmic time #

How about 1, 2, 4, 8, ...? Rounding up to powers of 2 gets us decreasing granularity, but time doesn't actually start at zero. We fix this by rounding up to multiples of powers of 2 instead; let's get an intuition of how it works:

ceil((2000 +  1) /  1) *  1 = 2001
ceil((2000 +  2) /  2) *  2 = 2002
ceil((2000 +  3) /  4) *  4 = 2004
ceil((2000 +  4) /  4) *  4 = 2004
ceil((2000 + 15) / 16) * 16 = 2016
ceil((2000 + 16) / 16) * 16 = 2016
ceil((2000 + 17) / 32) * 32 = 2048

So far so good, how about after some time has passed?

ceil((2013 +  1) /  1) *  1 = 2014
ceil((2013 +  2) /  2) *  2 = 2016
ceil((2013 +  3) /  4) *  4 = 2016
ceil((2013 +  4) /  4) *  4 = 2020
ceil((2013 + 15) / 16) * 16 = 2032
ceil((2013 + 16) / 16) * 16 = 2032
ceil((2013 + 17) / 32) * 32 = 2048

The beauty of aligned powers is that for a relatively constant number of expiry times, the number of buckets remains roughly the same over time – as closely packed buckets are removed from the beginning, new ones fill the gaps between the sparser ones towards the end.

OK, let's put it into code:

def log_bucket(now, maxage):
    next_power = 2 ** math.ceil(math.log2(maxage))
    expires = now + maxage
    bucket = math.ceil(expires / next_power) * next_power
    return bucket

>>> [log_bucket(0, i) for i in [1, 2, 3, 4, 15, 16, 17]]
[1, 2, 4, 4, 16, 16, 32]
>>> [log_bucket(2000, i) for i in [1, 2, 3, 4, 15, 16, 17]]
[2001, 2002, 2004, 2004, 2016, 2016, 2048]
>>> [log_bucket(2013, i) for i in [1, 2, 3, 4, 15, 16, 17]]
[2014, 2016, 2016, 2020, 2032, 2032, 2048]

Looking good!

There are two sources of error – first from rounding maxage, worst when it's one more than a power of 2, and second from rounding the expiry time, also worst when it's one more than a power of two. Together, they approach 200% of maxage:

>>> log_bucket(0, 17)  # (32 - 17) / 17 ~= 88%
32
>>> log_bucket(0, 33)  # (64 - 33) / 33 ~= 94%
64
>>> log_bucket(16, 17)  # (64 - 31) / 17 ~= 182%
64
>>> log_bucket(32, 33)  # (128 - 64) / 33 ~= 191%
128

200% error is quite a lot; before we set to fix it, let's confirm our reasoning.

def error(now, maxage, *args):
    """log_bucket() error."""
    bucket = log_bucket(now, maxage, *args)
    return (bucket - now) / maxage - 1

def max_error(now, max_maxage, *args):
    """Worst log_bucket() error for all maxages up to max_maxage."""
    return max(
        error(now, maxage, *args)
        for maxage in range(1, max_maxage)
    )

def max_error_random(n, *args):
    """Worst log_bucket() error for random inputs, out of n tries."""
    max_now = int(time.time()) * 2
    max_maxage = 3600 * 24 * 31
    rand = functools.partial(random.randint, 1)
    return max(
        error(rand(max_now), rand(max_maxage), *args)
        for _ in range(n)
    )

>>> max_error(0, 10_000)
0.9997558891736849
>>> max_error(2000, 10_000)
1.9527896995708156
>>> max_error_random(10_000_000)
1.9995498725910554

Looks confirmed enough to me.

So, how do we make the error smaller? Instead of rounding to the next power of 2, we round to the next half of a power of 2, or next quarter, or next eighth...

def log_bucket(now, maxage, shift=0):
    next_power = 2 ** max(0, math.ceil(math.log2(maxage) - shift))
    expires = now + maxage
    bucket = math.ceil(expires / next_power) * next_power
    return bucket

It seems to be working:

>>> for s in range(5):
...     print([log_bucket(0, i, s) for i in [1, 2, 3, 4, 15, 16, 17]])
...
[1, 2, 4, 4, 16, 16, 32]
[1, 2, 4, 4, 16, 16, 32]
[1, 2, 3, 4, 16, 16, 24]
[1, 2, 3, 4, 16, 16, 20]
[1, 2, 3, 4, 15, 16, 18]

>>> for s in range(10):
...     e = max_error_random(1_000_000, s)
...     print(f'{s} {e:6.1%}')
...
0 199.8%
1  99.9%
2  50.0%
3  25.0%
4  12.5%
5   6.2%
6   3.1%
7   1.6%
8   0.8%
9   0.4%

With shift=7, the error is less that two percent; I wonder how many buckets that is...

def max_buckets(max_maxage, *args):
    """Number of buckets to cover all maxages up to max_maxage."""
    now = time.time()
    buckets = {
        log_bucket(now, maxage, *args)
        for maxage in range(1, max_maxage)
    }
    return len(buckets)

>>> max_buckets(3600 * 24, 7)
729
>>> max_buckets(3600 * 24 * 31, 7)
1047
>>> max_buckets(3600 * 24 * 365, 7)
1279

A bit over a thousand buckets for the whole year, not bad!

Before we can use any of that, we need to convert expiry times to buckets; that looks a lot like the priority buckets code, the only notable part being eviction.

__init__():

        self.cache = {}
        self.expires_buckets = {}
        self.expires_order = PriorityQueue()
        self.priority_buckets = {}
        self.priority_order = PriorityQueue()

set():

        expires_bucket = self.expires_buckets.get(expires)
        if not expires_bucket:
            expires_bucket = self.expires_buckets[expires] = set()
            self.expires_order.push(expires)
        expires_bucket.add(key)

delete():

        expires_bucket = self.expires_buckets[expires]
        expires_bucket.remove(key)
        if not expires_bucket:
            del self.expires_buckets[expires]
            self.expires_order.remove(expires)

evict():

        while self.cache:
            expires = self.expires_order.peek()
            if expires > now:
                break
            expires_bucket = self.expires_buckets[expires]
            for key in list(self.expires_buckets[expires]):
                self.delete(key)

And now we use log_bucket(). Since we're at it, why not have unlimited priorities too? A hammer is a hammer and everything is a nail, after all.

        expires = log_bucket(now, maxage, shift=7)
        priority = log_bucket(0, priority+1, shift=7)
        item = Item(key, value, expires, priority)

bisect, redux #

Time to fix that priority queue.

We use insort() to add priorities and operator.neg() to keep the list reversed:⁹

    def push(self, item):
        bisect.insort(self.data, item, key=operator.neg)

We update peek() and pop() to handle the reverse order:

    def peek(self):
        return self.data[-1]

    def pop(self):
        return self.data.pop()

Finally, for remove() we adapt the index() recipe from Searching Sorted Lists:

    def remove(self, item):
        i = bisect.bisect_left(self.data, -item, key=operator.neg)
        if i != len(self.data) and self.data[i] == item:
            del self.data[i]
            return
        raise ValueError

And that's it, we're done!

Here's the cache in its full glory (click to expand):

class Cache:

    def __init__(self, maxsize, time=time.monotonic):
        self.maxsize = maxsize
        self.time = time

        self.cache = {}
        self.expires_buckets = {}
        self.expires_order = PriorityQueue()
        self.priority_buckets = {}
        self.priority_order = PriorityQueue()

    def get(self, key):
        item = self.cache.get(key)
        if not item:
            return None

        if self.time() >= item.expires:
            return None

        self.priority_buckets[item.priority].move_to_end(key)
        return item.value

    def set(self, key, value, *, maxage=10, priority=0):
        now = self.time()

        if key in self.cache:
            self.delete(key)
        elif len(self.cache) >= self.maxsize:
            self.evict(now)

        expires = log_bucket(now, maxage, shift=7)
        priority = log_bucket(0, priority+1, shift=7)
        item = Item(key, value, expires, priority)

        self.cache[key] = item

        expires_bucket = self.expires_buckets.get(expires)
        if not expires_bucket:
            expires_bucket = self.expires_buckets[expires] = set()
            self.expires_order.push(expires)
        expires_bucket.add(key)

        priority_bucket = self.priority_buckets.get(priority)
        if not priority_bucket:
            priority_bucket = self.priority_buckets[priority] = OrderedDict()
            self.priority_order.push(priority)
        priority_bucket[key] = None

    def evict(self, now):
        if not self.cache:
            return

        initial_size = len(self.cache)

        while self.cache:
            expires = self.expires_order.peek()
            if expires > now:
                break
            expires_bucket = self.expires_buckets[expires]
            for key in list(self.expires_buckets[expires]):
                self.delete(key)

        if len(self.cache) == initial_size:
            priority = self.priority_order.peek()
            priority_bucket = self.priority_buckets.get(priority)
            key = next(iter(priority_bucket))
            self.delete(key)

    def delete(self, key):
        *_, expires, priority = self.cache.pop(key)

        expires_bucket = self.expires_buckets[expires]
        expires_bucket.remove(key)
        if not expires_bucket:
            del self.expires_buckets[expires]
            self.expires_order.remove(expires)

        priority_bucket = self.priority_buckets[priority]
        del priority_bucket[key]
        if not priority_bucket:
            del self.priority_buckets[priority]
            self.priority_order.remove(priority)

class Item(NamedTuple):
    key: object
    value: object
    expires: int
    priority: int

class PriorityQueue:

    def __init__(self):
        self.data = []

    def push(self, item):
        bisect.insort(self.data, item, key=operator.neg)

    def peek(self):
        return self.data[-1]

    def pop(self):
        return self.data.pop()

    def remove(self, item):
        i = bisect.bisect_left(self.data, -item, key=operator.neg)
        if i != len(self.data) and self.data[i] == item:
            del self.data[i]
            return
        raise ValueError

    def __bool__(self):
        return bool(self.data)

def log_bucket(now, maxage, shift=0):
    next_power = 2 ** max(0, math.ceil(math.log2(maxage) - shift))
    expires = now + maxage
    bucket = math.ceil(expires / next_power) * next_power
    return bucket

(The entire file, with tests and everything.)

Conclusion #

Anyone expecting you to implement this in under an hour is delusional. Explaining what you would use and why should be enough for reasonable interviewers, although that may prove difficult if you haven't solved this kind of problem before.

Bullshit interviews aside, it is useful to have basic knowledge of time complexity. Again, can't recommend Big-O: How Code Slows as Data Grows enough.

But, what big O notation says and what happens in practice can differ quite a lot. Be sure to measure, and be sure to think of limits – sometimes, the n in O(n) is or can be made small enough you don't have to do the theoretically correct thing.

You don't need to know how to implement all the data structures, that's what (software) libraries and Wikipedia are for (and for that matter, book libraries too). However, it is useful to have an idea of what's available and when to use it.

Good libraries educate – the Python standard library docs already cover a lot of the practical knowledge we needed, and so did Sorted Containers. But, that won't show up in the API reference you see in your IDE, you have read the actual documentation.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Note the subtitle: if you're not sure what to do yet. ^[return]
This early on, the name doesn't really matter, but we'll go with the correct, descriptive one; in the first draft of the code, it was called MagicDS. ✨ ^[return]
You have to admit this is at least a bit weird; what you're looking at is an object in a trench coat, at least if you think closures and objects are equivalent. ^[return]
Another way of getting an "object" on the cheap. ^[return]
If we assume a relatively small number of buckets that will be reused soon enough, this isn't strictly necessary. I'm partly doing it to release the memory held by the dict, since dicts are resized only when items are added. ^[return]
There's even a PyCon talk with the same explanation, if you prefer that. ^[return]
bisect itself has a fast C implementation, so I guess technically it's not pure Python. But given that stdlib is already there, does that count? ^[return]
If the implementation is easy to explain, it may be a good idea. ^[return]
This limits priorities to values that can be negated, so tuples won't work anymore. We could use a "reversed view" wrapper if we really cared about that. ^[return]

reader 3.10 released – storage internal API

2023-11-15T22:00:00+00:00

Hi there!

I'm happy to announce version 3.10 of reader, a Python feed reader library.

What's new? #

Here are the highlights since reader 3.9.

Storage internal API #

The storage internal API is now documented!

This is important because it opens up reader to using other databases than SQLite.

The protocols are mostly stable, but some changes are still expected. The long term goal is full stabilization, but at least one other implementation needs to exists before that, to work out any remaining kinks.

A SQLAlchemy backend would be especially useful, since it would provide access to a variety of database engines mostly out of the box. (Alas, I do not have time nor a need for this at the moment. Interested on working on it? Let me know!)

Why not use SQLAlchemy from the start? #

In the beginning:

I wanted to keep things as simple as possible, so I stay motivated for the long term. I also wanted to follow a problem-solution approach, which cautions against solving problems you don't have. (Details on both here and here.)
By that time, I was already a SQLite fan, and due to the single-user nature of reader, I was relatively confident concurrency won't be an issue.
I didn't know exactly where and how I would deploy the web app; sqlite3 being in the standard library made it very appealing.

Since then, I did come up with some of my own complexity – reader has a query builder and a migration system (albeit both of them tiny), and there were some concurrency issues. SQLAlchemy would have likely helped with the first two, but not with the last. Overall, I still think plain SQLite was the right choice at the time.

Deprecated sqlite3 datetime support #

The default sqlite3 datetime adapters/converters were deprecated in Python 3.12. Since adapters/converters apply to all database connections, reader does not have the option of registering its own (as a library, it should not change global stuff), so datetime conversions now happen in the storage. As an upside, this provided an opportunity to change the storage to use timezone-aware datetimes.

There's a new share web app plugin to add social sharing links to the entry page.

Ideally, this functionality should end up in a plugin that adds them to Entry.links (to be exposed in #320), so all reader users can benefit from it.

Python versions #

None this time, but Python 3.12 support is coming soon!

For more details, see the full changelog.

That's it for now.

Want to contribute? Check out the docs and the roadmap.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

reader is a library
reader is for the long term
reader is extensible
reader is stable (within reason)
reader is simple to use; API matters
reader features work well together
reader is tested
reader is documented
reader has minimal dependencies

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

When to use classes in Python? When you repeat similar sets of functions

2025-07-22T18:00:00+00:00

Are you having trouble figuring out when to use classes or how to organize them?

Have you repeatedly searched for "when to use classes in Python", read all the articles and watched all the talks, and still don't know whether you should be using classes in any given situation?

Have you read discussions about it that for all you know may be right, but they're so academic you can't parse the jargon?

Have you read articles that all treat the "obvious" cases, leaving you with no clear answer when you try to apply them to your own code?

My experience is that, unfortunately, the best way to learn this is to look at lots of examples.

Most guidelines tend to either be too vague if you don't already know enough about the subject, or too specific and saying things you already know.

This is one of those things that once you get it seems obvious and intuitive, but it's not, and is quite difficult to explain properly.

So, instead of prescribing a general approach, let's look at:

one specific case where you may want to use classes
examples from real-world code
some considerations you should keep in mind

Contents

The heuristic
Example: Retrievers
Example: Flask's tagged JSON
Formalizing this
Counter-example: modules
Try it out

The heuristic #

If you repeat similar sets of functions, consider grouping them in a class.

That's it.

In its most basic form, a class is when you group data with functions that operate on that data; sometimes, there is no data, but it can still be useful to group the functions into an abstract object that exists only to make things easier to use / understand.

Depending on whether you choose which class to use at runtime, this is sometimes called the strategy pattern.

Note

As Wikipedia puts it, "A heuristic is a practical way to solve a problem. It is better than chance, but does not always work. A person develops a heuristic by using intelligence, experience, and common sense."

So, this is not the correct thing to do all the time, or even most of the time.

Instead, I hope that this and other heuristics can help build the right intuition for people on their way from "I know the class syntax, now what?" to "proper" object-oriented design.

Example: Retrievers #

My feed reader library retrieves and stores web feeds (Atom, RSS and so on).

Usually, feeds come from the internet, but you can also use local files. The parsers for various formats don't really care where a feed is coming from, so they always take an open file as input.

reader supports conditional requests – that is, only retrieve a feed if it changed. To do this, it stores the ETag HTTP header from a response, and passes it back as the If-None-Match header of the next request; if nothing changed, the server can respond with 304 Not Modified instead of sending back the full content.

Let's have a look at how the code to retrieve feeds evolved over time; this version omits a few details, but it will end up with a structure similar to that of the full version. In the beginning, there was a function – URL and old ETag in, file and new ETag out:

def retrieve(url, etag=None):
    if any(url.startswith(p) for p in ('http://', 'https://')):
        headers = {}
        if etag:
            headers['If-None-Match'] = etag
        response = requests.get(url, headers=headers, stream=True)
        response.raise_for_status()
        if response.status_code == 304:
            response.close()
            return None, etag
        etag = response.headers.get('ETag', etag)
        response.raw.decode_content = True
        return response.raw, etag

    # fall back to file
    path = extract_path(url)
    return open(path, 'rb'), None

We use Requests to get HTTP URLs, and return the underlying file-like object.¹

For local files, we suport both bare paths and file URIs; for the latter, we do a bit of validation – file:feed and file://localhost/feed are OK, but file://invalid/feed and unknown:feed² are not:

def extract_path(url):
    url_parsed = urllib.parse.urlparse(url)
    if url_parsed.scheme == 'file':
        if url_parsed.netloc not in ('', 'localhost'):
            raise ValueError("unknown authority for file URI")
        return urllib.request.url2pathname(url_parsed.path)
    if url_parsed.scheme:
        raise ValueError("unknown scheme for file URI")
    # no scheme, treat as a path
    return url

Problem: can't add new feed sources #

One of reader's goals is to be extensible. For example, it should be possible to add new feed sources like an FTP server (ftp://...) or Twitter without changing reader code; however, our current implementation makes it hard to do so.

We can fix this by extracting retrieval logic into separate functions, one per protocol:

def http_retriever(url, etag):
    headers = {}
    # ...
    return response.raw, etag

def file_retriever(url, etag):
    path = extract_path(url)
    return open(path, 'rb'), None

...and then routing to the right one depending on the URL prefix:

# sorted by key length (longest first)
RETRIEVERS = {
    'https://': http_retriever,
    'http://': http_retriever,
    # fall back to file
    '': file_retriever,
}

def get_retriever(url):
    for prefix, retriever in RETRIEVERS.items():
        if url.lower().startswith(prefix.lower()):
            return retriever
    raise ValueError("no retriever for URL")

def retrieve(url, etag=None):
    retriever = get_retriever(url)
    return retriever(url, etag)

Now, plugins can register retrievers by adding them to RETRIEVERS (in practice, there's a method for that, so users don't need to care about it staying sorted).

Problem: can't validate URLs until retrieving them #

To add a feed, you call add_feed() with the feed URL.

But what if you pass an invalid URL? The feed gets stored in the database, and you get an "unknown scheme for file URI" error on the next update. However, this can be confusing – a good API should signal errors near the action that triggered them. This means add_feed() needs to validate the URL without actually retrieving it.

For HTTP, Requests can do the validation for us; for files, we can call extract_path() and ignore the result. Of course, we should select the appropriate logic in the same way we select retrievers, otherwise we're back where we started.

Now, there's more than one way of doing this. We could keep a separate validator registry, but that may accidentally become out of sync with the retriever one.

URL_VALIDATORS = {
    'https://': http_url_validator,
    'http://': http_url_validator,
    '': file_url_validator,
}

Or, we could keep a (retriever, validator) pair in the retriever registry. This is better, but it's not all that readable (what if need to add a third thing?); also, it makes customizing behavior that affects both the retriever and validator harder.

RETRIEVERS = {
    'https://': (http_retriever, http_url_validator),
    'http://': (http_retriever, http_url_validator),
    '': (file_retriever, file_url_validator),
}

Better yet, we can use a class to make the grouping explicit:

class HTTPRetriever:

    def retrieve(self, url, etag):
        headers = {}
        # ...
        return response.raw, etag

    def validate_url(self, url):
        session = requests.Session()
        session.get_adapter(url)
        session.prepare_request(requests.Request('GET', url))

class FileRetriever:

    def retrieve(self, url, etag):
        path = extract_path(url)
        return open(path, 'rb'), None

    def validate_url(self, url):
        extract_path(url)

We then instantiate them, and update retrieve() to call the methods:

http_retriever = HTTPRetriever()
file_retriever = FileRetriever()

def retrieve(url, etag=None):
    retriever = get_retriever(url)
    return retriever.retrieve(url, etag)

validate_url() works just the same:

def validate_url(url):
    retriever = get_retriever(url)
    retriever.validate_url(url)

And there you have it – if you repeat similar sets of functions, consider grouping them in a class.

Not just functions, attributes too #

Say you want to update feeds in parallel, using multiple threads.

Retrieving feeds is mostly waiting around for I/O, so it will benefit the most from it. Parsing, on the other hand, is pure Python, CPU bound code, so threads won't help due to the global interpreter lock.

However, because we're streaming the reponse body, I/O is not done when the retriever returns the file, but when the parser finishes reading it.³ We can move all the (network) I/O in retrieve() by reading the response into a temporary file and returning it instead.

We'll allow any retriever to opt into this behavior by using a class attribute:

class HTTPRetriever:
    slow_to_read = True

class FileRetriever:
    slow_to_read = False

If a retriever is slow to read, retrieve() does the swap:

def retrieve(url, etag=None):
    retriever = get_retriever(url)
    file, etag = retriever.retrieve(url, etag)

    if file and retriever.slow_to_read:
        temp = tempfile.TemporaryFile()
        shutil.copyfileobj(file, temp)
        file.close()
        temp.seek(0)
        file = temp

    return file, etag

Liking this so far? Here's another article you might like:

When to use classes in Python? When your functions take the same arguments

Example: Flask's tagged JSON #

The Flask web framework provides an extendable compact representation for non-standard JSON types called tagged JSON (code). The serializer class delegates most conversion work to methods of various JSONTag subclasses (one per supported type):

check() checks if a Python value should be tagged by that tag
tag() converts it to tagged JSON
to_python() converts a JSON value back to Python (the serializer uses the key tag attribute to find the correct tag)

Interestingly, tag instances have an attribute pointing back to the serializer, likely to allow recursion – when (un)packing a possibly nested collection, you need to recursively (un)pack its values. Passing the serializer to each method would have also worked, but when your functions take the same arguments...

Formalizing this #

OK, the retriever code works. But, how should you communicate to others (readers, implementers, interpreters, type checkers) that an HTTPRetriever is the same kind of thing as a FileRetriever, and as anything else that can go in RETRIEVERS?

Duck typing #

Here's the definition of duck typing:

A programming style which does not look at an object's type to determine if it has the right interface; instead, the method or attribute is simply called or used ("If it looks like a duck and quacks like a duck, it must be a duck.") [...]

This is what we're doing now! If it retrieves like a retriever and validates URLs like a retriever, then it's a retriever.

You see this all the time in Python. For example, json.dump() takes a file-like object; now, the full text file interface has lots methods and attributes, but dump() only cares about write(), and will accept any object implementing it:

>>> class MyFile:
...     def write(self, s):
...         print(f"writing: {s}")
...
>>> f = MyFile()
>>> json.dump({'one': 1}, f)
writing: {
writing: "one"
writing: :
writing: 1
writing: }

The main way to communicate this is through documentation:

Serialize obj [...] to fp (a .write()-supporting file-like object)

Inheritance #

Nevertheless, you may want to be more explicit about the relationships between types. The easiest option is to use a base class, and require retrievers to inherit from it.

class Retriever:
    slow_to_read = False

    def retrieve(self, url, etag):
        raise NotImplementedError

    def validate_url(self, url):
        raise NotImplementedError

This allows you to check you the type with isinstance(), provide default methods and attributes, and will help type checkers and autocompletion, at the expense of forcing a dependency on the base class.

>>> class MyRetriever(Retriever): pass
>>> retriever = MyRetriever()
>>> retriever.slow_to_read
False
>>> isinstance(retriever, Retriever)
True

What it won't do is check subclasses actually define the methods:

>>> retriever.validate_url('myurl')
Traceback (most recent call last):
  ...
NotImplementedError

Abstract base classes #

This is where abstract base classes come in. The decorators in the abc module allow defining abstract methods that must be overriden:

class Retriever(ABC):

    @abstractproperty
    def slow_to_read(self):
        return False

    @abstractmethod
    def retrieve(self, url, etag):
        raise NotImplementedError

    @abstractmethod
    def validate_url(self, url):
        raise NotImplementedError

This is checked at runtime (but only that methods and attributes are present, not their signatures or types):

>>> class MyRetriever(Retriever): pass
>>> MyRetriever()
Traceback (most recent call last):
  ...
TypeError: Can't instantiate abstract class MyRetriever with abstract methods retrieve, slow_to_read, validate_url

>>> class MyRetriever(Retriever):
...     slow_to_read = False
...     def retrieve(self, url, etag): ...
...     def validate_url(self, url): ...
...
>>> MyRetriever()
<__main__.MyRetriever object at 0x1037aac50>

Tip

You can also use ABCs to register arbitrary types as "virtual subclasses"; this allows them to pass isinstance() checks without inheritance, but won't check for required methods:

>>> class MyRetriever: pass
>>> Retriever.register(MyRetriever)
<class '__main__.MyRetriever'>
>>> isinstance(MyRetriever(), Retriever)
True

Protocols #

Finally, we have protocols, aka structural subtyping, aka static duck typing. Introduced in PEP 544, they go in the opposite direction – what if instead declaring what the type of something is, we declare what methods it has to have to be of a specific type?

You define a protocol by inheriting typing.Protocol:

class Retriever(Protocol):

    @property
    def slow_to_read(self) -> bool:
        ...

    def retrieve(self, url: str, etag: str | None) -> tuple[IO[bytes] | None, str | None]:
        ...

    def validate_url(self, url: str) -> None:
        ...

...and then use it in type annotations:

def mount_retriever(prefix: str, retriever: Retriever) -> None:
    raise NotImplementedError

Some other code (not necessarily yours, not necessarily aware the protocol even exists) defines an implementation:

class MyRetriever:
    slow_to_read = False

    def validate_url(self):
        pass

...and then uses it with annotated code:

mount_retriever('my', MyRetriever())

A type checker like mypy will check if the provided instance conforms to the protocol – not only that methods exist, but that their signatures are correct too – all without the implementation having to declare anything.

$ mypy myproto.py
myproto.py:11: error: Argument 2 to "mount_retriever" has incompatible type "MyRetriever"; expected "Retriever"  [arg-type]
myproto.py:11: note: "MyRetriever" is missing following "Retriever" protocol member:
myproto.py:11: note:     retrieve
myproto.py:11: note: Following member(s) of "MyRetriever" have conflicts:
myproto.py:11: note:     Expected:
myproto.py:11: note:         def validate_url(self, url: str) -> None
myproto.py:11: note:     Got:
myproto.py:11: note:         def validate_url(self) -> Any
Found 1 error in 1 file (checked 1 source file)

Tip

If you decorate your protocol with runtime_checkable, you can use it in isinstance() checks, but like ABCs, it only checks methods are present.

Counter-example: modules #

If a class has no state and you don't need inheritance, you can use a module instead:

# module.py

slow_to_read = False

def retrieve(url, etag):
    raise NotImplementedError

def validate_url(url):
    raise NotImplementedError

From a duck typing perspective, this is a valid retriever, since it has all the expected methods and attributes. So much so, that it's also compatible with protocols:

import module

mount_retriever('mod', module)

$ mypy module.py
Success: no issues found in 1 source file

I tried to keep the retriever example stateless, but real world classes rarely are (it may be immutable state, but it's state nonetheless). Also, you're limited to exactly one implementation per module, which is usually too much like Java for my taste.

Tip

For a somewhat forced, but illustrative example of a stateful concurrent.futures executor implemented like this, and a comparison with class-based alternatives, check out Inheritance over composition, sometimes.

Try it out #

If you're doing something and you think you need a class, do it and see how it looks. If you think it's better, keep it, otherwise, revert the change. You can always switch in either direction later.

If you got it right the first time, great! If not, by having to fix it you'll learn something, and next time you'll know better.

Also, don't beat yourself up.

Sure, there are nice libraries out there that use classes in just the right way, after spending lots of time to find the right abstraction. But abstraction is difficult and time consuming, and in everyday code good enough is just that – good enough – you don't need to go to the extreme.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

If you've made it this far, you might like:

Write an SQL query builder in 150 lines of Python!

This code has a potential bug: if we were using a persistent session instead of a transient one, the connection would never be released, since we're not closing the response after we're done with it. In the actual code, we're doing both, but the only way do so reliably is to return a context manager; I omitted this because it doesn't add anything to our discussion about classes. ^[return]
We're handling unknown URI schemes here because bare paths don't have a scheme, so anything that didn't match a known scheme must be a bare path. Also, on Windows (not supported yet), the drive letter in a path like c:\feed.xml is indistinguishable from a scheme. ^[return]
Unless the response is small enough to fit in the TCP receive buffer. ^[return]

reader 3.9 released – update hook error handling

2023-08-28T20:00:00+00:00

Hi there!

I'm happy to announce version 3.9 of reader, a Python feed reader library.

What's new? #

Here are the highlights since reader 3.7.

Better handling of unexpected update errors #

Unexpected exceptions raised by update hooks, retrievers, and parsers are now wrapped in UpdateError, so errors for one feed don't prevent others from being updated. Also, hooks that run after a feed is updated are all run, regardless of individual failures. Plugins should benefit most from the improved fault isolation.

Exception hierarchy diagram #

The API docs got a cool new exception hierarchy diagram (yes, it's autogenerated):

ReaderError
 ├── ReaderWarning [UserWarning]
 ├── ResourceNotFoundError
 ├── FeedError
 │    ├── FeedExistsError
 │    ├── FeedNotFoundError [ResourceNotFoundError]
 │    └── InvalidFeedURLError [ValueError]
 ├── EntryError
 │    ├── EntryExistsError
 │    └── EntryNotFoundError [ResourceNotFoundError]
 ├── UpdateError
 │    ├── ParseError [FeedError, ReaderWarning]
 │    └── UpdateHookError
 │         ├── SingleUpdateHookError
 │         └── UpdateHookErrorGroup [ExceptionGroup]
 ├── StorageError
 ├── SearchError
 │    ├── SearchNotEnabledError
 │    └── InvalidSearchQueryError [ValueError]
 ├── PluginError
 │    ├── InvalidPluginError [ValueError]
 │    └── PluginInitError
 └── TagError
      └── TagNotFoundError

Parser cleanup #

I moved all modules related to feed retrieval and parsing to reader._parser, another step towards internal API stabilization. This has also given me an opportunity to make lazy imports a bit less intrusive.

Timer experimental plugin #

There's a new timer experimental plugin to collect per-call method timings.

The web app shows them in the footer like so:

Python versions #

Python 3.9 support is no more, as foretold in the ancient murals.

For more details, see the full changelog.

That's it for now.

Want to contribute? Check out the docs and the roadmap.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

reader is a library
reader is for the long term
reader is extensible
reader is stable (within reason)
reader is simple to use; API matters
reader features work well together
reader is tested
reader is documented
reader has minimal dependencies

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

Running async code from sync code in Python

2023-08-05T12:00:00+00:00

So, you're doing some sync stuff.

But you also need to do some async stuff, without making everything async.

Maybe the sync stuff is an existing application.

Maybe you still want to use your favorite sync library.

Or maybe you need just a little async, without having to pay the full price.

Of course, you can run a coroutine with asyncio.run(), and blocking sync code from a coroutine with asyncio.to_thread(), but the former isn't granular enough, and the latter doesn't solve async code being at the top.

As always, there must be a better way.

Maybe something like this?

async def double(i):
    return i * 2

async def arange(i):
   for n in range(i):
       yield n

runner = ThreadRunner()
print(runner.run(double(2)))  # 4
print(*runner.wrap_iter(arange(4)))  # 0 1 2 3

Now, it would take a long time to fully explore how you might build up to such a thing and why you would actually need it... so that's exactly what we're going to do.

Update (August 2025)

You can now install this from PyPI! ⭐️

Contents

But I need to do the thing more than once
But I want a long-lived event loop
But I want a long-lived session
But I want to do something else during the async stuff
But I want to do async stuff from multiple threads
But I don't want to manage async context managers by hand
But I want to use async iterables
So, should you do this?

But I need to do the thing more than once #

To make our example more life-like, we'll make some requests using aiohttp.

async def do_stuff(session):
    async with session.get('https://death.andgravity.com/') as response:
        loop = asyncio.get_running_loop()
        print('got', response.status, 'in loop', id(loop))

We create the session separately, since we might want to reuse it later on.

async def main():
    async with aiohttp.ClientSession() as session:
        await do_stuff(session)

asyncio.run() allows you to execute a coroutine in a transient event loop; it's meant to be the entry point of your program, and "should ideally only be called once" – but you can call it multiple times:

asyncio.run(main())
asyncio.run(main())

$ python brigde.py
got 200 in loop 4399589904
got 200 in loop 4386034640

But I want a long-lived event loop #

Of course, creating and cleaning up a loop for each call isn't that efficient, and we might have loop-bound resources that need to live across calls.

With a bit of diving into asyncio guts, we can manage our own long-lived loop:

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
    loop.run_until_complete(main())
    loop.run_until_complete(main())
finally:
    loop.run_until_complete(loop.shutdown_asyncgens())
    loop.run_until_complete(loop.shutdown_default_executor())
    loop.close()

These are low-level APIs, so a bit of care is needed to use them correctly:

We could have used get_event_loop() to both create and set the loop, but this behavior was deprecated in Python 3.10.
loop.close() already schedules executor shutdown, but does not wait for it to finish; for predictable behavior, it's best to wait cleanup tasks explicitly.

Anyway, it works – note the loop id:

$ python brigde.py
got 200 in loop 4509807312
got 200 in loop 4509807312

Thankfully, starting with Python 3.11, asyncio.Runner makes this much easier:

with asyncio.Runner() as runner:
    runner.run(main())
    runner.run(main())

But I want a long-lived session #

Reusing the loop is not enough, though.

We're throwing the Session away after each request, and with it the associated connection pool, which means each request creates a new connection. Reusing connections can make repeated requests much faster – most "client" libraries (e.g. aiobotocore) keep a connection pool around for this reason.

Creating the session outside an async function and passing it in works:

with asyncio.Runner() as runner:
    session = aiohttp.ClientSession()
    runner.run(do_stuff(session))
    runner.run(do_stuff(session))

...but with a bunch of warnings:

$ python brigde.py
.../bridge.py:16: DeprecationWarning: The object should be created within an async function
  session = aiohttp.ClientSession()
got 200 in loop 4570601360
got 200 in loop 4570601360
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10ff01c90>

Update (2025)

...until it does not work at all anymore:

$ python bridge.py
Traceback (most recent call last):
  File ".../bridge.py", line 16, in <module>
    session = aiohttp.ClientSession()
  File ".../aiohttp/client.py", line 316, in __init__
    loop = loop or asyncio.get_running_loop()
RuntimeError: no running event loop

The first warning is easy – the constructor wants to be called in an async function, so we write one to call it in (to pass in constructor arguments, we can use a partial()):

async def call_async(callable):
    return callable()

with asyncio.Runner() as runner:
    session = runner.run(call_async(aiohttp.ClientSession))

The second one happens because we're expected to async session.close() after we're done with it. But we didn't do that before – we used the (async) with statement, the idiomatic way for objects to clean up after themselves. Since async with is syntactic sugar over calling the asynchronous context manager protocol methods in a specific way, we can simulate it by doing that:

with asyncio.Runner() as runner:
    session = runner.run(call_async(aiohttp.ClientSession))
    runner.run(session.__aenter__())
    try:
        runner.run(do_stuff(session))
        runner.run(do_stuff(session))
    finally:
        runner.run(session.__aexit__(None, None, None))

But I want to do something else during the async stuff #

OK, we have a long-lived event loop and a long-lived session.

But when doing async stuff, we can't do anything else, because we are busy running the event loop; and as a consequence of that, outside of run() all async stuff is frozen.

So let's use the other way of doing more than one thing at a time, and call run() from another thread. First, more logging:

async def do_stuff(session):
    async with session.get('https://death.andgravity.com/') as response:
        loop = asyncio.get_running_loop()
        thread = threading.current_thread()
        print('got', response.status, 'in loop', id(loop), 'in', thread.name)

$ python brigde.py
got 200 in loop 4538933264 in MainThread
got 200 in loop 4538933264 in MainThread

Next up, the thread stuff – in actual code, the "do something else" part would happen between thread.start() and thread.join().

def do_stuff_in_threads(run, session, n=1):
    threads = [
        threading.Thread(target=run, args=(do_stuff(session),))
        for _ in range(n)
    ]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()

with asyncio.Runner() as runner:
    session = runner.run(call_async(aiohttp.ClientSession))
    runner.run(session.__aenter__())
    try:
        do_stuff_in_threads(runner.run, session)
    finally:
        runner.run(session.__aexit__(None, None, None))

Hey, it works!

$ python brigde.py
got 200 in loop 4448024592 in Thread-1 (run)

But I want to do async stuff from multiple threads #

OK, but does it work from two threads? This can be the case in an existing application. Or we can deliberately use threads most of the time, because they work with normal code, and only use async as an implementation detail. Or, we just have one other thread, like above, but then accidentally call run() from the main thread.

Glad you asked!

        do_stuff_in_threads(runner.run, session, n=2)

$ python brigde.py
Exception in thread Thread-2 (run):
Traceback (most recent call last):
  ...
  File ".../asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File ".../asyncio/base_events.py", line 629, in run_until_complete
    self._check_running()
  File ".../asyncio/base_events.py", line 588, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
got 200 in loop 4544447440 in Thread-1 (run)
got 200 in loop 4544447440 in Thread-1 (run)

Not only did one thread crash, both coroutines ran in the other thread. Worse, sometimes only one runs. In a long-running program with infrequent uses of run(), we might not even get a hard failure, but subtle, mysterious bugs. 👻

And why not? The docs have a Concurrency and Multithreading section that warns:

Almost all asyncio objects are not thread safe, which is typically not a problem unless there is code that works with them from outside of a Task or a callback.

...and that's exactly what we've been doing. But the warning comes with a solution:

To schedule a coroutine object from a different OS thread, the run_coroutine_threadsafe() function should be used.¹

So, we want a loop running continuously in another thread, not just when we call run():

runner = asyncio.Runner()
loop_created = threading.Event()

def run_forever():
    with runner:
        loop = runner.get_loop()
        asyncio.set_event_loop(loop)
        loop_created.set()
        loop.run_forever()

loop_thread = threading.Thread(target=run_forever, name='LoopThread')
loop_thread.start()
loop_created.wait()
loop = runner.get_loop()

def run(coro):
    return asyncio.run_coroutine_threadsafe(coro, loop).result()

We have to wait for the loop to be created in its own thread, otherwise the main thread can race ahead, create the loop first with its get_loop() call, and then deadlock when run_coroutine_threadsafe() is passed a loop that isn't running yet.

We can go ahead and plug in the new run(), winding things down after we're done:

try:
    session = run(call_async(aiohttp.ClientSession))
    run(session.__aenter__())
    try:
        do_stuff_in_threads(run, session, n=2)
    finally:
        run(session.__aexit__(None, None, None))
finally:
    loop.call_soon_threadsafe(loop.stop)
    loop_thread.join()

This time, it really works:

$ python brigde.py
got 200 in loop 4444839248 in LoopThread
got 200 in loop 4444839248 in LoopThread

Getting that in the right order every time looks tricky, though, so we should probably do something about it. Just like asyncio.Runner, we'll provide a higher-level API, and since it already does similar work, we get to reuse its API design for free:

class ThreadRunner:

    def __init__(self, *args, **kwargs):
        self._runner = asyncio.Runner(*args, **kwargs)
        self._thread = None

    def __enter__(self):
        self._lazy_init()
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.close()

    def close(self):
        loop = self.get_loop()
        loop.call_soon_threadsafe(loop.stop)
        self._thread.join()

    def get_loop(self):
        self._lazy_init()
        return self._runner.get_loop()

    def run(self, coro):
        loop = self.get_loop()
        return asyncio.run_coroutine_threadsafe(coro, loop).result()

    def _lazy_init(self):
        if self._thread:
            return

        loop_created = threading.Event()

        def run_forever():
            with self._runner as runner:
                loop = runner.get_loop()
                asyncio.set_event_loop(loop)
                loop_created.set()
                loop.run_forever()

        self._thread = threading.Thread(
            target=run_forever, name='LoopThread', daemon=True
        )
        self._thread.start()
        loop_created.wait()

This means using it looks just the same, which is pretty neat:

with ThreadRunner() as runner:
    session = runner.run(call_async(aiohttp.ClientSession))
    runner.run(session.__aenter__())
    try:
        do_stuff_in_threads(runner.run, session, n=2)
    finally:
        runner.run(session.__aexit__(None, None, None))

Liking this so far? Here's another article you might like:

Limiting concurrency in Python asyncio: the story of async imap_unordered()

But I don't want to manage async context managers by hand #

If I'm being honest, this whole "simulate async with by calling the protocol methods" is getting kinda tedious. But now we have a high-level API, so we can extend it with a sync context manager to take care of it for us.

    @contextlib.contextmanager
    def wrap_context(self, cm=None, factory=None):
        if cm is None:
            cm = self.run(call_async(factory))
        aenter = type(cm).__aenter__
        aexit = type(cm).__aexit__
        value = self.run(aenter(cm))
        try:
            yield value
        except:
            if not self.run(aexit(cm, *sys.exc_info())):
                raise
        else:
            self.run(aexit(cm, None, None, None))

This is the same code as before, except we're going by the book with what gets called when; see Brett Cannon's Unravelling the async with statement for details.

You use it like so:

with ThreadRunner() as runner:
    with runner.wrap_context(factory=aiohttp.ClientSession) as session:
        do_stuff_in_threads(runner.run, session, n=2)

That's good, but we can do better. Here, we don't really need granular control over the lifetime of the session, we just want it to live as long as the event loop, so it would be nice to give users a shorthand for that.² We can use an ExitStack to enter context managers on the user's behalf and exit them when the runner is closed.

    def __init__(self, *args, **kwargs):
        self._runner = asyncio.Runner(*args, **kwargs)
        self._thread = None
        self._stack = contextlib.ExitStack()

    def enter_context(self, cm=None, factory=None):
        cm = self.wrap_context(cm=cm, factory=factory)
        return self._stack.enter_context(cm)

    def close(self):
        try:
            self._stack.close()
        finally:
            loop = self.get_loop()
            loop.call_soon_threadsafe(loop.stop)
            self._thread.join()

Now, we just need to do:

with ThreadRunner() as runner:
    session = runner.enter_context(factory=aiohttp.ClientSession)
    do_stuff_in_threads(runner.run, session, n=2)

But I want to use async iterables #

One final thing: how would you consume an asynchronous iterable like the content of an aiohttp response? In async code, it would look something like this:

async with session.get('https://death.andgravity.com/') as response:
    lines = [line async for line in response.content]
print('got', len(lines), 'lines')

Like async with, async for has its own protocol we can wrap; for slightly better error messages, we'll use the aiter() and anext() built-ins instead of calling the corresponding special methods directly:

    def wrap_iter(self, it):
        it = aiter(it)
        while True:
            try:
                yield self.run(anext(it))
            except StopAsyncIteration:
                break

You use it like so:

with ThreadRunner() as runner:
    session = runner.enter_context(factory=aiohttp.ClientSession)
    coroutine = session.get('https://death.andgravity.com/')
    with runner.wrap_context(coroutine) as response:
        lines = [line for line in runner.wrap_iter(response.content)]
    print('got', len(lines), 'lines')

$ python brigde.py
got 624 lines

And with this, we're done!

Here's the final version of the code.

Update (August 2025)

...or even better, install it from PyPI, and check out the documented, tested, type-annotated version on GitHub! ⭐️

So, should you do this? #

I guess?

I used something like this on a "production" project without any problems.

Performance might be an issue if you run() many small tasks, but the overhead for bigger tasks should be negligible, especially compared with the cost of doing I/O (for example, returning an entire HTTP response is likely fine, wrapping an iterator of lines might not be).

In my case, I was forced to use an async library that didn't have a sync counterpart, and the benefit of retaining a mostly sync code base was overwhelming.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

If you've made it this far, you might like:

Write an SQL query builder in 150 lines of Python!

Which itself comes with another warning: to handle signals and to execute subprocesses, the event loop must be run in the main thread; this is a sacrifice we're willing to make. ^[return]
It could be argued that it is not ThreadRunner's job to do this, but remember it exists to make it easy to use async code from sync code, and this use case is quite common; also, there is precedent for this kind of API, and something something copy, something something steal. ^[return]

reader 3.7 released – contributor docs

2023-07-15T18:00:00+00:00

Hi there!

I'm happy to announce version 3.7 of reader, a Python feed reader library.

More importantly, reader has reached 300 stars on GitHub!

What's new? #

Here are the highlights since reader 3.4.

Contributor documentation #

reader now has contributor documentation! If you want to help or you're just curious where it's all going, check out the roadmap and the reader philosophy.

Thank you to Katharine Jarmul for getting me to do this! (it only took me five years...)

Explicitly unimportant #

The important entry flag is now bool|None instead of bool,¹ so there's a sane way to express "I don't care about this article".

The point of marking an article as explicitly unimportant is to give you a better understanding of how you consume content – if you consistently don't care about the articles in a feed, maybe it's time to ruthlessly Marie Kondo that crap.

Discontinued plugins #

I removed the Twitter plugin, since it's not possible to get tweets using the free API tier anymore, rendering it effectively useless. While at it, I also removed the Tumblr GDPR plugin (not needed since August 2020).

Nevertheless, they are valuable examples of how to implement reader plugins, so the docs still mention them in Discontinued plugins.

Python versions #

With PyPy 3.10 added in reader 3.7, this is the last release to support Python 3.9.

For more details, see the full changelog.

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

reader is a library
reader is for the long term
reader is extensible
reader is stable (within reason)
reader is simple to use; API matters
reader features work well together
reader is tested
reader is documented
reader has minimal dependencies

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

By the way, I'm quite pleased with how I managed to do it in an (almost) backwards-compatible way. ^[return]

Why you should still read the docs

2023-05-16T18:00:00+00:00

Do you feel you're fighting your tools?

Do you feel you're relying too much on autocomplete and inline documentation? ...always kinda guessing when using libraries?

Or maybe not, but getting things done just seems harder than it should be.

This can have many causes, but I've repeatedly seen junior developers struggle with a specific one, not even fully aware they're struggling in the first place.

This is a story about:

why you should still read the docs
finding the right way of doing things
command-line interfaces

tl;dr: Most good documentation won't show up in your IDE – rather, it is about how to use the library, and the problem the library is solving.

Contents

An example
Good libraries educate
Read the docs
- But the docs suck
- But I'm using lots of libraries
Conclusion

An example #

OK, instead of telling you, let me show you – we'll go through one possible path of developing something, and stop along the way to point out how we're doing things.

Say you're writing a command-line tool that gets data from an API and saves it in a CSV file. It might look something like this:

import csv
import time
import click

def get_data():
    # pretend some API calls happen here
    time.sleep(2)
    return [
        {"name": "Chaos", "short_name": "Chs"},
        {"name": "Discord", "short_name": "Dsc"},
        {"name": "Confusion", "short_name": "Cfn"},
        {"name": "Bureaucracy", "short_name": "Bcy"},
        {"name": "The Aftermath", "short_name": "Afm"},
    ]

def write_csv(data):
    with open("calendar.csv", 'w', newline='') as f:
        writer = csv.writer(f)
        writer.writerow(['name', 'short_name'])
        for row in data:
            writer.writerow([row['name'], row['short_name']])

@click.command()
def main():
    """Retrieve the names of the Discordian seasons to calendar.csv."""
    data = get_data()
    write_csv(data)

if __name__ == '__main__':
    main()

Here it is in action:

$ python seasons.py --help
Usage: seasons.py [OPTIONS]

  Retrieve the names of the Discordian seasons to calendar.csv.

Options:
  --help  Show this message and exit.
$ python seasons.py
$ cat calendar.csv
name,short_name
Chaos,Chs
Discord,Dsc
Confusion,Cfn
Bureaucracy,Bcy
The Aftermath,Afm

You're using Click because it’s powerful but comes with sensible defaults.

This is pretty good code – instead of calling get_data() from write_csv(), you're passing its result to write_csv() – in other words, instead of hiding the input, you've decoupled it.¹

As a result, write_csv() doesn't need to change when adding new arguments to get_data(), and we can test it without monkeypatching get_data():

import seasons

DATA = [
    {"name": "one", "short_name": "1"},
    {"name": "two", "short_name": "2"},
]
CSV_BYTES = b"name,short_name\r\none,1\r\ntwo,2\r\n"

def test_write_csv(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    seasons.write_csv(DATA)
    assert tmp_path.joinpath('calendar.csv').read_bytes() == CSV_BYTES

The path is hardcoded #

But, what if you need to run it twice, and save the output in different files? This becomes useful as soon as the output can change. Also, having to be in the right directory when running it is not cool.

The obvious solution is to pass the path in:

@click.command()
@click.option('--path', required=True)
def main(path):
    """Retrieve the names of the Discordian seasons."""
    data = get_data()
    write_csv(data, path)

def write_csv(data, path):
    with open(path, 'w', newline='') as f:

We're definitely on the right track here – the test is shorter, and we don't even need to monkeypatch the current working directory anymore:

def test_write_csv(tmp_path):
    out_path = tmp_path.joinpath('out.csv')
    seasons.write_csv(DATA, out_path)
    assert out_path.read_bytes() == CSV_BYTES

The path might not be valid #

But, what if you pass a directory as the path?

$ time python seasons.py --path .
Traceback (most recent call last):
  ...
IsADirectoryError: [Errno 21] Is a directory: '.'
python seasons.py --path .  0.07s user 0.02s system 4% cpu 2.104 total

OK, it fails with a traceback. Worse, we waited for the API calls only to throw the output away. Thankfully, Click has fancy parameter types that can take care of that:

@click.command()
@click.option('--path', type=click.Path(exists=False, dir_okay=False, writable=True), required=True)
def main(path):

$ time python seasons.py --path .
Usage: seasons.py [OPTIONS]
Try 'seasons.py --help' for help.

Error: Invalid value for '--path': File '.' is a directory.
python seasons.py --path .  0.08s user 0.02s system 92% cpu 0.112 total

Not only do we get a nice error message, we get it instantly!

The path to what, exactly? #

Thing is, --path isn't very descriptive. I wonder if there's a better name for it...

One thing I like to do is to look at what others are doing.

$ curl --help
Usage: curl [options...] <url>
 -o, --output <file> Write to file instead of stdout
 -O, --remote-name   Write output to a file named as the remote file
 ...

$ wget --help
GNU Wget 1.21.2, a non-interactive network retriever.
Usage: wget [OPTION]... [URL]...
  -o,  --output-file=FILE          log messages to FILE
  -O,  --output-document=FILE      write documents to FILE

$ sort --help
Usage: sort [OPTION]... [FILE]...
  -o, --output=FILE         write result to FILE instead of standard output

$ pandoc --help
pandoc [OPTIONS] [FILES]
  -o FILE               --output=FILE

Better yet, sometimes there's a comprehensive guide on a topic, like Command Line Interface Guidelines ...and under Arguments and flags, we find this:

Use standard names for flags, if there is a standard. If another commonly used command uses a flag name, it’s best to follow that existing pattern. That way, a user doesn’t have to remember two different options (and which command it applies to), and users can even guess an option without having to look at the help text.

Here’s a list of commonly used options: [...]

-o, --output: Output file. For example, sort, gcc.

Standard output #

Further down below, there's another guideline:

If input or output is a file, support - to read from stdin or write to stdout. This lets the output of another command be the input of your command and vice versa, without using a temporary file. [...]

I wonder how we could achieve this.

One way would be to check the value and use sys.stdout if it's -.

But let's pause and look at the Path docs a bit:

allow_dash (bool) – Allow a single dash as a value, which indicates a standard stream (but does not open it). Use open_file() to handle opening this value.

Encouraging; following along, open_file() says:

Open a file, with extra behavior to handle '-' to indicate a standard stream, lazy open on write, and atomic write. Similar to the behavior of the File param type.

What's this File, eh?

Declares a parameter to be a file for reading or writing. [...] The special value - indicates stdin or stdout depending on the mode. [...]

See File Arguments for more information.

Which finally takes us away from the API reference to the actual documentation:²

Since all the examples have already worked with filenames, it makes sense to explain how to deal with files properly. Command line tools are more fun if they work with files the Unix way, which is to accept - as a special file that refers to stdin/stdout.

Click supports this through the click.File type which intelligently handles files for you. [...]

OK, OK, I get it, let's use it:

@click.command()
@click.option('-o', '--output', type=click.File('w', lazy=False), required=True)
def main(output):
    """Retrieve the names of the Discordian seasons."""
    data = get_data()
    # click.File doesn't have a newline argument
    output.reconfigure(newline='')
    write_csv(data, output)

With passing a file object to write_csv(), the output part of I/O is also decoupled:

def write_csv(data, file):
    writer = csv.writer(file)
    writer.writerow(['name', 'short_name'])
    for row in data:
        writer.writerow([row['name'], row['short_name']])

Interestingly enough, csv.writer already takes an open file.

Anyway, it works:

$ python seasons.py --output -
name,short_name
Chaos,Chs
Discord,Dsc
Confusion,Cfn
Bureaucracy,Bcy
The Aftermath,Afm

Once again, the test gets simpler too – instead of writing anything to disk, we can use an in-memory stream:

def test_write_csv():
    file = io.StringIO()
    seasons.write_csv(DATA, file)
    assert file.getvalue().encode() == CSV_BYTES

...by default #

Some of the help texts we looked at say "write to FILE instead of stdout"...

Why instead?

Using commands in a pipeline is common enough that most Unix commands write to stdout by default; some don't even bother with an --output option, since you can always redirect stdout to a file:

$ tail -n+2 calendar.csv | head -n2
Chaos,Chs
Discord,Dsc
$ tail -n+2 calendar.csv > calendar-no-heading.csv
$ head -n2 calendar-no-heading.csv
Chaos,Chs
Discord,Dsc

It is so common that CLI Guidelines makes it the third thing in The Basics:

Send output to stdout. The primary output for your command should go to stdout. Anything that is machine readable should also go to stdout—this is where piping sends things by default.

We could remove --output, but with Click, doing both is trivially easy:

@click.option('-o', '--output', type=click.File('w', lazy=False), default='-')

$ python seasons.py | tail -n+2 | head -n2
Chaos,Chs
Discord,Dsc

Discussion #

Looking at our journey, I can't help but notice a few things:

The Click API reference seems to constantly funnel readers to the user guide part of the documentation, the part that tells you how and why to use stuff.
Click mentions supporting - for stdout, and more than once.
The Click user guide discusses File arguments before Path (in part, because it's more commonly used, but also because it's more useful out of the box).
csv.writer takes an open file, just like our own write_csv() does at the end.

These all hint that it may be a good idea to structure code a certain way; the more we use things the way they "want" to be used, the cleaner the code and the tests get.

Good libraries educate #

I am tempted to call Click a great library.

Yes, it does its job more than well, and has lots of features.

But, more importantly, its documentation educates the reader about the subject matter – it goes beyond API docs into how to use it, how to get specific things done with it, and the underlying problem it is solving.

Here are a few more examples:

Classics – other libraries in the Flask/Pallets ecosystem (like Werkzeug and Jinja), Requests, pytest, and many others – popular and old enough libraries usually have great documentation.
Python itself – you would be amazed at how many developers use Python every day without having gone through the tutorial even once.
feedparser teaches you about all kinds of Atom and RSS quirks, encodings on the web, HTTP, and how to be nice to servers you're making requests to.
- feedparser is close to my heart because I'm using it for my feed reader library, reader, where I too am doing my best to have docs worth reading.
Stepping away from Python a bit, I've learned a lot from both Dagger and Mockito (and for the latter, the API docs for the "main" class pack way more than just API, and that will show up in your IDE).
Beancount, a command-line accounting tool, does a great job of teaching double-entry bookkeeping (it taught me almost all I know about the topic).

Read the docs #

In a way, this whole article is a ruse to tell you to RTFM, but with a bit of nuance.

Read the documentation to find out how, and more importantly, why – a mental model of the thing you're using will make you more productive, and will improve any guesses you make.

Aside from the user guide part, I also like to skim the reference for libraries I use a lot; this gives me an idea of what's available, and often I remember later that "hey, there was something in the docs about X".

But the docs suck #

Sometimes, documentation sucks or is missing entirely (doubly so for internal code). It's easy to get a habit of not bothering to look, when most of the time it's not there.

Still, always check. Worst case, you'll have wasted a minute.

On the other hand, docs should definitely be one of the criteria when picking a library.

But I'm using lots of libraries #

Quoting a friend:

Well, for you, it might be "a library", but for them it's 10 new libraries, and a 2-day deadline.

Like with most things, the Pareto principle applies:

Read the docs for big stuff you use often – language, standard library, test framework, web framework or database of choice, and so on; these have an outsized return on investment, and start paying out quite quickly.
Don't read all the docs; skimming is often enough.

On the other hand, there is value in not using that many libraries.

Conclusion #

To sum up:

Good libraries educate on the subject matter.
Read the documentation of the things you are using a lot.
If you feel you're fighting your tools, maybe you're using them wrong (read the docs!), but maybe they're not all that good; it's OK to look for better ones.

I'll end with a related idea from a post called Why I Don’t Use Autocomplete:

But be sure that if you do use autocomplete that you’re not leaning on it as a crutch. Be sure that you’re not deciding what to write as you write it [...] and not just using a method because the IDE tells you it’s available.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

For a detailed example about decoupling I/O, how it affects affects testing, plus interesting historical background, check out Brandon Rhodes' great talk The Clean Architecture in Python. ^[return]
Is it not written, "A few hours of trial and error can save you five minutes of reading the docs"? ^[return]

Announcing linesieve: an unholy blend of grep, sed, awk, and Python, born out of spite

2023-04-26T16:00:00+00:00

Java is notoriously verbose, especially when used in a serious Enterprise Project™.

...so naturally, I made linesieve, a Python text munging tool to split output into sections and match/sub/split with the full power of Python's re module.

Here's an example of using it on a file listing (delay added to make it look cool):

ls -1 /* | linesieve -s '.*:' show bin match ^d head -n2 output

You can find short examples in the reference, and an advanced example below.

Features #

linesieve allows you to:

split text input into sections
apply filters to specific sections
search and highlight success/failure markers
match/sub/split with the full power of Python's re module
shorten paths, links and module names
chain filters into pipelines
colors!

Here's a list of filters available in 1.0:

head – Output the first part of sections.
tail – Output the last part of sections.
span – Output matching line spans.
match – Search for pattern (think grep).
split – Output selected parts of lines (think cut).
sub – Replace pattern (think sed s///g).
sub-paths – Shorten paths of existing files.
sub-cwd – Make working directory paths relative.
sub-link – Replace symlink targets.

Installing #

Install it using pip:

$ pip install --upgrade linesieve

Links #

You can find:

the documentation at Read the Docs
the code on GitHub
the latest release on PyPI

Example #

Here's a juicy Java example – this linesieve command:

linesieve \
span -v -X \
    --start '^ (\s+) at \s ( org\.junit\. | \S+ \. reflect\.\S+\.invoke )' \
    --end '^ (?! \s+ at \s )' \
    --repl '\1...' \
match -v '^\s+at \S+\.(rethrowAs|translateTo)IOException' \
sub-paths --include '{src,tst}/**/*.java' --modules-skip 1 \
sub -X '^( \s+ at \s+ (?! .+ \.\. | com\.example\. ) .*? ) \( .*' '\1' \
sub -X '^( \s+ at \s+ com\.example\. .*? ) \ ~\[ .*' '\1' \
sub -X '
    (?P<pre> \s+ at \s .*)
    (?P<cls> \w+ )
    (?P<mid> .* \( )
    (?P=cls) \.java
    (?P<suf> : .* )
    ' \
    '\g<pre>\g<cls>\g<mid>\g<suf>'

... shortens this 76 line traceback:

12:34:56.789 [main] ERROR com.example.someproject.somepackage.ThingDoer - exception while notifying done listener
java.lang.RuntimeException: listener failed
	at com.example.someproject.somepackage.ThingDoerTest$DummyListener.onThingDone(ThingDoerTest.java:420) ~[tests/:?]
	at com.example.someproject.somepackage.ThingDoer.doThing(ThingDoer.java:69) ~[library/:?]
	at com.example.otherproject.Framework.doAllTheThings(Framework.java:1066) ~[example-otherproject-2.0.jar:2.0]
	at com.example.someproject.somepackage.ThingDoerTest.listenerException(ThingDoerTest.java:666) ~[tests/:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	...
	... 60+ more lines of JUnit stuff we don't really care about ...
	...
12:34:56.999 [main] INFO done

... to just 7 lines:

12:34:56.789 [main] ERROR ..ThingDoer - exception while notifying done listener
java.lang.RuntimeException: listener failed
	at ..ThingDoerTest$DummyListener.onThingDone(:420) ~[tests/:?]
	at ..ThingDoer.doThing(:69) ~[library/:?]
	at com.example.otherproject.Framework.doAllTheThings(:1066)
	at ..ThingDoerTest.listenerException(:666) ~[tests/:?]
	...
12:34:56.999 [main] INFO done

Let's break that down a bit:

span gets rid of all the traceback lines coming from JUnit.
match -v skips some usually useless lines from stack traces.
sub-paths shortens and highlights the names of classes in the current project; com.example.someproject.somepackage.ThingDoer becomes ..ThingDoer (presumably that's enough info to open the file in your IDE).
The first sub gets rid of line numbers and JAR names for everything that's not either in the current project or in another com.example. package.
The second sub gets rid of JAR names for things in other com.example. packages.
The third sub gets rid of the source file name; ..ThingDoer.doThing(ThingDoer.java:69) becomes ..ThingDoer.doThing(:69) (in Java, the file name matches the class name).

For an even more advanced example, see how to clean up Apache Ant output.

Anyway, that's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Limiting concurrency in Python asyncio: the story of async imap_unordered()

2024-01-30T08:15:00+00:00

So, you're doing some async stuff, repeatedly, many times.

Like, hundreds of thousands of times.

Maybe you're scraping some data.

Maybe it's more complicated – you're calling an API, and then passing the result to another one, and then saving the result of that.

Either way, it's a good idea to not do it all at once. For one, it's not polite to the services you're calling. For another, it'll load everything in memory, all at once.

In sync code, you might use a thread pool and imap_unordered():

pool = multiprocessing.dummy.Pool(2)
for result in pool.imap_unordered(do_stuff, things_to_do):
    print(result)

Here, concurrency is limited by the fixed number of threads.

But what about async code? In this article, we'll look at a few ways of limiting concurrency in asycio, and find out which one is best.

Tip

No, it's not Semaphore, despite what Stack Overflow may tell you.

Tip

If you're in a hurry – it's wait().

Contents

Getting started
asyncio.gather()
asyncio.Semaphore
asyncio.as_completed()
asyncio.Queue
Aside: backpressure
asyncio.wait()
Async iterables
Bonus: exceptions
Bonus: better decorators?

Getting started #

In order to try things out more easily, we'll start with a test harness of sorts.

Our async map_unordered() behaves pretty much like imap_unordered() – it takes a coroutine function and an iterable of arguments, and runs the resulting awaitables limit at a time:

def map_unordered(func, iterable, *, limit):
    aws = map(func, iterable)
    return limit_concurrency(aws, limit)

The actual running is done by limit_concurrency(). For now, we run them one by one (we'll get back to this later on):

async def limit_concurrency(aws, limit):
    aws = iter(aws)
    while True:
        try:
            aw = next(aws)
        except StopIteration:
            break
        yield await aw

To simulate work being done, we just sleep():

async def do_stuff(i):
    1 / i  # raises ZeroDivisionError for i == 0
    await asyncio.sleep(i)
    return i

Putting it all together, we get a map_unordered.py LIMIT TIME... script that does stuff in parallel, printing timings as we get each result:

async def async_main(args, limit):
    start = time.monotonic()
    async for result in map_unordered(do_stuff, args, limit=limit):
        print(f"{(time.monotonic() - start):.1f}: {result}")
    print(f"{(time.monotonic() - start):.1f}: done")


def main():
    limit = int(sys.argv[1])
    args = [float(n) for n in sys.argv[2:]]
    timeout = sum(args) + 0.1
    asyncio.run(asyncio.wait_for(async_main(args, limit), timeout))


if __name__ == '__main__':
    main()

... like so:

$ python map_unordered.py 2
0.0: done
$ python map_unordered.py 2 .1 .2
0.1: 0.1
0.3: 0.2
0.3: done

Tip

If you need a refresher on lower level asyncio stuff related to waiting, check out Hynek Schlawack's excellent Waiting in asyncio.

asyncio.gather() #

In the Running Tasks Concurrently section of the asyncio docs, we find asyncio.gather(), which runs awaitables concurrently and returns their results.

We can use it to run limit-sized batches:

async def limit_concurrency(aws, limit):
    aws = iter(aws)
    while True:
        batch = list(itertools.islice(aws, limit))
        if not batch:
            break
        for result in await asyncio.gather(*batch):
            yield result

This seems to work:

$ python map_unordered.py 2 .1 .2
0.2: 0.1
0.2: 0.2
0.2: done

... except:

$ python map_unordered.py 2 .1 .2 .2 .1
0.2: 0.1
0.2: 0.2
0.4: 0.2
0.4: 0.1
0.4: done

... those should fit in 0.3 seconds:

| sleep(.1) |       sleep(.2)       |
|       sleep(.2)       | sleep(.1) |

... but we're waiting for the entire batch to finish, even if some tasks finish earlier:

| sleep(.1) |...........|       sleep(.2)       |
|       sleep(.2)       | sleep(.1) |...........|

asyncio.Semaphore #

Screw the docs, too much to read; after some googling, the first few Stack Overflow answers all point to asyncio.Semaphore.

Like its threading counterpart, we can use it to limit how many times the body of a with block is entered in parallel:

async def limit_concurrency(aws, limit):
    semaphore = asyncio.Semaphore(limit)

    async def wrapper(aw):
        async with semaphore:
            return await aw

    for result in await asyncio.gather(*map(wrapper, aws)):
        yield result

This works:

$ python map_unordered.py 2 .1 .2 .2 .1
0.3: 0.1
0.3: 0.2
0.3: 0.2
0.3: 0.1
0.3: done

... except, because gather() takes a sequence, we end up consuming the entire aws iterable before gather() is even called. Let's highlight this:

def on_iter_end(it, callback):
    for x in it:
        yield x
    callback()

    timeout = sum(args) + 0.1
    args = on_iter_end(args, lambda: print("iter end"))
    asyncio.run(asyncio.wait_for(async_main(args, limit), timeout))

As expected:

$ python map_unordered.py 2 .1 .2 .2 .1
iter end
0.3: 0.1
0.3: 0.2
0.3: 0.2
0.3: 0.1
0.3: done

For small iterables, this is fine, but for bigger ones, creating all the tasks upfront without running them might cause memory issues. Also, if the iterable is lazy (e.g. it comes from a paginated API), we only start work after it's all consumed in memory, instead processing it in a streaming fashion.

asyncio.as_completed() #

At a glance, asyncio.as_completed() might do what we need – it takes an iterable of awaitables, runs them concurrently, and returns an iterator of coroutines that "can be awaited to get the earliest next result from the iterable of the remaining awaitables".

Sadly, it still consumes the iterable right away:

def as_completed(fs, *, timeout=None):
    ...  # set-up
    todo = {ensure_future(f, loop=loop) for f in set(fs)}
    ...  # actual logic

But there's another, subtler issue.

as_completed() has no limits of its own – it's up to us to limit how fast we feed it awaitables. Presumably, we could wrap the input iterable into a generator that yields awaitables only if enough results came out the other end, and waits otherwise.

However, due to historical reasons, as_completed() takes a plain-old-sync-iterator – we cannot await anything in its (sync) __next__(), and sync waiting of any kind would block (and possibly deadlock) the entire event loop.

So, no as_completed() for you.

asyncio.Queue #

Speaking of threading counterparts, how would you implement imap_unordered() if there was no Pool? Queues, of course!

And asyncio has its own Queue, which you use in pretty much the same way: start limit worker tasks that loop forever, each pulling awaitables, awaiting them, and putting the results into a queue.

async def limit_concurrency(aws, limit):
    aws = iter(aws)
    queue = asyncio.Queue()
    ndone = 0

    async def worker():
        while True:
            try:
                aw = next(aws)
            except StopIteration:
                await queue.put((False, None))
                break

            try:
                await queue.put((True, await aw))
            except Exception as e:
                await queue.put((False, e))
                break

    worker_tasks = [asyncio.create_task(worker()) for _ in range(limit)]

    while ndone < limit or not queue.empty():
        ok, rv = await queue.get()
        if ok:
            yield rv
        elif rv:
            raise rv
        else:
            ndone += 1

The iterable is exhausted before the last "batch" starts:

$ python map_unordered.py 2 .1 .2 .3 .3 .2 .1
0.1: 0.1
0.2: 0.2
0.4: 0.3
0.5: 0.3
iter end
0.6: 0.1
0.6: 0.2
0.6: done

I was going to work up to this in a few steps, but I'll just point out three common bugs this type of code might have (that apply to threads too).

First, we could increment ndone from the worker, but this makes await queue.get() hang forever for empty iterables, since workers never get to run by the time we get to it; because there's no other await, it's not even a race condition.

async def limit_concurrency(aws, limit):
    aws = iter(aws)
    queue = asyncio.Queue()
    ndone = 0

    async def worker():
        nonlocal ndone

        while True:
            try:
                aw = next(aws)
            except StopIteration:
                ndone += 1
                break

            await queue.put(await aw)

    worker_tasks = [asyncio.create_task(worker()) for _ in range(limit)]

    while ndone < limit or not queue.empty():
        yield await queue.get()

$ python map_unordered.py 2
iter end
Traceback (most recent call last):
  ...
asyncio.exceptions.TimeoutError

The solution is to signal the worker is done in-band, by putting a sentinel on the queue. I guess a good rule of thumb is that you want a put() for each get() without a timeout.¹

Second, you have to catch all exceptions; otherwise, the worker gets killed, and get() waits forever for a sentinel that will never come.

async def limit_concurrency(aws, limit):
    aws = iter(aws)
    queue = asyncio.Queue()
    ndone = 0

    done = object()

    async def worker():
        while True:
            try:
                aw = next(aws)
            except StopIteration:
                await queue.put(done)
                break

            await queue.put(await aw)

    worker_tasks = [asyncio.create_task(worker()) for _ in range(limit)]

    while ndone < limit or not queue.empty():
        rv = await queue.get()
        if rv is done:
            ndone += 1
            continue
        yield rv

$ python map_unordered.py 2 .1 .2 0 .2 .1
0.1: 0.1
0.2: 0.2
0.4: 0.2
iter end
0.5: 0.1
Traceback (most recent call last):
  ...
asyncio.exceptions.TimeoutError
Task exception was never retrieved
future: <Task finished name='Task-3' coro=<limit_concurrency.<locals>.worker() done, defined at map_unordered.py:20> exception=ZeroDivisionError('float division by zero')>
Traceback (most recent call last):
  ...
ZeroDivisionError: float division by zero

Finally, our input iterator is synchronous (for now), so no other task can run during next(aws). But if it were async, any number of tasks could await anext(aws) in parallel, leading to concurrency issues. The fix is the same as with threads: either protect that call with a Lock, or feed awaitables to workers through an input queue.

Anyway, no need to worry about any of that – a better solution awaits.

Aside: backpressure #

At this point, we're technically done – the queue solution does everything Pool.imap_unordered() does.

So much so, that, like imap_unordered(), it lacks backpressure: when code consuming results from map_unordered() cannot keep up with the tasks producing them, the results accumulate in the internal queue, with potentially infinite memory usage.

>>> pool = multiprocessing.dummy.Pool(1)
>>> for _ in pool.imap_unordered(print, range(4)):
...     time.sleep(.1)
...     print('got result')
...
0
1
2
3
got result
got result
got result
got result

>>> async def async_print(arg):
...     print(arg, '(async)')
...
>>> async for _ in map_unordered(async_print, range(4), limit=1):
...     await asyncio.sleep(.1)
...     print('got result')
...
0 (async)
1 (async)
2 (async)
3 (async)
got result
got result
got result
got result

To fix this, we make the queue bounded, so that workers block while the queue is full.

    queue = asyncio.Queue(limit)

>>> async for _ in map_unordered(async_print, range(5), limit=1):
...     await asyncio.sleep(.1)
...     print('got result')
...
0 (async)
1 (async)
2 (async)
got result
3 (async)
got result
4 (async)
got result
got result
got result

Alas, we can't do the same thing for Pool.imap_unordered() because don't have access to its queue, but that's a story for another time.

asyncio.wait() #

Pretending we're using threads works, but it's not all that idiomatic.

If only there was some sort of low level, select()-like primitive taking a set of tasks and blocking until at least one of them finishes. And of course there is – we've been purposefully avoiding it this entire time – it's asyncio.wait(), and it does exactly that.

By default, it waits until all tasks are completed, which isn't much better than gather().

But, with return_when=FIRST_COMPLETED, it waits until at least one task is completed. We can use this to keep a limit-sized set of running tasks updated with new tasks as soon as the old ones finish:

async def limit_concurrency(aws, limit):
    aws = iter(aws)
    aws_ended = False
    pending = set()

    while pending or not aws_ended:
        while len(pending) < limit and not aws_ended:
            try:
                aw = next(aws)
            except StopIteration:
                aws_ended = True
            else:
                pending.add(asyncio.ensure_future(aw))

        if not pending:
            return

        done, pending = await asyncio.wait(
            pending, return_when=asyncio.FIRST_COMPLETED
        )
        while done:
            yield done.pop()

We change limit_concurrency() to yield awaitables instead of results, so it's more symmetric – awaitables in, awaitables out. map_unordered() then becomes an async generator function, instead of a sync function returning an async generator. This is functionally the same, but does make it a bit more self-documenting.

async def map_unordered(func, iterable, *, limit):
    aws = map(func, iterable)
    async for task in limit_concurrency(aws, limit):
        yield await task

This implementation has all the properties that the Queue one has:

$ python map_unordered.py 2 .1 .2 .2 .1
0.1: 0.1
0.2: 0.2
0.3: 0.1
0.3: 0.2
iter end
0.3: done

... and backpressure too:

>>> async for _ in map_unordered(async_print, range(4), limit=1):
...     await asyncio.sleep(.1)
...     print('got result')
...
0 (async)
got result
1 (async)
got result
2 (async)
got result
3 (async)
got result

Liking this so far? Here's another article you might like:

Running async code from sync code in Python

Async iterables #

OK, but what if we pass map_unordered() an asynchronous iterable? We are talking about async stuff, after all.

This opens up a whole looking-glass world of async iteration: instead of iter() you have aiter(), instead of next() you have anext(), some of them you await, some you don't... Thankfully, we can support both without making things much worse.

And we don't need to be particularly smart about it either; we can just feed the current code an async iterable from main(), and punch our way through the exceptions:

async def as_async_iter(it):
    for x in it:
        yield x

    args = on_iter_end(args, lambda: print("iter end"))
    args = as_async_iter(args)
    asyncio.run(asyncio.wait_for(async_main(args, limit), timeout))

$ python map_unordered.py 2 .1 .2 .2 .1
Traceback (most recent call last):
  ...
  File "map_unordered.py", line 9, in map_unordered
    aws = map(func, iterable)
TypeError: 'async_generator' object is not iterable

map() doesn't work with async iterables, so we use a generator expression instead.

async def map_unordered(func, iterable, *, limit):
    try:
        aws = map(func, iterable)
    except TypeError:
        aws = (func(x) async for x in iterable)

    async for task in limit_concurrency(aws, limit):
        yield await task

In true easier to ask for forgiveness than permission style, we handle the exception from map() instead of, say, checking if aws is an instance of collections.abc.Iterable.

We could wrap aws to always be an async iterable, but limit_concurrency() is useful on it its own, so it's better to support both.

$ python map_unordered.py 2 .1 .2 .2 .1
Traceback (most recent call last):
  ...
  File "map_unordered.py", line 19, in limit_concurrency
    aws = iter(aws)
TypeError: 'async_generator' object is not iterable

For async iterables, we need to use aiter():

async def limit_concurrency(aws, limit):
    try:
        aws = aiter(aws)
        is_async = True
    except TypeError:
        aws = iter(aws)
        is_async = False

$ python map_unordered.py 2 .1 .2 .2 .1
Traceback (most recent call last):
  ...
  File "map_unordered.py", line 32, in limit_concurrency
    aw = next(aws)
TypeError: 'async_generator' object is not an iterator

... and anext():

        while len(pending) < limit and not aws_ended:
            try:
                aw = await anext(aws) if is_async else next(aws)
            except StopAsyncIteration if is_async else StopIteration:
                aws_ended = True
            else:
                pending.add(asyncio.ensure_future(aw))

... which unlike aiter(), has to be awaited.

Here's limit_concurrency() in its entire glory:

async def limit_concurrency(aws, limit):
    try:
        aws = aiter(aws)
        is_async = True
    except TypeError:
        aws = iter(aws)
        is_async = False

    aws_ended = False
    pending = set()

    while pending or not aws_ended:
        while len(pending) < limit and not aws_ended:
            try:
                aw = await anext(aws) if is_async else next(aws)
            except StopAsyncIteration if is_async else StopIteration:
                aws_ended = True
            else:
                pending.add(asyncio.ensure_future(aw))

        if not pending:
            return

        done, pending = await asyncio.wait(
            pending, return_when=asyncio.FIRST_COMPLETED
        )
        while done:
            yield done.pop()

Not as clean as before, but it gets the job done:

$ python map_unordered.py 2 .1 .2 .2 .1
0.1: 0.1
0.2: 0.2
0.3: 0.1
0.3: 0.2
iter end
0.3: done

Anyway, that's it for now. Here's the final version of the code.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

If you've made it this far, you might like:

Why you should still read the docs

Bonus: exceptions #

OK, so what about exceptions?

A lot of times, you still want to do the rest of the things, even if one fails. Also, you probably want to know which one failed, but the map_unordered() results are not in order, so how could you tell?

The most flexible solution is to let the user handle it just like they would with Pool.imap_unordered() – by decorating the original function. Here's one way of doing it:

def return_args_and_exceptions(func):
    return functools.partial(_return_args_and_exceptions, func)

async def _return_args_and_exceptions(func, *args):
    try:
        return *args, await func(*args)
    except Exception as e:
        return *args, e

    wrapped = return_args_and_exceptions(do_stuff)
    async for arg, result in map_unordered(wrapped, args, limit=limit):
        print(f"{(time.monotonic() - start):.1f}: {arg} -> {result}")

$ python map_unordered.py 2 .1 .2 0 .2 .1
0.1: 0.1 -> 0.1
0.1: 0.0 -> float division by zero
0.2: 0.2 -> 0.2
0.3: 0.2 -> 0.2
0.3: 0.1 -> 0.1
iter end
0.3: done

Bonus: better decorators? #

Finally, here's a cool thing I learned from the asyncio docs.

When writing decorators, you can use partial() to bind the decorated function to an existing wrapper, instead of always returning a new one. The result is a more descriptive representation:

>>> return_args_and_exceptions(do_stuff)
functools.partial(<function _return_args_and_exceptions at 0x10647fd80>, <function do_stuff at 0x10647d8a0>)

Compare with the traditional version:

def return_args_and_exceptions(func):
    async def wrapper(*args):
        ...
    return wrapper

>>> return_args_and_exceptions(do_stuff)
<function return_args_and_exceptions.<locals>.wrapper at 0x103993560>

Does this have a fancy, academic name? Do let me know! ^[return]

yaml: while constructing a mapping found unhashable key

2023-02-21T23:02:21+00:00

So you're trying to read some YAML using PyYAML, and get an exception like this:

>>> yaml.safe_load("""\
... [0, 0]: top-left
... [1, 1]: bottom-right
... """)
Traceback (most recent call last):
  ...
yaml.constructor.ConstructorError: while constructing a mapping
found unhashable key
  in "<unicode string>", line 1, column 1:
    [0, 0]: top-left
    ^

What does it mean? #

The error message is pretty self-explanatory, but let's unpack it a bit.

First, it happened during construction – that is, while converting the generic representation of the YAML document to native data structures; in this case, converting a mapping to a Python dict.

YAML Processing Overview

The problem is that a key of the mapping, [0, 0], is not hashable:

An object is hashable if it has a hash value which never changes during its lifetime (it needs a __hash__() method), and can be compared to other objects (it needs an __eq__() method). Hashable objects which compare equal must have the same hash value.

Most immutable built-ins are hashable; mutable containers (such as lists) are not; immutable containers (such as tuples) are hashable only if their elements are.

Why does this happen? #

This is not a limitation of YAML itself; quoting the spec:

The content of a mapping node is an unordered set of key/value node pairs, with the restriction that each of the keys is unique. YAML places no further restrictions on the nodes. In particular, keys may be arbitrary nodes, the same node may be used as the value of several key/value pairs and a mapping could even contain itself as a key or a value.

We can load everything up to the representation:

>>> yaml.compose("[0, 0]: top-left")
MappingNode(
    tag='tag:yaml.org,2002:map',
    value=[
        (
            SequenceNode(
                tag='tag:yaml.org,2002:seq',
                value=[
                    ScalarNode(tag='tag:yaml.org,2002:int', value='0'),
                    ScalarNode(tag='tag:yaml.org,2002:int', value='0'),
                ],
            ),
            ScalarNode(tag='tag:yaml.org,2002:str', value='top-left'),
        )
    ],
)

The limitation comes from how dicts are implemented, specifically:

Hashability makes an object usable as a dictionary key and a set member, because these data structures use the hash value internally.

>>> {[0, 0]: "top-left"}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

If we use a (hashable) tuple instead, it works:

>>> {(0, 0): "top-left"}
{(0, 0): 'top-left'}

What now? #

Use the representation #

Depending on your needs, the representation might be enough. But probably not...

Use full_load() and Python-specific tags #

If you control the input, and you're OK with language-specific tags, use full_load(); it resolves all tags except those known to be unsafe, including all the Python-specific tags listed here.

>>> yaml.full_load("""\
... !!python/tuple [0, 0]: top-left
... !!python/tuple [1, 1]: bottom-right
... """)
{(0, 0): 'top-left', (1, 1): 'bottom-right'}

You could also use unsafe_load(), but most of the time it's not what you want:

Warning

yaml.unsafe_load() is unsafe for untrusted data, because it allows running arbitrary code. Consider using safe_load() or full_load() instead.

For example, you can do this:

>>> yaml.unsafe_load("!!python/object/new:os.system [echo WOOSH. YOU HAVE been compromised]")
WOOSH. YOU HAVE been compromised
0

There were a bunch of CVEs about it.

But, I don't control the input #

If you don't control the input, you can use a custom constructor to convert the keys to something hashable. Here, we convert list keys to tuples:

class Loader(yaml.SafeLoader):
    pass

def construct_mapping(self, node):
    pairs = self.construct_pairs(node, deep=True)
    try:
        return dict(pairs)
    except TypeError:
        rv = {}
        for key, value in pairs:
            if isinstance(key, list):
                key = tuple(key)
            rv[key] = value
        return rv

Loader.construct_mapping = construct_mapping
Loader.add_constructor('tag:yaml.org,2002:map', Loader.construct_mapping)

>>> yaml.load("""\
... [0, 0]: top-left
... [1, 1]: bottom-right
... """, Loader=Loader)
{(0, 0): 'top-left', (1, 1): 'bottom-right'}

We subclass SafeLoader to account for a PyYAML quirk – calling add_constructor() directly would modify it in-place, for everyone, which isn't necessarily great. We still override construct_mapping so that other constructors wanting to make a mapping get to use our version.

Alas, this is quite limited, because new key types need to be handled explicitly; for example, we might be able to convert dicts to a frozenset of items().

>>> yaml.load("{0: 1}: top-left", Loader=Loader)
Traceback (most recent call last):
  ...
    rv[key] = value
TypeError: unhashable type: 'dict'

But nested keys don't work either, we need to convert them recursively ourselves:

>>> yaml.load("[[0]]: top-left", Loader=Loader)
Traceback (most recent call last):
  ...
    rv[key] = value
TypeError: unhashable type: 'list'

But, I don't know the key types in advance #

A decent trade-off is to just let the mapping devolve into a list of pairs:

def construct_mapping(self, node):
    pairs = self.construct_pairs(node)
    try:
        return dict(pairs)
    except TypeError:
        return pairs

>>> yaml.load("""\
... [0, 0]: top-left
... [1, 1]: bottom-right
... """, Loader=Loader)
[([0, 0], 'top-left'), ([1, 1], 'bottom-right')]
>>> yaml.load("{0: 1}: top-left", Loader=Loader)
[({0: 1}, 'top-left')]

But, I need to round-trip the data #

This works, until you need to round-trip the data, or emit this kind of YAML yourself.

Luckily, I've already written a whole article on how to do that, complete with code; the trick is to mark the list of pairs in some way:

>>> value = yaml.load("""\
... [0, 0]: top-left
... [1, 1]: bottom-right
... """, Loader=Loader)
>>> value
Pairs([([0, 0], 'top-left'), ([1, 1], 'bottom-right')])

... so you can represent it back into a mapping:

>>> print(yaml.dump(value, Dumper=Dumper))
? - 0
  - 0
: top-left
? - 1
  - 1
: bottom-right

(The fancy ? syntax indicates a complex mapping key, but that's just another way of writing the original input.)

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Caching a lot of methods in Python

2023-02-13T20:00:00+00:00

So, you're using a Python API client to get a bunch of data, and you need to cache results and retry on errors.

You could write a wrapper class, but that's just one more thing to test and maintain.¹

Thankfully, this is Python, and more often than not, there must be a better way.

Contents

But I don't own the client code
But decorating methods keeps instances around for too long
But I need to cache a lot of methods
But I don't instantiate the client
But the proxy breaks completion
But the proxy fails isinstance() checks
But I only need to cache a few methods

Note

In this article, I will focus on caching using functools.lru_cache(). Retrying with a library like tenacity would work more or less the same.

But I don't own the client code #

We'll assume a client that looks like this (but keep in mind it's not our code – we can only use it, not change it):

class Client:

    def one(self, arg):
        print(f"Client.one({arg!r})")
        return arg.upper()

    def two(self, arg):
        print(f"Client.two({arg!r})")
        return arg.title()

If Client were our code, we could just slap @functools.lru_cache on the methods.

As is, we can subclass it, override each method, and decorate the overrides:

class SubclassOverride(Client):

    @functools.lru_cache(maxsize=1000)
    def one(self, arg):
        return super().one(arg)

    @functools.lru_cache(maxsize=1000)
    def two(self, arg):
        return super().two(arg)

Of course, this doesn't really help with maintainability, but it does give us an excuse to look at a potential issue with lru_cache().

But decorating methods keeps instances around for too long #

The lru_cache() docs say,

If a method is cached, the self instance argument is included in the cache. See How do I cache method calls?

... in turn:

The lru_cache approach [...] creates a reference to the instance [...] The disadvantage is that instances are kept alive until they age out of the cache or until the cache is cleared.

Depending on how often Client is instantiated, this might count as a memory and/or resource leak. For a more detailed explanation, see this article.

One solution is to decorate the bound method, after the object is created:

class SubclassShadow(Client):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.one = functools.lru_cache(maxsize=1000)(self.one)
        self.two = functools.lru_cache(maxsize=1000)(self.two)

This way, the instance is not in the cache keys, and we have one cache per instance.

There's still a reference cycle between the cache and the instance (via the bound method), but the garbage collector will take care of that – normally, all three go away at the same time. Before, the cache had the same lifetime as the unbound method (that is, the same lifetime as the class).

Important

You might be tempted to say, "but if I call one(arg) on two different instances, the underlying method will be called twice".

True, but this is the opposite of a problem, it's likely the correct behavior. Because each instance has different state, the results of the calls may be different (it helps to think of instance attributes as implicit arguments passed to each method); for example, clients instantiated with different users should not return the same thing.

But I need to cache a lot of methods #

OK, the above is a bit less verbose, but we still have to decorate methods one by one.

Thankfully, we can do it on the fly:

class SubclassDynamic(Client):

    def __getattribute__(self, name):
        # conveniently, also prevents infinite recursion
        if name.startswith('_'):
            return getattr(super(), name)
        try:
            return self.__dict__[name]
        except KeyError:
            pass
        value = getattr(super(), name)
        if not callable(value):
            return value
        cached = functools.lru_cache(maxsize=1000)(value)
        self.__dict__[name] = cached
        return cached

__getattribute__() allows us to intercept all attribute accesses (including methods). When a method is requested, we get it from the parent, decorate it, and store it in the instance dictionary; the second time around, we return the already decorated method.

We don't decorate methods whose names start with an underscore, including any __magic__ or _private ones; additional filtering logic (e.g. only cache methods starting with get_) would go here as well.

Note we can't use __getattr__(), because it's only called if the attribute is not found through the normal mechanism – and looking up the class tree is part of the normal mechanism.

We could just decorate all the methods upfront, in __init__(), but that might not work if the parent does magic stuff without implementing __dir__() properly.

But I don't instantiate the client #

Sometimes, you're not the one instantiating the client; it may be created by a callable factory that doesn't allow passing in a different class,² or you might just get it from a framework.

No worries, subclassing is not required for Python metaprogramming – instead, we can wrap the instance in an object proxy (also known as dynamic wrapper).

The code looks pretty much the same, but instead of delegating to the parent, we delegate to the wrapped instance:

class Proxy:

    def __init__(self, wrapped):
        self.__wrapped__ = wrapped

    def __getattr__(self, name):
        value = getattr(self.__wrapped__, name)
        if name.startswith('_'):
            return value
        if not callable(value):
            return value
        cached = functools.lru_cache(maxsize=1000)(value)
        setattr(self, name, cached)
        return cached

Because the proxy is a separate object, now we can use __getattr__(), and we don't have to check the instance dictionary explicitly – on the second call, the decorated method is already there, so __getattr__() isn't called anymore.

But the proxy breaks completion #

Oops, we're guilty of the not implementing __dir__() mentioned before.

Among others, this breaks completion:

>>> client = Proxy(Client())
>>> 'one' in dir(client)
False
>>> 'one' in dir(client.__wrapped__)
True
>>> client.<TAB>
           ... crickets ...

... but it's pretty easy to fix:

    def __dir__(self):
        return dir(self.__wrapped__)

>>> client = Proxy(Client())
>>> 'one' in dir(client)
True

With this, we have a solution that covers almost everything I've met in practice.

But the proxy fails isinstance() checks #

Turns out, writing a perfect proxy is pretty difficult.

For example, although not recommended, code may need to do isinstance() checks:

>>> isinstance(client, Client)
False

We could fix this by implementing __instancecheck__(), but I thought we were past wrapping things one by one. wrapt gives us an almost-perfect proxy:

class WraptProxy(wrapt.ObjectProxy):

    def __init__(self, wrapped):
        super().__init__(wrapped)
        self._self_cached_ones = {}

    def __getattr__(self, name):
        try:
            return self._self_cached_ones[name]
        except KeyError:
            pass
        value = super().__getattr__(name)
        if name.startswith('_'):
            return value
        if not callable(value):
            return value
        cached = functools.lru_cache(maxsize=1000)(value)
        self._self_cached_ones[name] = cached
        return cached

It's not as elegant as ours, but check this out:

>>> client = WraptProxy(Client())
>>> client
<WraptProxy at 0x106c2f5c0 for Client at 0x106c2ec10>
>>> print(client)
<__main__.Client object at 0x10d71ad90>
>>> client.__class__
<class '__main__.Client'>
>>> isinstance(client, Client)
True
>>> 'one' in dir(client)
True

But I only need to cache a few methods #

OK, all that seems a bit excessive if you only need to cache a couple of methods.

If we're the only ones using the client, we could go back to decorating the bound methods ...except, didn't we say the instance already exists?

So what? It's not like you can only set attributes in __init__():

client = Client()

def cache_method_in_place(self, name, **kwargs):
    one = getattr(self, name)
    cached = functools.lru_cache(**kwargs)(one)
    setattr(self, name, cached)

cache_method_in_place(client, 'one', maxsize=1000)
cache_method_in_place(client, 'two', maxsize=1000)

Unexpected? A bit. Unholy? Maybe, depending where you're coming from. But actually, no, not really – after all, it still walks and quacks like a Client, and we've only changed our instance.

In the end, that's one of the reasons people like Python – you're free to bring in the big guns of metaprogramming if you need to, but sometimes a tiny bit of monkey patching will do just as well.³

Anyway, that's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

I'm mainly dissing static (1:1) wrappers here. As a higher level of abstraction, wrappers can be very useful, both maintainability and readability-wise; for a great example of this, check out Raymond Hettinger's Beyond PEP 8 – Best practices for beautiful intelligible code talk. ^[return]
For example, sqlite3.connect() does, via the factory parameter; reader.make_reader() doesn't (at least, not yet). ^[return]
Just like cats and salami, a very small amount of monkey patching is probably fine, though you definitely shouldn't make it a staple of your diet. ^[return]

reader 3.4 released – 5 years, 2000 commits

2023-01-23T18:00:00+00:00

Hi there!

I'm happy to announce version 3.4 of reader, a Python feed reader library.

More importantly, reader is now 5 years old!

5 years, 2000 commits #

3.3, released in December, marks reader's 5th anniversary and its 2000th commit.

I don't really know what to say, so I'll just say something that maybe will help others.

I'm happy about reader. But more than that, I'm proud of it growing into what it is.

reader started as a hobby and learning project, scratch-your-own-itch kind of thing. Starting off, I had two goals: it should be a library, and it should be for the long term.¹

It continued beyond that, into an experiment.

First, on applying Pieter Hintjens' problem-solution approach, and the ideas in the Write code that is easy to delete, not easy to extend and Repeat yourself, do more than one thing, and rewrite everything essays by tef of programming is terrible.

Second, and more mundane, it was an exercise in how testing and type checking keep code maintainable over time. "Everyone knows" they are good things™, but most jobs rarely emphasize the over time part – I know mine at the time didn't. I wanted a place where I could get an intuitive feel for it.

(If you'd like to read more about any of the above, do let me know.)

Judging the success of either is probably highly subjective. Nevertheless, I think they were both wildly successful. reader is still alive and kicking, despite the limited time I have for it, and I've developed habits and ways of solving problems that I'm constantly using in my professional life.

Sure, reader has flaws, and it's definitely not done (what is?), but it's the longest project I've ever worked on, and probably the best code I've ever written.

Finally, I feel I have to say that while not hugely popular, reader is way more popular than I could've dreamt 5 years ago, especially for a niche, but hopefully still relevant topic like web feeds. Thank you!

What's new? #

Here are the highlights since reader 3.0.

Parser internal API #

The parser internal API is fully documented. It continues to be unstable, but this is a huge first step on the road to internal API stabilization.

I started with the parser because it's probably most useful for plugins (for example, the Twitter plugin uses it extensively), or if you want to retrieve and/or parse feeds with reader, but handle storage and so on yourself (here's a recipe for parsing feeds retrieved with HTTPX).

Faster imports #

The time from process start to usable Reader instance is 3x shorter, by postponing update-related imports until actually needed.

Simpler entry sorting #

When sorting by recent, newly added entries now always appear at the top, regardless of their published date, except on a feed's first update.

This replaces a more complicated heuristic that caused entries added to a feed months after their published (yup, that happens!) to never appear at the top (so you would never see them).

Fewer dependencies #

reader is a library, so I try to keep the number of dependencies as small as possible.

Starting with 3.1, the readtime built-in plugin has no dependencies, and it can be used even if lxml is not available (the library I was using before to calculate read time has a transitive dependency on lxml, which does not always have PyPy Windows wheels).

Python versions #

reader 3.3 adds support for Python 3.11.

reader 3.4 drops support for Python 3.8.

Bug fixes #

There were lots of bug fixes, mainly to plugins (I guess that's a good thing).

For more details, see the full changelog.

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
- and even follow Twitter accounts
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to follow Twitter accounts?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

A bit more on this here. ^[return]

Has your password been pwned? Or, how I almost failed to search a 37 GB text file in under 1 millisecond (in Python)

2022-12-15T09:42:01+00:00

So, there's this website, Have I Been Pwned, where you can check if your email address has appeared in a data breach.

There's also a Pwned Passwords section for passwords ... but, typing your password on a random website probably isn't such a great idea, right?

Of course, you could read about how HIBP protects the privacy of searched passwords, and understand how k-Anonymity works, and check that the website uses the k-Anonymity API, but that's waaay too much work.

Of course, you could use the API yourself, but where's the fun in that?

Instead, we'll do it the hard way – we'll check passwords offline.

And we're not stopping until it's fast.

Contents

The Pwned Passwords list
A minimal plausible solution
Problem: it's slow
- Skipping
Problem: it needs tuning, it's still slow
- Binary skipping
- Better timing
Failing to get to under 1 millisecond
Getting to under 1 millisecond
Bonus: better data structures

The Pwned Passwords list #

OK, first we need the password list.

Go to Pwned Passwords, scroll to Downloading the Pwned Passwords list, and download the SHA-1 ordered-by-hash file – be nice and use the torrent, if you can.

Note

You can also use the downloader to get an updated version of the file, but that didn't exist when I started writing this.

The archive extracts to a 37 GB text file:

$ pushd ~/Downloads
$ stat -f %z pwned-passwords-sha1-ordered-by-hash-v8.7z
16257755606
$ 7zz e pwned-passwords-sha1-ordered-by-hash-v8.7z
$ stat -f %z pwned-passwords-sha1-ordered-by-hash-v8.txt
37342268646
$ popd

... that looks like this:

$ head -n5 ~/Downloads/pwned-passwords-sha1-ordered-by-hash-v8.txt
000000005AD76BD555C1D6D771DE417A4B87E4B4:10
00000000A8DAE4228F821FB418F59826079BF368:4
00000000DD7F2A1C68A35673713783CA390C9E93:873
00000001E225B908BAC31C56DB04D892E47536E0:6
00000006BAB7FC3113AA73DE3589630FC08218E7:3

Each line has the format:

<SHA-1 of the password>:<times it appeared in various breaches>

Note

For more details: why hash the passwords, and why include a count.

To make commands shorter, we link the file in the current directory:

$ ln -s ~/Downloads/pwned-passwords-sha1-ordered-by-hash-v8.txt pwned.txt

A minimal plausible solution #

We'll take an iterative, problem-solution approach to this. But since right now we don't have any solution, we start with the simplest thing that could possibly work.

First, we get the imports out of the way:

import os
import sys
import time
import getpass
import hashlib

Then, we open the file:

path = sys.argv[1]
file = open(path, 'rb')

Opening the file in binary mode makes searching a bit faster, as it skips needless decoding. By opening it early, we get free error checking – no point in reading the password if the file isn't there.

Next, the password:

try:
    password = sys.argv[2]
except IndexError:
    password = getpass.getpass()
hexdigest = hashlib.sha1(password.encode()).hexdigest()
del password

print("looking for", hexdigest)

We either take the password as the second argument to the script, or read it with getpass(), so real passwords don't remain in the shell history. After computing its hash, we delete it; we don't go as far as to zero‍¹ it, but at least we avoid accidental printing.

Let's see if it works:

$ python pwned.py pwned.txt password
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
$ python pwned.py pwned.txt
Password:
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8

sha1sum seems to agree:

$ echo -n password | sha1sum
5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8  -

To find the hash, we go through the file line by line – remember, simplest thing that could possibly work.

def find_line(lines, prefix):
    for line in lines:
        if line.startswith(prefix):
            return line
        if line > prefix:
            break
    return None

If a line was found, we print the count:

line = find_line(file, hexdigest.upper().encode())

if line:
    times = int(line.decode().rstrip().partition(':')[2])
    print(f"pwned! seen {times:,} times before")
else:
    print("not found")

Finally, let's add some timing code:

start = time.monotonic()
line = find_line(file, hexdigest.upper().encode())
end = time.monotonic()

print(f"in {end-start:.6f} seconds")

And, it works:

$ python pwned.py pwned.txt blocking
looking for 000085013a02852372159cb94101b99ccaec59e1
pwned! seen 587 times before
in 0.002070 seconds

The code so far.

Problem: it's slow #

You may have noticed I switched from password to blocking. That's because I deliberately chose a password whose hash is at the beginning of the file.

On my 2013 laptop, password actually takes 86 seconds!

To put a lower bound on the time it takes to go through the file:

$ time python -c 'for line in open("pwned.txt", "rb"): pass'

real	1m28.325s
user	1m10.049s
sys	0m15.577s
$ time wc -l pwned.txt
 847223402 pwned.txt

real	0m51.729s
user	0m27.908s
sys	0m12.119s

There must be a better way.

Skipping #

There's a hint in the file name: the hashes are ordered.

That means we don't have to check all the lines. We can skip ahead repeatedly until we're past the hash, go back one step, and only check each line from there.

Lines are different lengths, so we can't skip exactly X lines without reading them. But we don't need to, any line that's a reasonable amount ahead will do.

def skip_to_before_line(file, prefix, offset):
    old_position = file.tell()

    while True:
        file.seek(offset, os.SEEK_CUR)

        file.readline()
        line = file.readline()
        # print("jumped to", (line or b'<eof>').decode().rstrip())

        if not line or line >= prefix:
            file.seek(old_position)
            break

        old_position = file.tell()

So we just seek() ahead a set number of bytes. Since that might not leave us at the start of a line, we discard the incomplete line, and use the next one.

Finally, we wrap the original find_line() to do the skipping:

def find_line(file, prefix):
    skip_to_before_line(file, prefix, 16 * 2**20)
    return find_line_linear(file, prefix)


def find_line_linear(lines, prefix):
    for line in lines:

It works:

$ python pwned.py pwned.txt password
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
pwned! seen 9,545,824 times before
in 0.027203 seconds

To find the magic 16M offset, I had to try a bunch values:

offset (MiB)   time (s)
           1      0.05
           4      0.035
           8      0.030
          16      0.027  <-- sweet spot
          32      0.14

The code so far.

Problem: it needs tuning, it's still slow #

While three orders of magnitude faster, we still have a bunch of issues:

The ideal offset depends on the computer you're running this on.
The run time still increases linearly with file size – we haven't really solved the problem, as much as made it smaller by a (large, but) constant factor.
The run time still increases linearly with where the hash is in the file.
It's still kinda slow. ¯\_(ツ)_/¯

There must be a better way.

Binary skipping #

To make the "linear" part painfully obvious, uncomment the jumped to line.

$ python pwned.py pwned.txt password | grep -o 'jumped to .' | uniq -c
 139 jumped to 0
 139 jumped to 1
 139 jumped to 2
 139 jumped to 3
 139 jumped to 4
 103 jumped to 5

Surely, after we've jumped to 0 once, we don't need to do it 138 more times, right?

We could jump directly to a line in the middle of the file; if there at all, the hash will be in either of the halves. We could then jump to the middle of that half, and to the middle of that half, and so on, until we either find the hash or there's nowhere left to jump.

Note

If that sounds a lot like binary search, that's because it is – it's just not wearing its regular array clothing.

And most of the work is already done: we can jump to a line at most X bytes from where the hash should be, we only need to do it repeatedly, in smaller and smaller fractions of the file size:

def skip_to_before_line(file, prefix, offset):
    while offset > 2**8:
        offset //= 2
        skip_to_before_line_linear(file, prefix, offset)


def skip_to_before_line_linear(file, prefix, offset):
    old_position = file.tell()

The only thing left is to get the file size:

def find_line(file, prefix):
    file.seek(0, os.SEEK_END)
    size = file.tell()
    file.seek(0)
    skip_to_before_line(file, prefix, size)
    return find_line_linear(file, prefix)

It works:

$ python pwned.py pwned.txt password
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
pwned! seen 9,545,824 times before
in 0.009559 seconds
$ python pwned.py pwned.txt password
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
pwned! seen 9,545,824 times before
in 0.000268 seconds

The huge time difference is due to operating system and/or disk caches – on the second run, the same parts of the file are likely already in memory.

Anyway, look again at the jumped to output: instead of jumping blindly through the whole file, now we're jumping around the hash, getting closer and closer to it.

$ python pwned.py pwned.txt password | grep -o 'jumped to .' | uniq -c
   1 jumped to 7
   1 jumped to 3
   1 jumped to 7
   1 jumped to 5
   1 jumped to 4
  39 jumped to 5

$ python pwned.py pwned.txt password | grep -o 'jumped to ..' | uniq -c
   1 jumped to 7F
   1 jumped to 3F
   1 jumped to 7F
   1 jumped to 5F
   1 jumped to 4F
   1 jumped to 5F
   1 jumped to 57
   1 jumped to 5F
   1 jumped to 5B
   1 jumped to 59
   1 jumped to 5B
   1 jumped to 5A
  32 jumped to 5B

Notice we land on the same 7F... prefix twice; this makes sense – we skip ahead by half the file size, then back, then ahead two times by a quarter. Caching allows us to not care about that.

The code so far.

Better timing #

Given the way caching muddies the waters, how fast is it really?

This function averages a hundred runs, each with a different password:

function average-many {
    for _ in {1..100}; do
        python $@ $( python -c 'import time; print(time.time())' )
    done \
    | grep seconds \
    | cut -d' ' -f2 \
    | paste -sd+ - \
    | sed 's#^#scale=6;(#' | sed 's#$#)/100#' \
    | bc
}

After a few repeats, the average time settles around 3 ms.

$ for _ in {1..20}; do average-many pwned.py pwned.txt; done
.004802
.003904
.004088
.003486
.003451
.003476
.003414
.003442
.003169
.003297
.002931
.003077
.003092
.003011
.002980
.003147
.003112
.002942
.002984
.002934

Again, this is due to caching: the more we run the script, the more likely it is that the pages at {half, quarters, eights, sixteenths, ...} of the file size are already in memory – and the path to any line starts with a subset of those.

And, we're done.

I wave my hands, get a 2020 laptop, and a miracle happens. It's far enough into the totally unpredictable future, now, that you can search any password in under 1 millisecond. You can do anything you want.

So, there we go. Wasn't that an interesting story? That's the end of the article.

Don't look at the scroll bar. Don't worry about it.

If you came here to check your password, you can go.

Subscribe on the way out if you'd like, but take care :)

Go solve Advent of Code or something.

I'm just chilling out.

See ya.

Failing to get to under 1 millisecond #

I swear this was supposed to be the end. This really was supposed to be a short one.

Here's a quote from a friend of mine, that chronologically should be way later into the article, but makes a great summary for what follows:

And now that you have arrived at this point, spend a moment to ponder the arbitrary nature of 1 millisecond given its dependency on the current year and the choice of your particular hardware.

After that moment, continue celebrating.

Nah, fuck it, it has to take less than 1 millisecond on the old laptop.

... so yeah, here's a bunch of stuff that didn't work.

Liking this so far? Here's another article you might like:

Learn by reading code: Python standard library design decisions explained

Profile before optimizing #

With the obvious improvements out of the way, it's probably a good time to stop and find out where time is being spent.

$ python -m cProfile -s cumulative pwned.py pwned.txt "$( date )"
looking for 3960626a8c59fe927d3cf2e991d67f4c505ae198
not found
in 0.004902 seconds
         1631 function calls (1614 primitive calls) in 0.010 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      ...
        1    0.000    0.000    0.005    0.005 02-binary-search.py:8(find_line)
        1    0.000    0.000    0.005    0.005 02-binary-search.py:22(skip_to_before_line)
       28    0.000    0.000    0.005    0.000 02-binary-search.py:28(skip_to_before_line_linear)
       86    0.004    0.000    0.004    0.000 {method 'readline' of '_io.BufferedReader' objects}
      ...
       71    0.000    0.000    0.000    0.000 {method 'seek' of '_io.BufferedReader' objects}
      ...
       44    0.000    0.000    0.000    0.000 {method 'tell' of '_io.BufferedReader' objects}
      ...

From the output above, we learn that:

Most of the time is spent in readline() calls.
Both seek() and tell() calls are basically free.

readline() is implemented in C, so there's not much we can change there.

What we can change, however, is how often we call it.

Another thing of interest is how much individual readline() calls take.

In skip_to_before_line_linear():

        start = time.monotonic()
        file.readline()
        line = file.readline()
        end = time.monotonic()
        print("jumped to", (line[:5] or b'<eof>').decode().rstrip(),
              f"in {(end-start)*1000000:4.0f} us",
              f"at offset {file.tell():16,}")

The output is pretty enlightening:

$ python pwned.py pwned.txt asdf
looking for 3da541559918a808c2402bba5012f6c60b27661c
jumped to 7FF9E in   10 us at offset   18,671,134,394  <-- 1/2 file size
jumped to 3FF89 in    4 us at offset    9,335,567,234  <-- 1/4 file size
jumped to 1FFBA in    3 us at offset    4,667,783,663  <-- 1/8 file size
jumped to 3FF89 in    3 us at offset    9,335,567,322  <-- 1/4 file size
jumped to 2FFA4 in    5 us at offset    7,001,675,508
jumped to 3FF89 in    4 us at offset    9,335,567,366  <-- 1/4 file size
jumped to 37F98 in    4 us at offset    8,168,621,453
jumped to 3FF89 in    3 us at offset    9,335,567,410  <-- 1/4 file size
jumped to 3BF94 in    3 us at offset    8,752,094,477
jumped to 3FF89 in    2 us at offset    9,335,567,498  <-- 1/4 file size
jumped to 3DF8E in    3 us at offset    9,043,831,007
jumped to 3CF90 in    3 us at offset    8,897,962,782
jumped to 3DF8E in    2 us at offset    9,043,831,095
jumped to 3D790 in    3 us at offset    8,970,896,964
jumped to 3DF8E in    2 us at offset    9,043,831,139
jumped to 3DB90 in  253 us at offset    9,007,364,072
jumped to 3D990 in  206 us at offset    8,989,130,552
jumped to 3DB90 in    6 us at offset    9,007,364,160
jumped to 3DA8F in  270 us at offset    8,998,247,402  <-- page 2,196,837
jumped to 3DA0F in  189 us at offset    8,993,689,007
jumped to 3DA8F in    5 us at offset    8,998,247,446  <-- page 2,196,837
jumped to 3DA4F in  212 us at offset    8,995,968,274
jumped to 3DA8F in    5 us at offset    8,998,247,534  <-- page 2,196,837
jumped to 3DA6F in  266 us at offset    8,997,107,921
jumped to 3DA5F in  203 us at offset    8,996,538,139
jumped to 3DA57 in  195 us at offset    8,996,253,241
jumped to 3DA53 in  197 us at offset    8,996,110,772
jumped to 3DA57 in    6 us at offset    8,996,253,285
jumped to 3DA55 in  193 us at offset    8,996,182,045
jumped to 3DA54 in  178 us at offset    8,996,146,471
jumped to 3DA54 in  189 us at offset    8,996,128,666
jumped to 3DA54 in  191 us at offset    8,996,119,760  <-- page 2,196,318
jumped to 3DA54 in   32 us at offset    8,996,128,710
jumped to 3DA54 in    5 us at offset    8,996,124,259
jumped to 3DA54 in   10 us at offset    8,996,122,057  <-- page 2,196,318
jumped to 3DA54 in    4 us at offset    8,996,120,955  <-- page 2,196,318
jumped to 3DA54 in    4 us at offset    8,996,120,382  <-- page 2,196,318
jumped to 3DA54 in    9 us at offset    8,996,120,112  <-- page 2,196,318
jumped to 3DA54 in    1 us at offset    8,996,120,470  <-- page 2,196,318
jumped to 3DA54 in    1 us at offset    8,996,120,338  <-- page 2,196,318
pwned! seen 324,774 times before
in 0.003654 seconds

Half the reads are pretty fast:

In the beginning, because searches start with the same few pages.
At the end, because searches end on the same page.
All reads of any page, after the first.

So, it's those reads in the middle that we need to get rid of.

Position heuristic #

In theory, the output of a good hash function should be uniformly distributed.

This means that with a bit of math, we can estimate where a hash would be – a hash that's ~1/5 in the range of all possible hashes should be at ~1/5 of the file.

Here's a tiny example:

>>> digest = '5b'  # 1-byte hash (2 hex digits)
>>> size = 1000    # 1000-byte long file
>>> int_digest = int(digest, 16)  # == 91
>>> int_end = 16 ** len(digest)   # == 0xff + 1 == 256
>>> int(size * int_digest / int_end)
355

We can do this once, and then binary search a safety interval around that position. Alas, this only gets rid of the fast jumps at the beginning of the binary search, and for some reason, it ends up being slightly slower than binary search alone. (code)

We can also narrow down around the estimated position iteratively, making the interval smaller by a constant factor each time. This seems to work: a factor of 1000 yields 1.7 ms, and a factor of 8000 yields 1.2 ms, both in 2 steps. (code)

However, it has deeper issues:

Having arbitrary start/end offsets complicates the code quite a bit.
I don't know how to reliably determine the factor.²
I don't know how to prove it's correct,³ especially for smaller intervals, where the hashes are less uniform. To be honest, I don't think it can be 100% correct, and I don't know how to estimate how correct it is.

Anyway, if the implementation is hard to explain, it's a bad idea.

Index file #

An arbitrary self-imposed restriction I had was that any solution should mostly use the original passwords list, with little to no preparation.

By relaxing this a bit, and going through the file once, we can build an index like:

<SHA-1 of the password>:<offset in file>

... that we can then search with skip_to_before_line().

Of course, we won't include all the hashes – by including lines a few kilobytes apart from each other, we can seek directly to within a few kilobytes in the big file.

The only thing left to figure out is how much "a few kilobytes" is.

After my endless harping about caching and pages, the answer should be obvious: one page size (4K). And this actually gets us 0.8 ms! But back when I wrote the code, that hadn't really sunk in, so after getting 1.2 ms with a 32K distance, I moved on.

Code: pwned.py, generate-index.py.

Binary file #

Already on the additional file slippery slope, I converted the list to binary, mostly to make it smaller – smaller file, fewer reads.

I packed each line into 24 bytes:

| binary hash (20 bytes) | count (4 bytes) |

This halved the file, but only lowered the runtime to a modest 2.6 ms.

More importantly, it made the code much, much simpler: because items are fixed size, you can know where the Nth item is, so I was able to use bisect for the binary search.

Code: pwned.py, convert-to-binary.py.

Getting to under 1 millisecond #

OK, what now? This is what we have so far:

The position heuristic kinda (maybe?) works, but is hard to reason about.
The index file gets us there, but barely, and the index is pretty big.
The binary file isn't much faster, and it creates a huge file. But, less code!

I don't know what to do with the first one, but I think we can combine the last two.

Generating the index #

Let's start with the script I made for the text index:

import os, sys

file = sys.stdin.buffer
outf = sys.stdout.buffer

while True:
    pos = file.tell()
    line = file.readline()
    if not line:
        break

    outf.write(line.partition(b':')[0] + b':' + str(pos).encode() + b'\n')

    file.seek(2**12, os.SEEK_CUR)
    file.readline()

The output looks like this:

$ python generate-index.py < pwned.txt 2>/dev/null | head -n5
000000005AD76BD555C1D6D771DE417A4B87E4B4:0
00000099A4D3034E14DF60EF50799F695C27C0EC:4157
00000172E8E1D1BD54AC23B3F9AB4383F291CA17:8312
000002C8F808A7DB504BBC3C711BE8A8D508C0F9:12453
0000047139578F13D70DD96BADD425C372DB64A9:16637

We need to pack that into bytes.

A hash takes 20 bytes. But, we only need slightly more than 3 bytes (6 hex digits) to distinguish between the index lines:

$ python generate-index.py < pwned.txt 2>/dev/null | cut -c-6 | uniq -c | head
   2 000000
   1 000001
   1 000002
   1 000004
   1 000005
   1 000007
   1 000008
   1 00000A
   1 00000B
   1 00000C

To represent all the offsets in the file, we need log2(35G) / 8 = 4.39... bytes, which results in a total of 9 bytes (maybe even 8, if we mess with individual bits).

Let's make it future-proof: 6 bytes for the hash buys at least 55 trillion lines (2.4 petabyte files), and 6 bytes for the offset buys 0.28 petabyte files.

    digest, _, _ = line.partition(b':')
    outf.write(bytes.fromhex(digest.decode())[:6])
    outf.write(pos.to_bytes(6, 'big'))

If you look at the text index, you'll notice the offsets are 4K + ~50 bytes apart; this results in sometimes having to read 2 pages, because not all pages have an index entry. Let's fix that by reading the first whole line after a 4K boundary instead:

    file.seek((pos // 4096 + 1) * 4096)
    file.readline()

OK, we're done:

$ time python generate-index.py < pwned.txt > index.bin

real	1m2.729s
user	0m34.292s
sys	0m21.392s

Using the index #

We start with a skeleton that's functionally identical to the naive script. The only difference is that I've added stubs for passing and using the index:

def find_line(file, prefix, index):
    skip_to_before_line_index(file, prefix, index)
    return find_line_linear(file, prefix)

def skip_to_before_line_index(file, prefix, index):
    ...

index_path = sys.argv[2]
index = ...

line = find_line(file, hexdigest.upper().encode(), index)

The whole skeleton, if you want to see it:

import os
import sys
import time
import bisect
import getpass
import hashlib


def find_line(file, prefix, index):
    skip_to_before_line_index(file, prefix, index)
    return find_line_linear(file, prefix)


def find_line_linear(lines, prefix):
    for line in lines:
        if line.startswith(prefix):
            return line
        if line > prefix:
            break
    return None


def skip_to_before_line_index(file, prefix, index):
    ...


path = sys.argv[1]
file = open(path, 'rb')

index_path = sys.argv[2]
index = ...

try:
    password = sys.argv[3]
except IndexError:
    password = getpass.getpass()
hexdigest = hashlib.sha1(password.encode()).hexdigest()
del password

print("looking for", hexdigest)

start = time.monotonic()
line = find_line(file, hexdigest.upper().encode(), index)
end = time.monotonic()

if line:
    times = int(line.decode().rstrip().partition(':')[2])
    print(f"pwned! seen {times:,} times before")
else:
    print("not found")

print(f"in {end-start:.6f} seconds")

As mentioned before, we'll use the standard library bisect module to search the index.

We could read the entire index in memory, as a list of 12-byte bytes. But that would still be slow, even if outside the current timing code, and memory usage would increase with the size of the file.

Fortunately, bisect doesn't only work with lists, it works with any sequence – that is, any object that can pretend to be a list. So we'll build our own, by implementing the two required⁴ special methods.

class BytesArray:

    item_size = 12

    def __init__(self, file):
        self.file = file

We can go ahead and plug it in:

index_path = sys.argv[2]
index = BytesArray(open(index_path, 'rb'))

The first special method is __getitem__(), for a[i]:

    def __getitem__(self, index):
        self.file.seek(index * self.item_size)
        buffer = self.file.read(self.item_size)
        if len(buffer) != self.item_size:
            raise IndexError(index)  # out of bounds
        return buffer

The second special method is __len__(), for len(a):

    def __len__(self):
        self.file.seek(0, os.SEEK_END)
        size = self.file.tell()
        return size // self.item_size

Using the index becomes straightforward:

def skip_to_before_line_index(file, prefix, index):
    item_prefix = bytes.fromhex(prefix.decode())[:6]
    item = find_lt(index, item_prefix)
    offset = int.from_bytes(item[6:], 'big') if item else 0
    file.seek(offset)


def find_lt(a, x):
    i = bisect.bisect_left(a, x)
    if i:
        return a[i-1]
    return None

We get the first 6 bytes of the hash, find the rightmost value less than that, extract the offset from it, and seek to there. find_lt() comes from bisect's recipes for searching sorted lists.

And we're done:

$ average-many pwned.py pwned.txt index.bin
.002546

Huh? ... that's unexpected...

Oh.

I said we won't read the index in memory. But we can force it into the cache by reading it a bunch of times:

$ for _ in {1..10}; do cat index.bin > /dev/null; done

Finally:

$ average-many pwned.py pwned.txt index.bin
.000421

Code: pwned.py, generate-index.py.

I heard you like indexes (the end) #

Hmmm... isn't that cold start bugging you? If we make an index for the big index, we get 1.2 ms from a cold start. Maybe another smaller index can take us to below 1 ms?

...

Just kidding, this is it, this really is the end.

And now, let's take that moment to ponder:

method	statements	time (ms, order of magnitude)
linear	29	100,000
linear+skip	42	100
binary search	49	10
binary index	59 (72)	1

For twice the code, it's 5 orders of magnitude faster! I'm deliberately not counting bisect or the OS cache here, beacuse that's the point, they're basically free.

Turns out, you can get pretty far with just a few tricks.

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

If you've made it this far, you might like:

Write an SQL query builder in 150 lines of Python!

Bonus: better data structures #

As always, a specialized data structure can solve a problem better.

In Sketchy Pwned Passwords, Scott Helme manages to "pack" the entire passwords list into a 1.5G in-memory count-min sketch, which can then be queried in 1 microsecond. And, if you don't care about the counts, a plain Bloom filter can even take you to 0.1 µs! (There is a trade-off, and it's that it takes 11 hours to create either.)

It's kinda difficult to do in Python anyway. ^[return]
Something like (size / 4096) ** (1 / int(log(size, 4096))), maybe? ^[return]
I mean, I did cross-check it with other solutions for a few thousand values, and it seems correct, but that's not proof. ^[return]
We're only implementing the parts of the protocol that bisect uses.

For the full protocol, __getitem__() would need to implement negative indexes and slicing. For more, free sequence methods, inherit collections.abc.Sequence.

Interestingly, the class will work in a for loop without an __iter__() method. That's because there are actually two iteration protocols: an older one, using __getitem__(), and a newer one, added in Python 2.1, using __iter__() and __next__(). For details on the logic, see Unravelling for statements. ^[return]

reader 3.0 released – multithreading

2022-09-16T20:00:00+00:00

Hi there!

I'm happy to announce version 3.0 of reader, a Python feed reader library.

This release removes a number of deprecated methods and attributes, for a cleaner, more consistent API. See the changelog for details.

Contents

2.x recap
What is reader?
Why use a feed reader library?
Why make your own feed reader?

2.x recap #

2.0 was released over a year ago; let's have look at what happened since.

Unified tag API + entry and global tags #

Tags and metadata are now the same thing, generic resource tags:

>>> reader.get_tag(feed, 'one', 'default')
'default'
>>> reader.set_tag(feed, 'one', 'value')
>>> reader.get_tag(feed, 'one')
'value'
>>> reader.set_tag(feed, 'two')
>>> dict(reader.get_tags(feed))
{'one': 'value', 'two': None}

This means you can filter by metadata keys, and attach values to tags.

Even better, tags aren't just for feeds anymore – you can add tags to entries, and to a global namespace.

Search enabled by default #

Full-text search works out of the box: no extra dependencies, no setup needed.

Statistics #

There are now statistics on feed and user activity, to give you a better understanding of how you consume content.

First, you can get the average number of entries per day for the last 1, 3, 12 months, so you know how often a feed publishes new entries, and how that changed over time – think sparklines: 36 entries ▄▃▁ (4.0, 2.0, 0.6).

Second, reader records the time when an entry was last marked as read or important. This will allow you to see how you engage with new entries – I'm still working on how to translate this data into a useful summary.

A nice side-effect of knowing when entry flags changed is that it's possible to tell if an entry was explicitly marked as unimportant (entries are unimportant by default).

User-added entries #

You can now add entries to existing feeds. This is useful when you want to keep track of an article that is not in the feed anymore because it "fell off the end".

It can also be used to build bookmarking / read later functionality similar to that of Tiny Tiny RSS; extracting content from arbitrary pages would be pretty helpful here.

Twitter support #

You can now follow Twitter accounts (experimental, requires a Twitter account).

Read time #

The new readtime plugin calculates the entry read time during feed updates.

This makes available to any reader user a feature that was only available in the web app, and makes the web app faster.

Improved duplicate handling #

Duplicate handling got significantly better:

False negatives are reduced by using approximate string matching and heuristics to detect truncated content.
You can trigger entry deduplication manually, for the existing entries of a feed – just add the .reader.dedupe.once tag to the feed, and wait for the next update. Also, you can deduplicate entries by title alone, ignoring content.
Old duplicates are deleted instead of marked as read/unimportant.

Memory usage improvements #

reader update uses about 22% less memory.

The main change is not to reader itself, but was contributed upstream to feedparser: instead of reading the whole feed in memory to detect encoding, use a prefix of the feed, and decode the rest on the fly. The result is a 35% decrease in update_feeds() maximum RSS when compared to baseline!

You can find more details here.

Multithreading #

You can now use the same Reader object from multiple threads. So, you can do stuff like this:

>>> Thread(target=reader.update_feeds).start()

Also, you can reuse Reader objects after closing.

Typing #

reader has had type annotations for most of its existence; starting with 2.14, user code can use them too.

New Python versions #

Over the course of 2.x, reader got support for Python 3.10, and PyPy 3.8 and 3.9, and dropped support for Python 3.7.

Other changes #

Aside from the changes mentioned above, a lot of convenience methods, arguments, and attributes were added. Among the more notable ones, now you can:

filter feeds in the same way both when getting and when updating feeds – including by tags
run arbitrary actions before and after updating feeds

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
- and even follow Twitter accounts
mark articles as read or important
add arbitrary tags/metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?
want to follow Twitter accounts?
want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

reader 2.14 released – Twitter, read time

2022-07-01T18:18:18.180000+00:00

Hi there!

I'm happy to announce version 2.14 of reader, a Python feed reader library.

What's new? #

Here are the most important changes since reader 2.11!

Twitter support #

You can now use reader to follow Twitter accounts.

You still need a Twitter account to get a bearer token, but after that, https://twitter.com/user works like any other feed.

Each thread corresponds to an entry. Threads are rendered as HTML, but you can access the original JSON too, in case you want to do your own rendering.

Here's what it looks like in the web app:

thread with quote

thread with media

Read time #

The new readtime plugin calculates the read time of an entry during feed update.

This makes available to any reader user a feature that was only available in the web app, and makes the web app faster.

Typing #

You can now type check code that uses reader.

reader has had type annotations for most of its existence, but user code would fail type checking because reader did not explicitly declare it.

Python versions #

reader 2.14 drops support for Python 3.7, and adds support for PyPy 3.9.

Bug fixes #

reader now skips RSS entries that have no <guid> or <link> with a warning, instead of failing the entire feed. Thanks to Mirek Długosz for the pull request.

For more details, see the full changelog.

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

The unreasonable effectiveness of f‍-‍strings and re.VERBOSE

2022-06-20T15:02:01+00:00

... in which we look at one or two ways to make life easier when working with Python regular expressions.

tl;dr: You can compose verbose regular expressions using f‍-‍strings.

Here's a real-world example – instead of this:

pattern = r"((?:\(\s*)?[A-Z]*H\d+[a-z]*(?:\s*\+\s*[A-Z]*H\d+[a-z]*)*(?:\s*[\):+])?)(.*?)(?=(?:\(\s*)?[A-Z]*H\d+[a-z]*(?:\s*\+\s*[A-Z]*H\d+[a-z]*)*(?:\s*[\):+])?(?![^\w\s])|$)"

... do this:

code = r"""
[A-Z]*H  # prefix
\d+      # digits
[a-z]*   # suffix
"""

multicode = fr"""
(?: \( \s* )?               # maybe open paren and maybe space
{code}                      # one code
(?: \s* \+ \s* {code} )*    # maybe followed by other codes, plus-separated
(?: \s* [\):+] )?           # maybe space and maybe close paren or colon or plus
"""

pattern = fr"""
( {multicode} )             # code (capture)
( .*? )                     # message (capture): everything ...
(?=                         # ... up to (but excluding) ...
    {multicode}             # ... the next code
        (?! [^\w\s] )       # (but not when followed by punctuation)
    | $                     # ... or the end
)
"""

For comparison, the same pattern without f‍-‍strings (click to expand).

pattern = r"""
(                       # code (capture)
    # BEGIN multicode

    (?: \( \s* )?       # maybe open paren and maybe space

    # code
    [A-Z]*H  # prefix
    \d+      # digits
    [a-z]*   # suffix

    (?:                 # maybe followed by other codes,
        \s* \+ \s*      # ... plus-separated

        # code
        [A-Z]*H  # prefix
        \d+      # digits
        [a-z]*   # suffix
    )*

    (?: \s* [\):+] )?   # maybe space and maybe close paren or colon or plus

    # END multicode
)

( .*? )                 # message (capture): everything ...

(?=                     # ... up to (but excluding) ...
    # ... the next code

    # BEGIN multicode

    (?: \( \s* )?       # maybe open paren and maybe space

    # code
    [A-Z]*H  # prefix
    \d+      # digits
    [a-z]*   # suffix

    (?:                 # maybe followed by other codes,
        \s* \+ \s*      # ... plus-separated

        # code
        [A-Z]*H  # prefix
        \d+      # digits
        [a-z]*   # suffix
    )*

    (?: \s* [\):+] )?   # maybe space and maybe close paren or colon or plus

    # END multicode

        # (but not when followed by punctuation)
        (?! [^\w\s] )

    # ... or the end
    | $
)

"""

It's better than the non-verbose one, but even with careful formatting and comments, the repetition makes it pretty hard to follow – and wait until you have to change something!

Read on for details and some caveats.

Prerequisites #

Formatted string literals (f‍-‍strings) were added in Python 3.6¹, and provide a way to embed expressions inside string literals, using a syntax similar to that of str.format():

>>> name = "world"
>>>
>>> "Hello, {name}!".format(name=name)
'Hello, world!'
>>>
>>> f"Hello, {name}!"
'Hello, world!'

Verbose regular expressions (re.VERBOSE) have been around since forever², and allow writing regular expressions with non-significant whitespace and comments:

>>> text = "H1 code (AH2b+EUH3) fancy code"
>>>
>>> code = r"[A-Z]*H\d+[a-z]*"
>>> re.findall(code, text)
['H1', 'AH2b', 'EUH3']
>>>
>>> code = r"""
... [A-Z]*H  # prefix
... \d+      # digits
... [a-z]*   # suffix
... """
>>> re.findall(code, text, re.VERBOSE)
['H1', 'AH2b', 'EUH3']

The "one weird trick" #

Once you see it, it's obvious – you can use f‍-‍strings to compose regular expressions:

>>> multicode = fr"""
... (?: \( )?         # maybe open paren
... {code}            # one code
... (?: \+ {code} )*  # maybe other codes, plus-separated
... (?: \) )?         # maybe close paren
... """
>>> re.findall(multicode, text, re.VERBOSE)
['H1', '(AH2b+EUH3)']

It's so obvious, it only took me three years to do it after I started using Python 3.6+, despite using both features during all that time.

Of course, there's any number of libraries for building regular expressions; the benefit of this is that it has zero dependencies, and zero extra things you need to learn.

Liking this so far? Here's another article you might like:

Learn by reading code: Python standard library design decisions explained

Caveats #

Hashes and spaces need to be escaped #

Because a hash is used to mark the start of a comment, and spaces are mostly ignored, you have to represent them in some other way.

The documentation of re.VERBOSE is quite helpful:

When a line contains a # that is not in a character class and is not preceded by an unescaped backslash, all characters from the leftmost such # through the end of the line are ignored.

That is, this won't work as the non-verbose version:

>>> re.findall("\d+#\d+", "1#23a")
['1#23']
>>> re.findall("\d+ # \d+", "1#23a", re.VERBOSE)
['1', '23']

... but these will:

>>> re.findall("\d+ [#] \d+", "1#23a", re.VERBOSE)
['1#23']
>>> re.findall("\d+ \# \d+", "1#23a", re.VERBOSE)
['1#23']

The same is true for spaces:

>>> re.findall("\d+ [ ] \d+", "1 23a", re.VERBOSE)
['1 23']
>>> re.findall("\d+ \  \d+", "1 23a", re.VERBOSE)
['1 23']

Hashes need extra care #

When composing regexes, ending a pattern on the same line as a comment might accidentally comment the following line in the enclosing pattern:

>>> one = "1 # comment"
>>> onetwo = f"{one} 2"
>>> re.findall(onetwo, '0123', re.VERBOSE)
['1']
>>> print(onetwo)
1 # comment 2

This can be avoided by always ending the pattern on a new line:

>>> one = """\
... 1 # comment
... """
>>> onetwo = f"""\
... {one} 2
... """
>>> re.findall(onetwo, '0123', re.VERBOSE)
['12']

While a bit cumbersome, in real life most patterns would span multiple lines anyway, so it's not really an issue.

(Note that this is only needed if you use comments.)

Brace quantifiers need to be escaped #

Because f‍-‍strings already use braces for replacements, to represent brace quantifiers you must double the braces:

>>> re.findall("m{2}", "entire mm but only two of mmm")
['mm', 'mm']
>>> letter = "m"
>>> pattern = f"{letter}{{2}}"
>>> re.findall(pattern, "entire mm but only two of mmm")
['mm', 'mm']

I don't control the flags #

Maybe you'd like to use verbose regexes, but don't control the flags passed to the re functions (for example, because you're passing the regex to an API).

Worry not! The regular expression syntax supports inline flags:

(?aiLmsux)

(One or more letters [...]) The group matches the empty string; the letters set the corresponding flags: [...] re.X (verbose), for the entire regular expression. [...] This is useful if you wish to include the flags as part of the regular expression, instead of passing a flag argument to the re.compile() function. Flags should be used first in the expression string.

(?aiLmsux-imsx:...)

[...] The letters set or remove the corresponding flags [...] for the part of the expression. [...]

So, you can do this:

>>> onetwo = """\
... (?x)
... 1 # look, ma
... 2 # no flags
... """
>>> re.findall(onetwo, '0123')
['12']

... or this:

>>> onetwo = """\
... (?x:
...     1 # verbose until the close paren
... )2"""
>>> re.findall(onetwo, '0123')
['12']

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Bonus: I don't use Python #

Lots of other languages support the inline verbose flag, too! You can build a pattern in whichever language is more convenient, and use it in any other one.³ Languages like...

C (with PCRE – and by extension, C++, PHP, and many others):

echo '0123' | pcregrep -o '(?x)
1 2  # such inline
'

... yeah, the C version is actually really long, click to expand.

char *pattern =
    "(?x)\n"
    "1 2  # much verbose\n"
;
char *subject = "0123";
int subject_length = strlen(subject);

int errornumber;
PCRE2_SIZE erroroffset;

pcre2_code *re = pcre2_compile(
    (PCRE2_SPTR)pattern,
    PCRE2_ZERO_TERMINATED,
    0,
    &errornumber,
    &erroroffset,
    NULL
);

pcre2_match_data *match_data = pcre2_match_data_create_from_pattern(re, NULL);

pcre2_match(
    re,
    (PCRE2_SPTR)subject,
    subject_length,
    0,
    0,
    match_data,
    NULL
);

PCRE2_SIZE *ovector = pcre2_get_ovector_pointer(match_data);
PCRE2_SPTR substring_start = (PCRE2_SPTR)subject + ovector[0];
size_t substring_length = ovector[1] - ovector[0];
printf("%.*s\n", (int)substring_length, (char *)substring_start);

C#:

Console.WriteLine(new Regex(@"(?x)
1 2  # wow
").Match("0123"));

grep (only the GNU one):

echo '0123' | grep -Po '(?x) 1 2  # no line'

Java (and by extension, lots of JVM languages, like Scala):

var p = Pattern.compile(
    "(?x)\n" +
    "1 2  # much class\n"
);
var m = p.matcher("0123");
m.find();
System.out.println(m.group(0));

Perl:

"0123" =~ /(?x)(
1 2  # no scare
)/;
print $1 . "\n";

PostgreSQL:

select substring(
  '0123' from
    $$(?x)
    1 2  # such declarative
    $$
);

Ruby:

puts /(?x)
1 2  # nice
/.match('0123')

Rust:

let re = Regex::new(
    r"(?x)
    1 2  # much safe
    "
).unwrap();
println!("{}", re.find("0123").unwrap().as_str());

Swift:

let string = "0123"
let range = string.range(
    of : """
    (?x)
    1 2  # omg hi
    """,
    options : .regularExpression
)
print(string[range!])

Notable languages that don't support inline verbose flags out of the box:

C (regex.h – POSIX regular expressions)
C++ (regex)
Go (regexp)
Javascript
Lua

If you've made it this far, you might like:

Struggling to structure code in larger programs? Great resources a beginner might not find so easily

Bonus: DEFINE #

(update) Interestingly, Perl, PCRE, and the regex Python library all support reusing subpatterns without string interpolation.⁴

To define a named subpattern, use the DEFINE pseudo-condition:

(?(DEFINE)
    (?<name> subpattern )
    ...
)

To use it, do a "subroutine" call:

(?&subpattern)

The example at the beginning of the article would then look like this:

(?(DEFINE)
    (?<code>
        [A-Z]*H  # prefix
        \d+      # digits
        [a-z]*   # suffix
    )
    (?<multicode>
        (?: \( \s* )?               # maybe open paren and maybe space
        (?&code)                    # one code
        (?: \s* \+ \s* (?&code) )*  # maybe followed by other codes, plus-separated
        (?: \s* [\):+] )?           # maybe space and maybe close paren or colon or plus
    )
)

( (?&multicode) )           # code (capture)
( .*? )                     # message (capture): everything ...
(?=                         # ... up to (but excluding) ...
    (?&multicode)           # ... the next code
        (?! [^\w\s] )       # (but not when followed by punctuation)
    | $                     # ... or the end
)

In PEP 498 – Literal String Interpolation. ^[return]
That is, since at least Python 1.5.2, released in 1998 – for all except a tiny minority of Python users, that's before forever. ^[return]
If they both support inline flags, they likely share most other features. ^[return]
Thanks to webstrand and asicsp for pointing it out! ^[return]

reader 2.11 released – metadata is tags

2022-03-21T18:28:11+00:00

Hi there!

I'm happy to announce version 2.11 of reader, a Python feed reader library.

What's new? #

Quite a lot happened since reader 2.5!

Unified tag API + entry and global tags #

Tags and metadata are now the same thing, generic resource tags:

>>> reader.get_tag(feed, 'one', 'default')
'default'
>>> reader.set_tag(feed, 'one', 'value')
>>> reader.get_tag(feed, 'one')
'value'
>>> reader.set_tag(feed, 'two')
>>> dict(reader.get_tags(feed))
{'one': 'value', 'two': None}

This means you can filter by metadata keys, and attach values to tags.

Even better, tags aren't just for feeds¹ anymore – you can add tags to entries, and to a global namespace.

Memory usage improvements #

reader update uses about 22% less memory, owed to two changes.

The first one is not in reader itself, but was contributed to feedparser: instead of reading the whole feed in memory to detect encoding, use a prefix of the feed, and decode the rest on the fly.²

The result is a ~20% decrease in update_feeds() maximum resident set size (35%, when compared to baseline!).

reader will vendor the patched feedparser until the change is released upstream, so you can reap the benefits now.

The second one is parsing feeds serially, using workers only to retrieve them. Since parsing time is mostly spent in pure Python code, there's no speed-up from doing it in parallel – but each thread takes up extra memory.

This decreased update_feeds() memory usage by another ~20% when using more than one worker (but only on Linux; on macOS it's less notable).

Bug fixes #

The way reader checked SQLite has JSON support was somewhat brittle, causing it to fail on SQLite 3.38; reader 2.11 fixes this.

Usability improvements #

Among a number of smaller API improvements, now you can:

filter feeds in the same way both when getting and when updating feeds – including by tags
run arbitrary actions before updating a feed
add an existing feed without getting an exception
delete a missing feed or entry without getting an exception

For more details, see the full changelog.

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark articles as read or important
add arbitrary metadata to feeds and articles
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

The old feed-specific tag and metadata methods are still available, but are deprecated and will be removed in version 3.0. ^[return]
If you'd like to read more about the whole thing, drop me a line. ^[return]

yaml: could not determine a constructor for the tag

2022-02-22T22:02:22+00:00

So you're trying to read some YAML using PyYAML, and get an exception like this:

>>> yaml.safe_load("!!python/tuple [0,0]")
Traceback (most recent call last):
  ...
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/tuple'
  in "<unicode string>", line 1, column 1:
    !!python/tuple [0,0]
    ^

... or like this:

>>> yaml.safe_load("!GetAZs us-east-1")
Traceback (most recent call last):
  ...
yaml.constructor.ConstructorError: could not determine a constructor for the tag '!GetAZs'
  in "<unicode string>", line 1, column 1:
    !GetAZs us-east-1
    ^

What does it mean? #

First, a bit of background.

On top of basic types (strings, integers, sequences, and so on), YAML can represent native and user-defined data structures. To denote the type of a node, you mark it with an explicit tag. Even basic types end up with a tag; the following are all equivalent:

>>> yaml.safe_load("[implicit]")
['implicit']
>>> yaml.safe_load("!!seq [global, shorthand]")
['global', 'shorthand']
>>> yaml.safe_load("!<tag:yaml.org,2002:seq> [global, full]")
['global', 'full']

The errors above both mean that the loader encountered an explicit tag, but doesn't know how to construct objects with that tag.

Why does this happen? #

!!python/tuple is a language-specific tag corresponding to a Python native data structure (a tuple). However, safe_load() resolves only basic YAML tags, known to be safe for untrusted input.

!GetAZs is an application-specific tag (in this case, specific to AWS CloudFormation). There's no way for PyYAML to know about it without being told explicitly.

This is by design – from the spec:

That said, tag resolution is specific to the application. YAML processors should therefore provide a mechanism allowing the application to override and expand these default tag resolution rules.

What now? #

Python-specific tags #

For Python-specific tags, you can use full_load(), which resolves all tags except those known to be unsafe; this includes all the tags listed here.

You could also use unsafe_load(), but most of the time it's not what you want:

Warning

yaml.unsafe_load() is unsafe for untrusted data, because it allows running arbitrary code. Consider using safe_load() or full_load() instead.

For example, you can do this:

>>> yaml.unsafe_load("!!python/object/new:os.system [echo WOOSH. YOU HAVE been compromised]")
WOOSH. YOU HAVE been compromised
0

There were a bunch of CVEs about it.

Application-specific tags #

For application-specific tags, you can define a constructor for the tag:

@dataclass
class GetAZs:
    region: str

class Loader(yaml.SafeLoader):
    pass

def construct_GetAZs(loader, node):
    return GetAZs(loader.construct_scalar(node))

Loader.add_constructor('!GetAZs', construct_GetAZs)

Here, we're wrapping the value in a dataclass, to indicate is isn't just a simple string.

Note

We are subclassing SafeLoader because calling add_constructor() on it would modify it in-place, for everyone, which isn't necessarily great; imagine getting a GetAZs from safe_load(), when you were expecting only built-in types.

To use it, pass the loader class to load():

>>> yaml.load("!GetAZs us-east-1", Loader=Loader)
GetAZs(region='us-east-1')

Of course, you don't have to store the value, you can do something with it – after all, that's what CloudFormation does:

KNOWN_AZS = {
    'us-east-1': ['us-east-1a', 'us-east-1b', 'us-east-1c', 'us-east-1d', 'us-east-1e'],
    'eu-west-1': ['eu-west-1a', 'eu-west-1b', 'eu-west-1c'],
}

def construct_GetAZs(loader, node):
    value = loader.construct_scalar(node)
    if value not in KNOWN_AZS:
        raise yaml.constructor.ConstructorError(
            None, None, f"GetAZs got unknown region {value!r}", node.start_mark
        )
    return KNOWN_AZS[value]

Loader.add_constructor('!GetAZs', construct_GetAZs)

>>> yaml.load("!GetAZs us-east-1", Loader=Loader)
['us-east-1a', 'us-east-1b', 'us-east-1c', 'us-east-1d', 'us-east-1e']

But I don't know the tags in advance #

For the above to work, you need to register constructors for each expected tag.

But sometimes you don't know the tags in advance, or there's too many of them, or you just want to access the data, without caring what it means (for example, because you just want to change a little thing and write it back out).

YAML allows you to register a catch-all constructor for unknown tags ...but you still need to implement some sort of generic wrapper to go with it.

Luckily, I've already written a whole article on how to do that, complete with code:

>>> yaml.load("!GetAZs us-east-1", Loader=Loader)
Tagged('!GetAZs', 'us-east-1')

It works with arbitrarily nested YAML:

>>> value = yaml.load("""
... Properties:
...   ImageId: !FindInMap [RegionMap, !Ref 'AWS::Region', HVM64]
... """, Loader=Loader)
>>> value
{
    'Properties': {
        'ImageId': Tagged(
            '!FindInMap',
            ['RegionMap', Tagged('!Ref', 'AWS::Region'), 'HVM64']
        )
    }
}

... allows you to ignore tags most of the time:

>>> value['Properties']['ImageId'][-1] = 'HVMG2'

... and can output tagged YAML too:

>>> print(yaml.dump(value, Dumper=Dumper))
Properties:
  ImageId: !FindInMap
  - RegionMap
  - !Ref 'AWS::Region'
  - HVMG2

Check it out: Dealing with YAML with arbitrary tags in Python.

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Dealing with YAML with arbitrary tags in Python

2022-02-07T15:21:00+00:00

... in which we use PyYAML to safely read and write YAML with any tags, in a way that's as straightforward as interacting with built-in types.

If you're in a hurry, you can find the code at the end.

Contents

Why is this useful?
A note on PyYAML extensibility
Preserving tags
Unhashable keys
- Constructing pairs
- Representing pairs
Conclusion
Bonus: hashable wrapper
Bonus: broken YAML

Why is this useful? #

People mostly use YAML as a friendlier alternative to JSON¹, but it can do way more.

Among others, it can natively represent user-defined and native data structures.

Say you need to read (or write) an AWS CloudFormation template:

EC2Instance:
  Type: AWS::EC2::Instance
  Properties:
    ImageId: !FindInMap [
      AWSRegionArch2AMI,
      !Ref 'AWS::Region',
      !FindInMap [AWSInstanceType2Arch, !Ref InstanceType, Arch],
    ]
    InstanceType: !Ref InstanceType

>>> yaml.safe_load(text)
Traceback (most recent call last):
  ...
yaml.constructor.ConstructorError: could not determine a constructor for the tag '!FindInMap'
  in "<unicode string>", line 4, column 14:
        ImageId: !FindInMap [
                 ^

... or, you need to safely read untrusted YAML that represents Python objects:

!!python/object/new:module.Class { attribute: value }

>>> yaml.safe_load(text)
Traceback (most recent call last):
 ...
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/new:module.Class'
  in "<unicode string>", line 1, column 1:
    !!python/object/new:module.Class ...
    ^

Warning

Historically, yaml.load(thing) was unsafe for untrusted data, because it allowed running arbitrary code. Consider using safe_load() instead.

Details.

For example, you could do this:

>>> yaml.load("!!python/object/new:os.system [echo WOOSH. YOU HAVE been compromised]")
WOOSH. YOU HAVE been compromised
0

There were a bunch of CVEs about it.

To address the issue, load() requires an explicit Loader since PyYAML 6. Also, version 5 added two new functions and corresponding loaders:

full_load() resolves all tags except those known to be unsafe (note that this was broken before 5.4, and thus vulnerable)
unsafe_load() resolves all tags, even those known to be unsafe (the old load() behavior)

safe_load() resolves only basic tags, remaining the safest.

Can I just get the data, without it being turned into objects?

You can! The YAML spec says:

In a given processing environment, there need not be an available native type corresponding to a given tag. If a node’s tag is unavailable, a YAML processor will not be able to construct a native data structure for it. In this case, a complete representation may still be composed and an application may wish to use this representation directly.

And PyYAML obliges:

>>> text = """\
... one: !myscalar string
... two: !mysequence [1, 2]
... """
>>> yaml.compose(text)
MappingNode(
    tag='tag:yaml.org,2002:map',
    value=[
        (
            ScalarNode(tag='tag:yaml.org,2002:str', value='one'),
            ScalarNode(tag='!myscalar', value='string'),
        ),
        (
            ScalarNode(tag='tag:yaml.org,2002:str', value='two'),
            SequenceNode(
                tag='!mysequence',
                value=[
                    ScalarNode(tag='tag:yaml.org,2002:int', value='1'),
                    ScalarNode(tag='tag:yaml.org,2002:int', value='2'),
                ],
            ),
        ),
    ],
)
>>> print(yaml.serialize(_))
one: !myscalar 'string'
two: !mysequence [1, 2]

... the spec didn't say the representation has to be concise. ¯\_(ツ)_/¯

Here's how YAML processing works, to give you an idea what we're looking at:

YAML Processing Overview

The output of compose() above is the representation (node graph).

From that, safe_load() does its best to construct objects, but it can't do anything for tags it doesn't know about.

There must be a better way!

Thankfully, the spec also says:

That said, tag resolution is specific to the application. YAML processors should therefore provide a mechanism allowing the application to override and expand these default tag resolution rules.

We'll use this mechanism to convert tagged nodes to almost-native types, while preserving the tags.

A note on PyYAML extensibility #

PyYAML is a bit unusual.

For each processing direction, you have a corresponding Loader/Dumper class.

For each processing step, you can add callbacks, stored in class-level registries.

The callbacks are method-like – they receive the Loader/Dumper as the first argument:

Dice = namedtuple('Dice', 'a b')

def dice_representer(dumper, data):
    return dumper.represent_scalar(u'!dice', u'%sd%s' % data)

yaml.Dumper.add_representer(Dice, dice_representer)

You may notice the add_...() methods modify the class in-place, for everyone, which isn't necessarily great; imagine getting a Dice from safe_load(), when you were expecting only built-in types.

We can avoid this by subclassing, since the registry is copied from the parent. Note that because of how copying is implemented, registries from two direct parents are not merged – you only get the registry of the first parent in the MRO.

So, we'll start by subclassing SafeLoader/Dumper:

class Loader(yaml.SafeLoader):
    pass

class Dumper(yaml.SafeDumper):
    pass

Preserving tags #

Constructing unknown objects #

For now, we can use named tuples for objects with unknown tags, since they are naturally tag/value pairs:

class Tagged(typing.NamedTuple):
    tag: str
    value: object

Tag or no tag, all YAML nodes are either a scalar, a sequence, or a mapping. For unknown tags, we delegate construction to the loader's default constructors, and wrap the resulting value:

def construct_undefined(self, node):
    if isinstance(node, yaml.nodes.ScalarNode):
        value = self.construct_scalar(node)
    elif isinstance(node, yaml.nodes.SequenceNode):
        value = self.construct_sequence(node)
    elif isinstance(node, yaml.nodes.MappingNode):
        value = self.construct_mapping(node)
    else:
        assert False, f"unexpected node: {node!r}"
    return Tagged(node.tag, value)

Loader.add_constructor(None, construct_undefined)

Constructors are registered by tag, with None meaning "unknown".

Things look much better already:

>>> yaml.load(text, Loader=Loader)
{
    'one': Tagged(tag='!myscalar', value='string'),
    'two': Tagged(tag='!mysequence', value=[1, 2]),
}

A better wrapper #

That's nice, but every time we use any value, we have to check if it's tagged, and then go through value if is:

>>> one = _['one']
>>> one.tag
'!myscalar'
>>> one.value.upper()
'STRING'

We could subclass the Python types corresponding to core YAML tags (str, list, and so on), and add a tag attribute to each. We could subclass most of them, anyway – neither bool nor NoneType can be subclassed.

Or, we could wrap tagged objects in a class with the same interface, that delegates method calls and attribute access to the wrapee, with a tag attribute on top.

Tip

This is known as the decorator pattern design pattern (not to be confused with Python decorators). Caching and retrying methods dynamically is another interesting use for it.

Doing this naively entails writing one wrapper per type, with one wrapper method per method and one property per attribute. That's even worse than subclassing!

There must be a better way!

Of course, this is Python, so there is.

We can use an object proxy instead (also known as "dynamic wrapper"). While they're not perfect in general, the one wrapt provides is damn near perfect enough²:

class Tagged(wrapt.ObjectProxy):

    # tell wrapt to set the attribute on the proxy, not the wrapped object
    tag = None

    def __init__(self, tag, wrapped):
        super().__init__(wrapped)
        self.tag = tag

    def __repr__(self):
        return f"{type(self).__name__}({self.tag!r}, {self.__wrapped__!r})"

>>> yaml.load(text, Loader=Loader)
{
    'one': Tagged('!myscalar', 'string'),
    'two': Tagged('!mysequence', [1, 2]),
}

The proxy behaves identically to the proxied object:

>>> one = _['one']
>>> one.tag
'!myscalar'
>>> one.upper()
'STRING'
>>> one[:3]
'str'

...up to and including fancy things like isinstance():

>>> isinstance(one, str)
True
>>> isinstance(one, Tagged)
True

And now you don't have to care about tags if you don't want to.

Representing tagged objects #

The trip back is exactly the same, but much shorter:

def represent_tagged(self, data):
    assert isinstance(data, Tagged), data
    node = self.represent_data(data.__wrapped__)
    node.tag = data.tag
    return node

Dumper.add_representer(Tagged, represent_tagged)

Representers are registered by type.

>>> print(yaml.dump(Tagged('!hello', 'world'), Dumper=Dumper))
!hello 'world'

Let's mark the occasion with some tests.

Since we still have stuff to do, we parametrize the tests from the start.

BASIC_TEXT = """\
one: !myscalar string
two: !mymapping
  three: !mysequence [1, 2]
"""

BASIC_DATA = {
    'one': Tagged('!myscalar', 'string'),
    'two': Tagged('!mymapping', {'three': Tagged('!mysequence', [1, 2])}),
}

DATA = [
    (BASIC_TEXT, BASIC_DATA),
]

Loading works:

@pytest.mark.parametrize('text, data', DATA)
def test_load(text, data):
    assert yaml.load(text, Loader=Loader) == data

And dumping works:

@pytest.mark.parametrize('text', [t[0] for t in DATA])
def test_roundtrip(text):
    data = yaml.load(text, Loader=Loader)
    assert data == yaml.load(yaml.dump(data, Dumper=Dumper), Loader=Loader)

... but only for known types:

def test_dump_error():
    with pytest.raises(yaml.representer.RepresenterError):
        yaml.dump(object(), Dumper=Dumper)

Unhashable keys #

Let's try an example from the PyYAML documentation:

>>> text = """\
... ? !!python/tuple [0,0]
... : The Hero
... ? !!python/tuple [1,0]
... : Treasure
... ? !!python/tuple [1,1]
... : The Dragon
... """

This is supposed to result in something like:

>>> yaml.unsafe_load(text)
{(0, 0): 'The Hero', (1, 0): 'Treasure', (1, 1): 'The Dragon'}

Instead, we get:

>>> yaml.load(text, Loader=Loader)
Traceback (most recent call last):
  ...
TypeError: unhashable type: 'list'

That's because the keys are tagged lists, and neither type is hashable:

>>> yaml.load("!!python/tuple [0,0]", Loader=Loader)
Tagged('tag:yaml.org,2002:python/tuple', [0, 0])

This limitation comes from how Python dicts are implemented,³ not from YAML; quoting from the spec again:

The content of a mapping node is an unordered set of key/value node pairs, with the restriction that each of the keys is unique. YAML places no further restrictions on the nodes. In particular, keys may be arbitrary nodes, the same node may be used as the value of several key/value pairs and a mapping could even contain itself as a key or a value.

Constructing pairs #

What now?

Same strategy as before: wrap the things we can't handle.

Specifically, whenever we have a mapping with unhashable keys, we return a list of pairs instead. To tell it apart from plain lists, we use a subclass:

class Pairs(list):
    def __repr__(self):
        return f"{type(self).__name__}({super().__repr__()})"

Again, we let the loader do most of the work:

def construct_mapping(self, node):
    value = self.construct_pairs(node)
    try:
        return dict(value)
    except TypeError:
        return Pairs(value)

Loader.construct_mapping = construct_mapping
Loader.add_constructor('tag:yaml.org,2002:map', Loader.construct_mapping)

We set construct_mapping so that any other Loader constructor wanting to make a mapping gets to use it (like our own construct_undefined() above). Don't be fooled by the assignment, it's a method like any other⁴ ...but we're changing the class from outside anyway, so it's best to stay consistent.

Note that overriding construct_mapping() is not enough: we have to register the constructor explictly, otherwise SafeDumper's construct_mapping() will be used (since that's what was in the registry before).

Note

In case you're wondering, this feature is orthogonal from handling unknown tags; we could have used different classes for them. However, as mentioned before, the constructor registry breaks multiple inheritance, so we couldn't use the two features together.

Anyway, it works:

>>> yaml.load(text, Loader=Loader)
Pairs(
    [
        (Tagged('tag:yaml.org,2002:python/tuple', [0, 0]), 'The Hero'),
        (Tagged('tag:yaml.org,2002:python/tuple', [1, 0]), 'Treasure'),
        (Tagged('tag:yaml.org,2002:python/tuple', [1, 1]), 'The Dragon'),
    ]
)

Representing pairs #

Like before, the trip back is short and uneventful:

def represent_pairs(self, data):
    assert isinstance(data, Pairs), data
    node = self.represent_dict(data)
    return node

Dumper.add_representer(Pairs, represent_pairs)

>>> print(yaml.dump(Pairs([([], 'one')]), Dumper=Dumper))
[]: one

Let's test this more thoroughly.

Because the tests are parametrized, we just need to add more data:

UNHASHABLE_TEXT = """\
[0,0]: one
!key {0: 1}: {[]: !value three}
"""

UNHASHABLE_DATA = Pairs(
    [
        ([0, 0], 'one'),
        (Tagged('!key', {0: 1}), Pairs([([], Tagged('!value', 'three'))])),
    ]
)

DATA = [
    (BASIC_TEXT, BASIC_DATA),
    (UNHASHABLE_TEXT, UNHASHABLE_DATA),
]

Conclusion #

YAML is extensible by design. I hope that besides what it says on the tin, this article shed some light on how to customize PyYAML for your own purposes, and that you've learned at least one new Python thing.

You can get the code here, and the tests here.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Bonus: hashable wrapper #

You may be asking, why not make the wrapper hashable?

Most unhashable (data) objects are that for a reason: because they're mutable.

We have two options:

Make the wrapper hash change with the content. This this will break dictionaries in strange and unexpected ways (and other things too) – the language requires mutable objects to be unhashable.
Make the wrapper hash not change with the content, and wrappers equal only to themselves – that's what user-defined classes do by default anyway.

This works, but it's not very useful, because equal values don't compare equal anymore (data != load(dump(data))). Also, it means you can only get things from a dict if you already have the object used as key:
```
>>> data = {Hashable([1]): 'one'}
>>> data[Hashable([1])]
Traceback (most recent call last):
  ...
KeyError: Hashable([1])
>>> key = list(data)[0]
>>> data[key]
'one'
```
I'd file this under "strange and unexpected" too.

(You can find the code for the example above here.)

Bonus: broken YAML #

We can venture even farther, into arguably broken YAML. Let's look at some examples.

First, there are undefined tag prefixes:

>>> yaml.load("!m!xyz x", Loader=Loader)
Traceback (most recent call last):
  ...
yaml.parser.ParserError: while parsing a node
found undefined tag handle '!m!'
  in "<unicode string>", line 1, column 1:
    !m!xyz x
    ^

A valid version:

>>> yaml.load("""\
... %TAG !m! !my-
... ---
... !m!xyz x
... """, Loader=Loader)
Tagged('!my-xyz', 'x')

Second, there are undefined aliases:

>>> yaml.load("two: *anchor", Loader=Loader)
Traceback (most recent call last):
  ...
yaml.composer.ComposerError: found undefined alias 'anchor'
  in "<unicode string>", line 1, column 6:
    two: *anchor
         ^

A valid version:

>>> yaml.load("""\
... one: &anchor [1]
... two: *anchor
... """, Loader=Loader)
{'one': [1], 'two': [1]}

It's likely possible to handle these in a way similar to how we handled undefined tags, but we'd have to go deeper – the exceptions hint to which processing step to look at.

Since I haven't actually encountered them in real life, we'll "save them for later" :)

Of which YAML is actually a superset. ^[return]
Timothy 20:9. ^[return]
Using a hash table. For nice explanation of how it all works, complete with a pure-Python implementation, check out Raymond Hettinger's talk Modern Python Dictionaries: A confluence of a dozen great ideas (code). ^[return]
Almost. The zero argument form of super() won't work for methods defined outside of a class definition, but we're not using it here. ^[return]

reader 2.5 released – usage statistics

2021-11-01T07:25:00+00:00

Hi there!

I'm happy to announce version 2.5 of reader, a Python feed reader library.

What's new? #

Here are the most important changes since reader 2.0.

Search enabled by default #

Full-text search works out of the box: no extra dependencies, no setup needed.

Statistics #

There are now statistics on feed and user activity, to give you a better understanding of how you consume content.

A nice side-effect of knowing when entry flags changed is that now it's possible to tell if an entry was explicitly marked as unimportant (new entries are unimportant by default).

Improved duplicate handling #

Duplicate handling got significantly better:

False negatives are reduced by using approximate string matching and heuristics to detect truncated content.
You can trigger entry deduplication manually, for the existing entries of a feed – just add the .reader.dedupe.once tag to the feed, and wait for the next update. Also, you can deduplicate entries by title alone, ignoring content.
Old duplicates are deleted instead of marked as read/unimportant.

User-added entries #

You can now add entries to existing feeds. This is useful when you want to keep track of an article that is not in the feed anymore because it "fell off the end".

It can also be used to build bookmarking / read later functionality similar to that of Tiny Tiny RSS; extracting content from arbitrary pages would be pretty helpful here.

New Python versions #

reader now supports Python 3.10 and PyPy 3.8.

Other changes #

Aside from the changes mentioned above, I added a new plugin hook, added a few convenience methods and attributes, updated the web application and plugins to take advantage of the new features, and fixed a few minor bugs.

See the changelog for details.

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark entries as read or important
add tags and metadata to feeds
filter feeds and articles
full-text search articles
get statistics on feed and user activity
write plugins to extend its functionality

...all these with:

a stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

Write an SQL query builder in 150 lines of Python!

2021-08-27T13:47:00+00:00

Previously

This is the fourth article in a series about writing an SQL query builder for my feed reader library.

Today, we'll dive into the code by rewriting it from scratch.

Think of it as part walk-through, part tutorial; along the way, we'll explore:

API design
knowing when to be lazy
worse and better ways of doing things

As you read this, keep in mind it is a story, thus linear by necessity. Development was decidedly not so: I tried things out, I changed my mind multiple times, and I rewrote everything once. Even now, there are other equally-good or better implementations; this one is simply good enough.

Contents

What are we trying to build?
- Trade-offs
A minimal plausible solution
Separators
Aliases
Subqueries
Joins
Distinct
More tests
More init
Bonus: things that didn't make the cut

What are we trying to build? #

We want a way of building SQL strings that takes care of formatting:

>>> query = Query()
>>> query.SELECT('one').FROM('table')
<builder.Query object at 0x7fc953e60640>
>>> print(query)
SELECT
    one
FROM
    table

... and allows us to add parts incrementally:

>>> query.SELECT('two').WHERE('condition')
<builder.Query object at 0x7fc953e60640>
>>> print(query)
SELECT
    one,
    two
FROM
    table
WHERE
    condition

While not required, I recommend reading the previous articles to get a better idea of the problem we're trying to solve, and the context we're solving it in.

In short, whatever we build should:

support SELECT with conditional WITH, WHERE, ORDER BY, JOIN etc.
expose the names of the result columns (for scrolling window queries)
be easy to use, understand and maintain

Attention

This query builder is not directly comparable with that of an ORM. Instead, it is an alternative to building plain SQL strings by hand.

The caveats that apply to plain SQL apply to it as well: Using user-supplied values directly in an SQL query exposes you to SQL injection attacks. Instead, use parametrized queries whenever possible, and escaping only as a last resort.

Trade-offs #

Our solution does not exist in a void; it exists to be used by my feed reader library.

Notably, we're not making a general-purpose library with external users whose needs we're trying to anticipate; there's exactly one user with a pretty well-defined use case, and strict backwards compatibility is not necessary.

This allows us to make some upfront decisions to help with maintainability:

No needless customization. We can change the code directly if we need to.
No other features except the known requirements. We can add new ones when we need them.
No effort to support other syntax than SQLite.
No extensive testing. We can rely on the exising comprehensive functional tests.
No SQL validation. The database does this already.
- However, it would be nice to get at least a little error checking. No need for custom exceptions, any kind is acceptable – they should come up only during development and testing anyway.

A minimal plausible solution #

Data representation #

As mentioned before, my prototype was based on the idea that queries can be represented as plain data structures.

Looking at a nicely formatted query, a natural representation may reveal itself:

SELECT
    one,
    two
FROM
    table
WHERE
    condition AND
    another-condition

See it?

It's a mapping with a list of strings for each clause:

{
    'SELECT': [
        'one',
        'two',
    ],
    'FROM': [
        'table',
    ],
    'WHERE': [
        'condition',
        'another-condition',
    ],
}

Let's use this as our starting model, and make ourselves a query builder.

Classes #

We start with a class:

class Query:

    keywords = [
        'WITH',
        'SELECT',
        'FROM',
        'WHERE',
        'GROUP BY',
        'HAVING',
        'ORDER BY',
        'LIMIT',
    ]

    def __init__(self):
        self.data = {k: [] for k in self.keywords}

We use a class because most of the time we don't want to interact with the underlying data structure, since it's more likely to change. We're not subclassing dict, since that would unintentionally expose its methods (and thus, behavior), and we may need those names for something else.

Also, a class allows us to reduce verbosity:

# we want
query.SELECT('one', 'two').FROM('table')
# not
query['SELECT'].extend(['one', 'two'])
query['FROM'].append('table')

We use class variables for "static" data instead of hardcoding or module variables so it's easy to override (more on that later).

We don't customize anything in __init__() fow now; if we need more clauses, we can add them to keywords directly. Adding all known keywords to data upfront gets us free error checking: data[keyword] raises KeyError for unknown keywords.

Note

Unless specified otherwise, I'll use clause and keyword to mean "item in self.data", not "SQL clause or keyword in general".

We could use dataclasses, but of the generated magic methods, we'd only use __repr__(), and its output would be too long to be useful anyway.

Adding things #

Next, we add code for adding string fragments to each clause:

    def add(self, keyword, *args):
        target = self.data[keyword]

        for arg in args:
            target.append(_clean_up(arg))

        return self

    def __getattr__(self, name):
        if not name.isupper():
            return getattr(super(), name)
        return functools.partial(self.add, name.replace('_', ' '))


def _clean_up(thing: str) -> str:
    return textwrap.dedent(thing.rstrip()).strip()

add() is roughly equivalent to data[keyword].extend(args).

The main difference is that we dedent the arguments and remove trailing whitespace. This is intentional: we clean everything up and make as many choices when adding things, so we don't have to care about that when generating output, and so error checking happens as early as possible.

Also, add() returns self to enable method chaining: query.add(...).add(...).

__getattr__() is called when an attribute does not exist, and allows us to return something instead of getting the default AttributeError.

What we return is a KEYWORD(*args) callable made on the fly by wrapping add() in a partial; a closure capturing name would be functionally equivalent.

Requiring the keywords to be uppercase is a stylistic choice, but does have advantages: it signals to the reader these are special "methods", and avoids shadowing dunder methods like __deepcopy__() without extra checks.

To indicate the attribute really doesn't exist, we need to raise AttributeError; we let getattr() do it for us (the parent object doesn't have a custom __getattr__()).

We could store the partial on the instance, which would side-step __getattr__() on subsequent calls, so we only make one partial per keyword; we could do it in __init__(), and not use __getattr__() at all; we could even use partialmethod, so there's only one per keyword per class! Or we can do nothing – they're likely premature optimization, and what we're doing now is more flexible anyway.

I said error checking happens as early as possible; that's almost true: if you look carefully at the code, you may notice query.ESLECT doesn't raise an exception until called – query.ESLECT().

Doing most of the work in add() does have some benefits, though: we can use it with partial and get chaining for free, and it's an escape hatch for when we want to use a "keyword" that's not a Python identifier (this will be useful later).

Output #

Finally, we turn the query into SQL:

    default_separator = ','

    def __str__(self):
        return ''.join(self._lines())

    def _lines(self):
        for keyword, things in self.data.items():
            if not things:
                continue

            yield f'{keyword}\n'
            yield from self._lines_keyword(keyword, things)

    def _lines_keyword(self, keyword, things):
        for i, thing in enumerate(things, 1):
            last = i == len(things)

            yield self._indent(thing)

            if not last:
                yield self.default_separator

            yield '\n'

    _indent = functools.partial(textwrap.indent, prefix='    ')

The only output API is str(); being the standard way of turning objects into strings in Python, it requires zero effort to learn.

str(query) calls __str__, which delegates to _lines(). We use a generator mainly because it allows us to write yield line instead of rv.append(line), making for somewhat cleaner code.

Another benefit of a generator is that it's lazy, so we can pass it around without having to build intermediary lists in memory; for example, to a file's writelines() method, or in yield from in another generator (e.g. for nested subqueries). We don't need it here, but it's useful when generating a lot of values.

We split the logic for individual clauses into _lines_keyword(), because we'll keep adding stuff to it. (I initially left everything in _lines(), and refactored when things got too complicated; no need to do that now.)

Since we'll want to indent things in the same way in more than one place, we make it a static "method" using partial.

You may notice we're not sorting the clauses in any way; dicts guarantee insertion order in Python 3.6+¹, and we built data from keywords, so the order is preserved.

Tests #

Let's add a simple test to make sure we don't break already working stuff:

def test_query_simple():
    query = Query().SELECT('select').FROM('from-one', 'from-two')
    assert str(query) == dedent(
        """\
        SELECT
            select
        FROM
            from-one,
            from-two
        """
    )

We'll keep adding to it with each feature.

For a minimal solution, we are done. We've "spent" 62 lines, or 38 statements.

The code so far: builder.py, test_builder.py.

Separators #

At this point, WHERE doesn't really make sense:

>>> print(Query().WHERE('a', 'b'))
WHERE
    a,
    b

We fix it by special-casing separators for a few clauses:

    separators = dict(WHERE='AND', HAVING='AND')

    def _lines_keyword(self, keyword, things):
        for i, thing in enumerate(things, 1):
            last = i == len(things)

            yield self._indent(thing)

            if not last:
                try:
                    yield ' ' + self.separators[keyword]
                except KeyError:
                    yield self.default_separator

            yield '\n'

We could've used defaultdict instead of using default_separator, but then we'd have to remember non-comma separators need a space: ' AND'; putting it in code means we don't have to remember anything.

Also, we could've put the separator on a new line: 'one\nAND two' vs. 'one AND\ntwo'. While slightly better style, it makes code more complicated for little benefit, and makes it less obvious that AND is just another separator.

We add WHERE to the test.

def test_query_simple():
    query = (
        Query()
        .SELECT('select')
        .FROM('from-one', 'from-two')
        .WHERE('where-one', 'where-two')
    )
    assert str(query) == dedent(
        """\
        SELECT
            select
        FROM
            from-one,
            from-two
        WHERE
            where-one AND
            where-two
        """
    )

The code so far: builder.py, test_builder.py.

Aliases #

One of the requirements is that it should be possible to implement scrolling window queries on top. For this, code needs to get the result column names – the SELECT expressions or their aliases – and add them to a generated WHERE condition.

Parsing the result column is straightforward only for simple cases:

>>> query = Query().SELECT(
...   'column',
...   'column AS alias',
...   'column as alias',
...   '(SELECT column FROM table AS another-table)',
... )
>>> [s.rpartition(' AS ')[2] for s in query.data['SELECT']]
['column', 'alias', 'column as alias', 'another-table)']

An acceptable compromise is using pairs of strings for aliased columns. Since the column expression might be quite long, we'll make the alias the first thing in the pair.

>>> print(Query().SELECT(('alias', 'one'), 'two'))
SELECT
    one AS alias,
    two

As mentioned earlier, we store everything in a standard way to keep output code simpler. A plain 2-tuple is a decent choice, but a named tuple is more readable.

class _Thing(NamedTuple):
    value: str
    alias: str = ''

    @classmethod
    def from_arg(cls, arg):
        if isinstance(arg, str):
            alias, value = '', arg
        elif len(arg) == 2:
            alias, value = arg
        else:
            raise ValueError(f"invalid arg: {arg!r}")
        return cls(_clean_up(value), _clean_up(alias))

Conveniently, this gives us a place where to convert the string-or-pair: the from_arg() alternate constructor. We could've made it a stand-alone function, but this way it's easier to see what type is being returned.

Note

Note that we use an empty string to mean "no alias". In general, it's a good idea to distinguish this kind of absence by using None, since the empty string may be a valid input, and None can prevent some bugs – e.g. you can't concatenate None to a string. Here, an empty string cannot be a valid alias, and we use format strings, so we don't bother.

Using it is just a one-line change to add():

    def add(self, keyword, *args):
        target = self.data[keyword]

        for arg in args:
            target.append(_Thing.from_arg(arg))

        return self

On output, we have two concerns:

there may or may not be an alias
the order differs depending on the keyword: you have SELECT expr AS column-alias, but WITH table-name AS (stmt) (we treat the CTE table name as an alias)

We can model this with mostly-empty defaultdicts with per-clause format strings:

    formats = (
        defaultdict(lambda: '{value}'),
        defaultdict(lambda: '{value} AS {alias}', WITH='{alias} AS {value}'),
    )

... and choose the right defaultdict using the alias's boolean value:²

    def _lines_keyword(self, keyword, things):
        for i, thing in enumerate(things, 1):
            last = i == len(things)

            format = self.formats[bool(thing.alias)][keyword]
            yield self._indent(format.format(value=thing.value, alias=thing.alias))

            if not last:
                try:
                    yield ' ' + self.separators[keyword]
                except KeyError:
                    yield self.default_separator

            yield '\n'

We add an aliased expression to the test.

def test_query_simple():
    query = (
        Query()
        .SELECT('select-one', ('alias', 'select-two'))
        .FROM('from-one', 'from-two')
        .WHERE('where-one', 'where-two')
    )
    assert str(query) == dedent(
        """\
        SELECT
            select-one,
            select-two AS alias
        FROM
            from-one,
            from-two
        WHERE
            where-one AND
            where-two
        """
    )

The code so far: builder.py, test_builder.py.

Subqueries #

Currently, WITH is still a little broken:

>>> print(Query().WITH(('table-name', 'SELECT 1')))
WITH
    table-name AS SELECT 1

Since common table expressions always have the SELECT statement paranthesized, we'd like to have it out of the box, with proper indentation:

WITH
    table-name AS (
        SELECT 1
    )

A simple way of handling this is to change the WITH format string to '{alias} AS (\n{indented}\n)', where indented is the value, but indented.³

This kinda works, but is limited in usefulness; for instance, we can't easily build something like this on top:

Query().FROM(('alias', 'SELECT 1'), is_subquery=True)

Instead, let's keep refining our model, and use a flag to mark subqueries:

class _Thing(NamedTuple):
    value: str
    alias: str = ''
    is_subquery: bool = False

    @classmethod
    def from_arg(cls, arg, **kwargs):
        if isinstance(arg, str):
            alias, value = '', arg
        elif len(arg) == 2:
            alias, value = arg
        else:
            raise ValueError(f"invalid arg: {arg!r}")
        return cls(_clean_up(value), _clean_up(alias), **kwargs)

We can then check if a clause always has subqueries, and set the flag accordingly:

    subquery_keywords = {'WITH'}

    def add(self, keyword, *args):
        target = self.data[keyword]

        kwargs = {}
        if keyword in self.subquery_keywords:
            kwargs.update(is_subquery=True)

        for arg in args:
            target.append(_Thing.from_arg(arg, **kwargs))

        return self

Using it for output is just an extra if:

    def _lines_keyword(self, keyword, things):
        for i, thing in enumerate(things, 1):
            last = i == len(things)

            format = self.formats[bool(thing.alias)][keyword]
            value = thing.value
            if thing.is_subquery:
                value = f'(\n{self._indent(value)}\n)'
            yield self._indent(format.format(value=value, alias=thing.alias))

            if not last:
                try:
                    yield ' ' + self.separators[keyword]
                except KeyError:
                    yield self.default_separator

            yield '\n'

We add WITH to our test.

def test_query_simple():
    query = (
        Query()
        .WITH(('alias', 'with'))
        .SELECT('select-one', ('alias', 'select-two'))
        .FROM('from-one', 'from-two')
        .WHERE('where-one', 'where-two')
    )
    assert str(query) == dedent(
        """\
        WITH
            alias AS (
                with
            )
        SELECT
            select-one,
            select-two AS alias
        FROM
            from-one,
            from-two
        WHERE
            where-one AND
            where-two
        """
    )

The code so far: builder.py, test_builder.py.

Joins #

One clause that's entirely missing is JOIN. And it's important, changing your mind about what you're selecting from happens quite often.

JOIN is a bit more complicated, mostly because it has different forms – JOIN, LEFT JOIN and so on; SQLite supports at least 10 variations.

I initially treated any keyword containing JOIN as a separate keyword, and dealt with it during output. This has a few drawbacks, though; aside from making the code more complicated, it reorders the tables: query.JOIN('a').LEFT_JOIN('b').JOIN('c') results in JOIN a JOIN c LEFT JOIN b.

A better solution is to refine our model even further.

Take a look at these railroad diagrams for the SELECT statement:

select-core (FROM clause)

join-clause

join-operator

You may notice table-or-subquery followed by , in FROM is actually a subset of table-or-subquery followed by join-operator in join-clause. That is, for SQLite, a comma is just another join operator.

Put the other way around, a join operator is just another separator.

Because our separators come after things, not before, we'll model join operators separately, as fake keywords (that is, not used to index into data).

First, let's set them:

class _Thing(NamedTuple):
    value: str
    alias: str = ''
    keyword: str = ''
    is_subquery: bool = False

    fake_keywords = dict(JOIN='FROM')

    def add(self, keyword, *args):
        keyword, fake_keyword = self._resolve_fakes(keyword)
        target = self.data[keyword]

        kwargs = {}
        if fake_keyword:
            kwargs.update(keyword=fake_keyword)
        if keyword in self.subquery_keywords:
            kwargs.update(is_subquery=True)

        for arg in args:
            target.append(_Thing.from_arg(arg, **kwargs))

        return self

    def _resolve_fakes(self, keyword):
        for part, real in self.fake_keywords.items():
            if part in keyword:
                return real, keyword
        return keyword, ''

We could've probably just hardcoded this in add() (if 'JOIN' in keyword: ...), but doing it like this makes it easier to see at a glance that "JOIN is a fake FROM".

Using keyword as a separator is relatively straightforward:

    def _lines(self):
        for keyword, things in self.data.items():
            if not things:
                continue

            yield f'{keyword}\n'

            grouped = [], []
            for thing in things:
                grouped[bool(thing.keyword)].append(thing)
            for group in grouped:
                yield from self._lines_keyword(keyword, group)

    def _lines_keyword(self, keyword, things):
        for i, thing in enumerate(things, 1):
            last = i == len(things)

            if thing.keyword:
                yield thing.keyword + '\n'

            format = self.formats[bool(thing.alias)][keyword]
            value = thing.value
            if thing.is_subquery:
                value = f'(\n{self._indent(value)}\n)'
            yield self._indent(format.format(value=value, alias=thing.alias))

            if not last and not thing.keyword:
                try:
                    yield ' ' + self.separators[keyword]
                except KeyError:
                    yield self.default_separator

            yield '\n'

Since FROM always comes before JOIN, we make sure to output the real ones first.

We add a JOIN to the test.

def test_query_simple():
    query = (
        Query()
        .WITH(('alias', 'with'))
        .SELECT('select-one', ('alias', 'select-two'))
        .FROM('from-one', 'from-two')
        .JOIN('join')
        .WHERE('where-one', 'where-two')
    )
    assert str(query) == dedent(
        """\
        WITH
            alias AS (
                with
            )
        SELECT
            select-one,
            select-two AS alias
        FROM
            from-one,
            from-two
        JOIN
            join
        WHERE
            where-one AND
            where-two
        """
    )

The code so far: builder.py, test_builder.py.

Distinct #

The final model change is to support SELECT DISTINCT.

DISTINCT and ALL are flags that apply to the whole clause; we'll model them as such:

class _FlagList(list):
    flag: str = ''

    def __init__(self):
        self.data = {k: _FlagList() for k in self.keywords}

Since most of the time we're OK with the default flag, we don't bother setting it in __init__, and use a class variable instead. If we need to customize it, we can set flag on the instance, shadowing the class variable.

A __repr__ showing the flag would be nice, but it'd only be useful during debugging, so we skip it as well.

We set the flag based on a known set for each clause; like with fake keywords, we pull the "parsing" logic into a separate method:

    flag_keywords = dict(SELECT={'DISTINCT', 'ALL'})

    def add(self, keyword, *args):
        keyword, fake_keyword = self._resolve_fakes(keyword)
        keyword, flag = self._resolve_flags(keyword)
        target = self.data[keyword]

        if flag:
            if target.flag:
                raise ValueError(f"{keyword} already has flag: {flag!r}")
            target.flag = flag

        kwargs = {}
        if fake_keyword:
            kwargs.update(keyword=fake_keyword)
        if keyword in self.subquery_keywords:
            kwargs.update(is_subquery=True)

        for arg in args:
            target.append(_Thing.from_arg(arg, **kwargs))

        return self

    def _resolve_flags(self, keyword):
        prefix, _, flag = keyword.partition(' ')
        if prefix in self.flag_keywords:
            if flag and flag not in self.flag_keywords[prefix]:
                raise ValueError(f"invalid flag for {prefix}: {flag!r}")
            return prefix, flag
        return keyword, ''

Using it for output is again straightforward:

    def _lines(self):
        for keyword, things in self.data.items():
            if not things:
                continue

            if things.flag:
                yield f'{keyword} {things.flag}\n'
            else:
                yield f'{keyword}\n'

            grouped = [], []
            for thing in things:
                grouped[bool(thing.keyword)].append(thing)
            for group in grouped:
                yield from self._lines_keyword(keyword, group)

We add a SELECT DISTINCT to our test.

def test_query_simple():
    query = (
        Query()
        .WITH(('alias', 'with'))
        .SELECT('select-one', ('alias', 'select-two'))
        .FROM('from-one', 'from-two')
        .JOIN('join')
        .WHERE('where-one', 'where-two')
        .SELECT_DISTINCT('select-three')
    )
    assert str(query) == dedent(
        """\
        WITH
            alias AS (
                with
            )
        SELECT DISTINCT
            select-one,
            select-two AS alias,
            select-three
        FROM
            from-one,
            from-two
        JOIN
            join
        WHERE
            where-one AND
            where-two
        """
    )

The code so far: builder.py, test_builder.py.

More tests #

Our only test isn't all that simple anymore; maybe it's time to split it in two: one with a really simple query, and one with a really complicated query.

... something like this.

def test_query_simple():
    query = Query().SELECT('select').FROM('from').JOIN('join').WHERE('where')
    assert str(query) == dedent(
        """\
        SELECT
            select
        FROM
            from
        JOIN
            join
        WHERE
            where
        """
    )


def test_query_complicated():
    """Test a complicated query:

    * order between different keywords does not matter
    * arguments of repeated calls get appended, with the order preserved
    * SELECT can receive 2-tuples
    * WHERE and HAVING arguments are separated by AND
    * JOIN arguments are separated by the keyword, and come after plain FROM
    * no-argument keywords have no effect, unless they are flags

    """
    query = (
        Query()
        .WHERE()
        .OUTER_JOIN('outer join')
        .JOIN('join')
        .LIMIT('limit')
        .JOIN()
        .ORDER_BY('first', 'second')
        .SELECT('one')
        .HAVING('having')
        .SELECT(('two', 'expr'))
        .GROUP_BY('group by')
        .FROM('from')
        .SELECT('three', 'four')
        .FROM('another from')
        .WHERE('where')
        .ORDER_BY('third')
        .OUTER_JOIN('another outer join')
        # this isn't technically valid
        .WITH('first cte')
        .GROUP_BY('another group by')
        .HAVING('another having')
        .WITH(('fancy', 'second cte'))
        .JOIN('another join')
        .WHERE('another where')
        .NATURAL_JOIN('natural join')
        .SELECT()
        .SELECT_DISTINCT()
    )
    assert str(query) == dedent(
        """\
        WITH
            (
                first cte
            ),
            fancy AS (
                second cte
            )
        SELECT DISTINCT
            one,
            expr AS two,
            three,
            four
        FROM
            from,
            another from
        OUTER JOIN
            outer join
        JOIN
            join
        OUTER JOIN
            another outer join
        JOIN
            another join
        NATURAL JOIN
            natural join
        WHERE
            where AND
            another where
        GROUP BY
            group by,
            another group by
        HAVING
            having AND
            another having
        ORDER BY
            first,
            second,
            third
        LIMIT
            limit
        """
    )

The code so far: builder.py, test_builder.py.

More init #

One last feature: I'd like to reuse the formatting logic for paranthesized lists.

Good thing __init__ doesn't take any arguments yet:

    def __init__(self, data=None, separators=None):
        self.data = {}
        if data is None:
            data = dict.fromkeys(self.keywords, ())
        for keyword, args in data.items():
            self.data[keyword] = _FlagList()
            self.add(keyword, *args)

        if separators is not None:
            self.separators = separators

Using it looks like:

>>> print(Query({'(': ['one', 'two'], ')': ['']}, separators={'(': 'OR'}))
(
    one OR
    two
)

We could have required data to have the same structure as the attribute; however, it would be too verbose to use, and I'd have to do all the clean up myself; that's not very convenient. Instead, we make it mean "add() these strings for these keywords".⁴

We add a separate test for the fancy __init__.

def test_query_init():
    query = Query({'(': ['one', 'two', 'three'], ')': ['']}, {'(': 'OR'})
    assert str(query) == dedent(
        """\
        (
            one OR
            two OR
            three
        )

        """
    )

OK, now we're really done. We've spent 148 lines, or 101 statements.

The final version: builder.py, test_builder.py.

Note

Since the code in this article is essentially the reader one, but without type annotations, feel free to use it under the same BSD 3-Clause license.

That's it for now. :)

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Bonus: things that didn't make the cut #

When talking about trade-offs, I said we'll only add features as needed; this may seem a bit handwavy – how can I tell adding them won't make the code explode?

Because I did add them; that's what prototyping was for. But since they weren't actually used, I removed them – there's no point in them rotting away.

Here's how you'd go about implementing a few of them.

Insert / update / delete #

Make them flag keywords, to support the OR ABORT/FAIL/... variants.

To make VALUES bake in the parentheses, set its format to ({value}). That's to add one values tuple at a time.

To add one column at a time, we could do this:

allow add()ing INSERT with arbitrary flags
make INSERT('column', into='table') a synonym of add('INSERT INTO table', 'column')
classify INSERT and VALUES as parens_keywords – like subquery_keywords, but they apply once per keyword, not per value

It'd look like this:

# first insert sets flag
query.INSERT('one', into='table').VALUES(':one')
# later we just add stuff
query.INSERT('two').VALUES(':two')

Arbitrary strings as subqueries #

Allow setting add(..., is_subquery=True); you'd then do:

query.FROM('subquery', is_subquery=True)
query.FROM('not subquery')

Query objects as subqueries #

Using Query objects as subqueries without having to convert them explicitly to strings would allow changing them after being add()ed.

To do it, we just need to allow _Thing.value to be a Query, and override its is_subquery based on an isinstance() check.

Union / intersect / except #

This one goes a bit meta:

add a "virtual" COMPOUND keyword
add a new compound(keyword) method, which moves everything except WITH and ORDER BY to a subquery, and appends the subquery to data['COMPOUND'] with the appropriate fake keyword
make __getattr__ return a compound() partial for compound keywords
special-case COMPOUND in _lines()

The guaranteed insertion order was actually added to the language specification in 3.7, but in 3.6 both CPython and PyPy had it as an implementation detail. ^[return]
This works because False == 0 and True == 1, and is likely too clever. ^[return]
That's what I did initially. ^[return]
An alternate constructor or a subclass might have been a better choice here. We'll fix it if we need __init__ for something else. ¯\_(ツ)_/¯ ^[return]

namedtuple in a post-dataclasses world

2021-07-21T14:15:00+00:00

namedtuple has been around since forever,¹ and over time, its convenience saw it used far outside its originally intended purpose.

With dataclasses now covering part of those use cases, what should one use named tuples for?

In this article, we take a look at exactly that, with a few examples from real code.

Contents

What are named tuples used for?
The problems with named tuples
What are named tuples still good for?

What are named tuples used for? #

namedtuple exists in the standard library since Python 2.6, and allows building tuple subclasses that also have fields accessible by attribute lookup.

>>> from collections import namedtuple
>>> Point = namedtuple('Point', 'x y')

In general, this is useful when wrapping structured data; from the docs:

Named tuples are especially useful for assigning field names to result tuples returned by the csv or sqlite3 modules.

Because of how easy they are to define, named tuples have also been used for:

quick-and-dirty temporary data structures, more readable than plain tuples and regular classes (you get constructor keyword arguments and a __repr__ for free)
hashable instances (to use as dict keys or set members, or as arguments to functions decorated with e.g. functools.lru_cache)
immutable instances²

dataclasses was added in Python 3.7, and allows writing regular classes just as easily, by generating the required special methods. With frozen instances, it even covers hashable and immutable instances.

Before dataclasses, named tuples were used for the last three use cases because there were no other good alternatives in the standard library – you can do it with a normal class definition, but you have to write all the special methods by hand.

In case you've never used them, here's a comparison.

I'm using typing.NamedTuple because it looks similar to dataclasses; the result is the same as that of the traditional collections.namedtuple factory.

>>> class Point(NamedTuple):
...     x: int
...     y: int
...
>>> p = Point(1, y=2)
>>> p
Point(x=1, y=2)
>>> p.x
1
>>> p[0]
1
>>> list(p)
[1, 2]

>>> @dataclass
... class Point:
...     x: int
...     y: int
...
>>> p = Point(1, y=2)
>>> p
Point(x=1, y=2)
>>> p.x
1
>>> p[0]
Traceback (most recent call last):
  ...
TypeError: 'Point' object is not subscriptable
>>> list(p)
Traceback (most recent call last):
  ...
TypeError: 'Point' object is not iterable

The problems with named tuples #

PEP 557³ explains why sometimes namedtuple isn't good enough; in summary:

The instances are always iterable; this can make it difficult to add fields, because adding a new field will break code that uses unpacking.
- Also, if used as return value in a backwards-compatible API, it means the result must remain iterable/indexable forever, even if you later stop using namedtuple.
Instances can be accidentally compared with any other tuple.
There's no mutable version (in the standard library).
Fields can't be combined by inheritance.

What are named tuples still good for? #

With the drawbacks mentioned above, and with dataclasses covering a lot of their (maybe unintended) use cases, are named tuples good for anything anymore?

As you'd expect, the answer is yes.

The data is naturally a tuple #

Named tuples remain perfect for their originally intended purpose: ordered, structured data.

Some examples:

rows returned by a database query
the result of parsing a binary file format
pairs of things, like HTTP headers (a dict is not always appropriate, since the same header can appear more than once, and the order does matter in some cases)

Pairs of things are interesting, because both unpacking and attribute access are valid usage patterns.

For example, for my feed reader library I use a named tuple to model the result of a feed update, a (feed URL, update details or exception) pair.

This makes it easier to make sense of what a value means in interactive sessions or when debugging; compare the named and unnamed versions:

>>> result = next(reader.update_feeds_iter())
>>> result
UpdateResult(url='http://antirez.com/rss', value=None)
>>> tuple(result)
('http://antirez.com/rss', None)

Also, the distinct class allows having a docstring that users can look at with help(), and better semantics via properties (error/ok here).

You're already using a tuple #

You're already using a tuple, and want to make new code more readable: a namedtuple gets you this, but guarantees you won't break old code.

Some people argue that wherever you return a non-trivial tuple, you should be returning a namedtuple instead. I tend to agree.

You want consumers that do unpacking to fail #

In some cases, you want consumers that do unpacking to fail.

For example, in my feed reader library, I use a named tuple to group arguments related to filtering, because there's a lot of them, and they get passed around quite a bit before being used (I cover why in more detail here).

I know all arguments should always be handled, so I use unpacking specifically because I want the code to fail when a new one is added – if I used attribute access, the code would silently succeed. This is no substitute for tests, but the early warning is nice, especially in a larger code base.

Memory and speed #

Last, but not least, named tuples are useful if you care about memory or speed; they are much smaller and faster than the equivalent (data)class. In most cases, the difference doesn't matter, but it can become noticeable if you create millions of instances.

Setting __slots__ helps with memory, but doesn't really help with speed.

Here's a quick comparison:

	cls(1, 2)	obj.a	hash(obj)	size	total size
dataclass	846.9	49.7	361.7	152	320
dataclass + slots	709.1	45.5	342.5	48	104
namedtuple	465.3	43.2	99.6	56	112
dataobject + gc	150.3	43.1	104.1	48	48
dataobject	136.4	45.1	106.5	32	32

cls(1, 2), obj.a, hash(obj) are timings for that expression, in nanoseconds.

size is the sys.getsizeof of the object itself plus that of its __dict__ (if any), excluding the actual values. total size includes the values, as returned by Pympler's asizeof.

I ran this with 64-bit CPython 3.8 on macOS; Linux looks roughly the same.

When increasing the number of fields, obj.a remains constant, while the other timings increase proportionally. The slots dataclass is always 8 bytes smaller than the namedtuple.

For the dataobject rows I used recordclass, which provides dataclass/namedtuple-equivalent types. The version without gc doesn't participate in cyclic garbage collection, so it shouldn't be used for recursive data structures.

The library still has some rough edges, though: the documentation is a bit confusing, and I had to use the (yet unreleased) 0.15 version to get it working; also, note the wrong total size (it may be a Pympler bug). Nevertheless, the numbers are pretty compelling, and if you have this problem, it's definitely worth a look.

The class definitions:

For dataclasses, __slots__ must be set explicitly; this was fixed in Python 3.10.

from typing import NamedTuple
from dataclasses import dataclass
from recordclass import dataobject

class NT(NamedTuple):
    a: int
    b: int

@dataclass(frozen=True)
class DC:
    a: int
    b: int

@dataclass(frozen=True)
class DS:
    a: int
    b: int
    __slots__ = ('a', 'b')

class DO(dataobject):
    a: int
    b: int
    __options__ = dict(readonly=True, fast_new=True)

class DG(dataobject):
    a: int
    b: int
    __options__ = dict(readonly=True, fast_new=True, gc=True)

That's it for now. :)

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

2007–2008 seems like forever enough these days. ^[return]
See this for an example of why you might want immutable instances. ^[return]
I also cover PEP 557 and what can be learned from it here. ^[return]

reader 2.0 released

2021-07-19T14:12:00+00:00

Hi there!

I'm happy to announce version 2.0 of reader, a Python feed reader library.

This release brings you a cleaner API, more consistently named methods and attributes, timezone-aware datetimes, and safer defaults. See the changelog for details.

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

retrieve, store, and manage Atom, RSS, and JSON feeds
mark entries as read or important
add tags and metadata to feeds
filter feeds and articles
full-text search articles
write plugins to extend its functionality

...all these with:

a high-level, stable, clearly documented API
excellent test coverage
fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

never knew where to start?
it seemed like too much work?
you don't like writing backend code?

Are you already working with feedparser, but:

want an easier way to store, filter, sort and search feeds and entries?
want to get back type-annotated objects instead of dicts?
want to restrict or deny file-system access?
want to change the way feeds are retrieved by using Requests?
want to also support JSON Feed?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

Why make your own feed reader? #

So you can:

have full control over your data
control what features it has or doesn't have
decide how much you pay for it
make sure it doesn't get closed while you're still using it
really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

Why I wrote my own SQL query builder (in Python)

2021-07-13T14:48:00+00:00

Previously

This is the third article in a series about writing an SQL query builder in 150 lines of Python.

Today, we'll talk about:

why I decided to write my own
the alternatives I considered
why I didn't use an existing library
how I knew it wouldn't become a maintenance burden

The first part is in rough chronological order. If you're only interested in the libraries I looked at, you can find the list at the end.

Contents

Background
The first prototype
Requirements, and existing libraries
The second prototype
Deciding to use my own
Other alternatives

Background #

reader is a Python feed reader library – it allows users to retrieve, store, and manage web feeds through a high-level API, without having to deal with feed-related details.

It is a hobby and learning project, and I can only spend a limited amount of time on it, sometimes quite far in between. It's not necessarily about learning technologies; rather, it is about library design, writing code that's maintainable long-term, and showing restraint – if I were to make the best library I could, what would it look like?

Note

Because of this, the conclusions of this article, if any, may not be directly applicable to regular "work" projects. With reader, I have different constraints over different time scales, a somewhat different definition of success, and more freedom in some aspects.

However, not all projects are the same, and not all parts of a project are the same. Sometimes, this kind of long-term thinking can be useful, and it can actually be achieved through a combination of planning, strategical technical debt, and saying no to, reducing the scope of, or postponing features.

In the spirit of keeping things maintainable, the core library is fully typed and has 100% test coverage, to make refactoring straightforward (if not painless).

Also, almost from the start, I put all storage code in a single module, behind a data access object; the rest of the library doesn't even know it's talking to a database.

One of reader's main features is filtering articles by article metadata, user-set metadata (read, important, feed tags), and full-text search; the results can be sorted in various ways and paginated.

In May 2019, with less than half of the above implemented, the function building the SQL query was over 100 lines, and I had already felt the need to add this comment:

This needs some sort of query builder so badly.

The first prototype #

I opened an issue, and did some research.

At some point I stumbled upon sqlbuilder.mini, which was built around an interesting insight – queries can be represented as plain data structures:

>>> sql = [
...     'SELECT', ['first_name'],
...     'FROM', ['author'],
...     'WHERE', ['status', '==', P('new')],
... ]
>>> compile(sql)
('SELECT first_name FROM author WHERE status == %s', ['new'])

You can then modify the query directly:

>>> sql[sql.index('SELECT') + 1].append('last_name')
>>> compile(sql)
('SELECT first_name, last_name FROM author WHERE status == %s', ['new'])

Or via a wrapper that simplifies navigation:

>>> sql = Q(sql)
>>> sql.append_child(
...     ['SELECT'],  # path
...     ['age']
... )  # returns self to allow method chaining
<sqlbuilder.mini.Q object>
>>> compile(sql)
('SELECT first_name, last_name, age FROM author WHERE status == %s', ['new'])

I really liked how simple and flexible this is, and the choice of not dealing with SQL correctness or dialects – a middle ground between building strings by hand and "proper" query builders. On the other hand, it seemed too verbose (even with the wrapper), and the generated SQL wasn't very readable.

Surely, it would be possible to make this look like sql.SELECT('age'), right?

So I made a prototype – with no real intention of using it, just to see how easy it is to do. The core was quite short, about 80 lines; my thoughts at the time:

The end result looks nice, but using it would add ~150 lines of code (that need to be tested), and it's less useful for simpler queries.

Also, it looks nice now, when I just wrote it; 6 months from now it may be hard to understand.

Afraid I'm too happy with it, and with my curiosity satisfied, I did just that: postponed for six months.

Requirements, and existing libraries #

My main concern with building my own was that over time, with additions and fixes, the effort would be greater than that of getting an existing library to do what I needed.

I did two things to deal with this.

First, I wrote down detailed requirements.

Whatever I used had to support the following:

SELECT with conditional WHERE, ORDER BY, JOIN etc. (example)
common table expressions (WITH)
scrolling window queries (or be possible to build on top)
arbitrary SQL (so I don't have to use the query builder for everything)
the order in which you add clauses shouldn't matter

Also, it should be easy to understand and maintain, and it should be possible to support additional SQL features.

Because reader is a library, I wanted to keep the number of (transitive) dependencies as small as possible, since any extra dependency gets passed down to the users.

Both to keep things simple and due to historical reasons, I did not want to switch to an abstraction layer like SQLAlchemy Core just for query building – I needed (and still need) only SQLite support, and already had code to deal with stuff like migrations.

Second, I did a slightly more serious survey of existing libraries.

I didn't feel any of the ones I looked at was ideal, for at least one of these reasons:

They came with a full abstraction layer. This isn't bad in itself, but meant I had to switch everything, eventually – using a mix would make things worse.
They had too many features. Usually this is good, but it means there's more of everything: more features, more documentation to go through, more concepts to keep in your head, more things contributors need to know or learn.
They didn't make things more readable or more composable.
They weren't actively maintained.

Again, I chose to wait until more features are implemented.

The second prototype #

By May 2020, most of the features were implemented. The function building the query was 150 lines, with part of it duplicated for search. At some point, I tried to optimize it to use indexes, but gave up because trying things simply took too long.

So, a full year later, I made the prototype support all the required features and a few extra (UNION, nested queries), and tried it out on the full real queries.

It didn't take all that long, and it remained around 100 lines.

Deciding to use my own #

At this point, most of the work was already done, and integrating it took less than an hour.

Excluding the 136 lines of the builder itself with scrolling window query support, the code went from 1400 to 1300 lines. I took that as a win, since for the price of 36 lines I was able to reuse the filtering logic. (Now, one year later, it enabled a lot more reuse, without growing significantly.)

I ended up keeping it, because:

Using an existing library would take too much effort. (I'll reconsider when the requirements change, though.)
It is tiny, which makes it relatively easy to understand and modify; the prototypes made me quite confident it's likely to stay that way. Because it is only used internally, I can leave out a lot of nice things that aren't actually needed (including the extra features).
It has 0 dependencies. That's even better than 1.
reader already had great test coverage, so little additional testing was required.

Attention

My query builder is not directly comparable with that of an ORM. Instead, it is an alternative to building plain SQL strings by hand.

Since I was coming from plain SQL, I was already doing all of this.

Other alternatives #

Here's a non-exhaustive list of other things I looked at. I'm only covering the libraries I actually considered using, or that are interesting in some way. There are others out there; some aren't actively maintained, some I simply missed.

Update (July 2021)

I added a few more interesting libraries.

Do nothing #

It's worth keeping in mind that "do nothing" is always an option – probably the first option to consider, in many cases.

There's two kinds of doing nothing: passive, where you wait for new requirements to come up – for the problem to reveal itself –, and active, where you explore options, but don't commit to anything just yet.

I ended up doing both, to a point.

Disable parts of the query #

An interesting (but quickly abandoned) idea was to not build queries; instead, have just one query, and disable parts of it with boolean or optional parameters, and hope the query planner optimizes it:

SELECT ...
FROM entries
WHERE
    (:read IS NOT NULL AND entries.read = :read) AND
    ...  -- 7 more expressions like this

There are two huge issues:

I'm not sure any optimizer is that smart (also, the query might be optimized before the parameters are passed in). Even if it were, I'm not smart enough to design indexes for a query like this.
It doesn't seem possible to do it for JOIN, different ORDER BY terms, or even an arbitrary number of WHERE conditions (e.g. for tags).

Even if they were all possible, the result would be almost impossible to understand.

SQLAlchemy Core, Peewee query builder #

SQLAlchemy and Peewee are both SQL toolkits / object-relational mappers.

SQLAlchemy has over 15 years of history, and is the database toolkit for Python. Hell, there's even an Architecture of Open Source Applications chapter about it.

Peewee is a bit younger (~10 years), simple and small by design.

Both have a lot of extensions¹, and can be used without the ORM part; Peewee can even generate plain SQL without defining models.

In the end, both seemed too complicated, and meant I had to switch to them eventually, adding the burden of researching a use case I don't have yet. However, if I ever need multi-database support, it's likely I'll use one of them.²

SQLBuilder, PyPika, python-sql #

SQLBuilder and PyPika are standalone query builders – no ORM, no connection management, just SQL generation; they are similar to the Peewee query builder.

SQLBuilder doesn't seem actively maintained. Aside from that, I didn't use it because it would make a potential migration to SQLAlchemy or Peewee more difficult.

PyPika I discovered while writing this article; it is actively maintained and has somewhat better documentation.

(update) Another actively maintained query builder is python-sql. It is part of Tryton, an open-source ERP platform; it's been around for a while, and will likely continue to be. I missed this one during my research :)

sqlbuilder.mini, psycopg2.sql #

SQLBuilder comes with another, extremely lightweight SQL builder, sqlbuilder.mini.

As dicussed in the beginning, I like the overall approach (and at ~500 lines, it's small enough to vendor), but it still seems verbose, and the generated SQL isn't very readable.

(update) psycopg2.sql has a similar philosophy in how it treats SQL strings. Unlike my builder, it's "inside-out" (you append stuff to lists explicitly), so it's more verbose. It does support escaping identifiers and placeholders, though; I didn't really deal with that in any way.

JinjaSQL, SQLpy, PugSQL #

These two are interesting because they use templating.

JinjaSQL is exactly what you'd expect: generate SQL from Jinja templates. I didn't use it because composition through macros would still be verbose, and a bit tricky (careful with the comma after that last column).

(update) Template engines like Jinja are useful if you need to allow end users to build queries, since you can sandbox templates (sandboxing Python is not easy, at least not with CPython). dbt, brought to my attention by a reader, seems to be using Jinja in this way.

SQLpy is similar, but different. You put your named queries in a separate file, and access them from Python as functions. Query building happens via named parameters: if you don't pass a parameter when executing the query, the lines using that parameter aren't included in the query (as you'd expect, this comes with a lot of caveats).

(update) I also re-discovered PugSQL, which is like SQLpy, but is actively maintained and doesn't do the magic line disappearing stuff. Turns out they're both inspired by a pair of Clojure libraries.

If your SQL doesn't change, but is parametrized, PugSQL looks like a good lightweight solution. For me, adding WHERE conditions was a strong requirement.

Pony #

I don't think I considered Pony at the time, but it's worth mentioning: it has been around since 2012, is actively maintained, and has commercial support.

And it can translate generator expressions like this one into SQL queries:

select(c for c in Customer if sum(c.orders.price) > 1000)

For reader it is definitely overkill.

It does look really, really interesting, though (too interesting, maybe?).

That's it for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

Next: Write an SQL query builder in 150 lines Python!

That's one of the benefits of using libraries that have been around for a while. Some extensions relevant to my project: SQLAlchemy has Alembic for migrations (from the same author) and sqlakeyset for scrolling window queries; Peewee has a lot of SQLite-specific functionality. ^[return]
They'd also be my first choice for a project with resources and deadlines. ^[return]

When your functions take the same arguments, consider using a class: counter-examples

2021-06-18T15:02:00+00:00

In a previous article, I talk about this heuristic for using classes in Python:

If you have functions that take the same set of arguments, consider using a class.

Thing is, heuristics don't always work.

To make the most out of them, it helps to know what the exceptions are.

So, let's look at a few real-world examples where functions taking the same arguments don't necessarily make a class.

Counter-example: two sets of arguments #

Consider the following scenario:

We have a feed reader web application. It shows a list of feeds and a list of entries (articles), filtered in various ways.

Because we want to do the same thing from the command-line, we pull database-specific logic into functions in a separate module. The functions take a database connection and other arguments, query the database, and return the results.

def get_entries(db, feed=None, read=None, important=None): ...
def get_entry_counts(db, feed=None, read=None, important=None): ...
def search_entries(db, query, feed=None, read=None, important=None): ...
def get_feeds(db): ...

The main usage pattern is: at the start of the program, connect to the database; depending on user input, repeatedly call the functions with the same connection, but different options.

Taking the heuristic to the extreme, we end up with this:

class Storage:

    def __init__(self, db, feed=None, read=None, important=None):
        self._db = db
        self._feed = feed
        self._read = read
        self._important = important

    def get_entries(self): ...
    def get_entry_counts(self): ...
    def search_entries(self, query): ...
    def get_feeds(self): ...

This is not very useful: every time we change the options, we need to create a new Storage object (or worse, have a single one and change its attributes). Also, get_feeds() doesn't even use them – but somehow leaving it out seems just as bad.

What's missing is a bit of nuance: there isn't one set of arguments, there are two, and one of them changes more often than the other.

Let's take care of the obvious one first.

The database connection changes least often, so it makes sense to keep it on the storage, and pass a storage object around:

class Storage:

    def __init__(self, db):
        self._db = db

    def get_entries(self, feed=None, read=None, important=None): ...
    def get_entry_counts(self, feed=None, read=None, important=None): ...
    def search_entries(self, query, feed=None, read=None, important=None): ...
    def get_feeds(self): ...

The most important benefit of this is that it abstracts the database from the code using it, allowing you to have more than one kind of storage.

Want to store entries as files on disk? Write a FileStorage class that reads them from there. Want to test your application with various combinations of made-up entries? Write a MockStorage class that keeps the entries in a list, in memory. Whoever calls get_entries() or search_entries() doesn't have to know or care where the entries are coming from or how the search is implemented.

Tip

This is the data access object design pattern. In object-oriented programming terminology, a DAO provides an abstract interface that encapsulates a persistence mechanism.

OK, the above looks just about right to me – I wouldn't really change anything else.

Some arguments are still repeating, but it's useful repetition: once a user learns to filter entries with one method, they can do it with any of them. Also, people use different arguments at different times; from their perspective, it's not really repetition.

And anyway, we're already using a class...

Counter-example: data classes #

Let's add more requirements.

There's more functionality beyond storing things, and we have multiple users for that as well (web app, CLI, someone using our code as a library). So we leave Storage to do only storage, and wrap it in a Reader object that has a storage:

class Reader:

    def __init__(self, storage):
        self._storage = storage

    def get_entries(self, feed=None, read=None, important=None):
        return self._storage.get_entries(feed=feed, read=read, important=important)

    ...

    def update_feeds(self):
        # calls various storage methods multiple times:
        # get feeds to be retrieved from storage,
        # store new/modified entries
        ...

Now, the main caller of Storage.get_entries() is Reader.get_entries(). Furthermore, the filter arguments are rarely used directly by storage methods, most of the time they're passed to helper functions:

class Storage:

    def get_entries(self, feed=None, read=None, important=None):
        query = make_get_entries_query(feed=feed, read=read, important=important)
        ...

Problem: When we add a new entry filter option, we have to change the Reader methods, the Storage methods, and the helpers. And it's likely we'll do so in the future.

Solution: Group the arguments in a class that contains only data.

from typing import NamedTuple, Optional

class EntryFilterOptions(NamedTuple):
    feed: Optional[str] = None
    read: Optional[bool] = None
    important: Optional[bool] = None

class Storage:

    ...

    def get_entries(self, filter_options):
        query = make_get_entries_query(filter_options)
        ...

    def get_entry_counts(self, filter_options): ...
    def search_entries(self, query, filter_options): ...
    def get_feeds(self): ...

Now, regardless of how much they're passed around, there are only two places where it matters what the options are:

in a Reader method, which builds the EntryFilterOptions object
where they get used, either a helper or a Storage method

Note that while we're using the Python class syntax, EntryFilterOptions is not a class in the traditional object-oriented programming sense, since it has no behavior.¹ Sometimes, these are known as "passive data structures" or "plain old data".

Tip

A plain class or a dataclass would have been a decent choice as well; why I chose a named tuple is a discussion for another article.

Tip

I used type hints because it's a cheap way of documenting the options, but you don't have to, not even for dataclasses.

The example above is a simplified version of the code in my feed reader library. In the real world, EntryFilterOptions has more options (with more on the way), and the Reader and Storage get_entries() are a bit more complicated.

Another real-world example of this pattern is Requests:

get(), post() and so on end up calling Session.request()
which packages the arguments into a Request
and turns it into a PreparedRequest
which is finally sent by an HTTPAdapter

That's pretty much it for now – hang around for some extra stuff, though ;)

I hope I managed to add more nuance to the original article, and that you're now at least a little bit better equipped to use classes. Keep in mind that this is more an art than a science, and that you can always change your mind later.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

If you've made it this far, you might like:

Caching a lot of methods in Python

Bonus: other alternatives #

Still here? Cool!

Let's look at some of the other options I considered, and why I didn't go that way.

Why not a dict? #

Instead of defining a whole new class, we could've used a dict:

{'feed': ..., 'read': ..., 'important': ...}

But this has a number of drawbacks:

Dicts are not type-checked. TypedDict helps, but doesn't prevent using the wrong keys at runtime.
Dicts break code completion. TypedDict may help with smarter tools like PyCharm, but doesn't in interactive mode or IPython.
Dicts are mutable. For our use case, immutability is a plus: the options don't have much reason to change, so it's useful to disallow it.

Why not **kwargs? #

Why not pass **kwargs directly to EntryFilterOptions?

class Reader:
    def get_entries(self, **kwargs):
        return self._storage.get_entries(**kwargs)

Because:

It also breaks code completion.
It makes the code less self-documenting: you don't know what arguments get_entries() takes, even if you read the source. Presumably, they're in the docstring, but not everybody writes one all the time.
If we introduce another options object (say, for pagination), we still have to write code to split the kwargs between the two.

Why not EntryFilterOptions? #

Why not take an EntryFilterOptions directly, then?

from reader import make_reader, EntryFilterOptions
reader = make_reader(...)
options = EntryFilterOptions(read=True)
entries = reader.get_entries(options)

Because it makes things verbose for the user: they have to import EntryFilterOptions, and build and pass one to get_entries() for every call. That's not very friendly.

The Reader and Storage method signatures differ because they're used differently:

Reader methods are mostly called by external users in many ways
Storage methods are mostly called by internal users (Reader) in a few ways

Ted Kaminski discusses this distinction in more detail in Data, objects, and how we're railroaded into poor design. ^[return]

Python sentinel objects, type hints, and PEP 661

2021-06-10T11:42:00+00:00

PEP 661 "Sentinel Values" recently brought to attention the sentinel object pattern.¹

While by no means new², this time the pattern appears in the context of typing, so it's worth taking a look at how the two interact.

Contents

What's a sentinel, and why do I need one?
- Real world examples
- Non-private sentinels
What's this got to do with typing?
What's with PEP 661?
- How does this affect me?
- Is this worth a PEP?

What's a sentinel, and why do I need one? #

The PEP 661 abstract summarizes it best:

Unique placeholder values, widely known as "sentinel values", are useful in Python programs for several things, such as default values for function arguments where None is a valid input value.

The simplest use case I can think of is a function that returns a default value only if explicitly provided, otherwise raises an exception.

The next() built-in function is a good example:

next(iterator[, default])

Retrieve the next item from the iterator by calling its __next__() method. If default is given, it is returned if the iterator is exhausted, otherwise StopIteration is raised.

Given this definition, let's try to re-implement it.

next() essentially has two signatures³:

next(iterator) -> item or raise exception
next(iterator, default) -> item or default

There are two main ways to write a function that supports both:

next(*args, **kwargs); you have to extract iterator and default from args and kwargs, and raise TypeError if there are too many / too few / unexpected arguments
next(iterator, default=None); Python checks the arguments, you just need to check if default is None

To me, the second seems easier to implement than the first.

But the second version has a problem: for some users, None is a valid default – how can next() distinguish between raise-exception-None and default-value-None?

In your own code, you may be able to guarantee None is never a valid value, making this a non-issue.

In a library, however, you don't want to restrict users in this way, since you usually can't foresee all their use cases. Even if you did choose to restrict valid values like this, you'd have to document it, and the users would have to learn about it, and always remember the exception.⁴

Here's where a private, internal-use only sentinel object helps:

_missing = object()

def next(iterator, default=_missing):
    try:
        return iterator.__next__()
    except StopIteration:
        if default is _missing:
            raise
        return default

Example output:

>>> it = iter([1])
>>> print(next(it, None))
1
>>> print(next(it, None))
None
>>> print(next(it))
Traceback (most recent call last):
  ...
StopIteration

Now, next() knows that default=_missing means raise exception, and default=None is just a regular default value to be returned.

You can think of _missing as of another None, for when the actual None is already taken – a "higher-order" None. Because it's private to the module, users can never (accidentally) use it as a default value, and never have know about it.

Tip

For a more in-depth explanation of sentinel objects and related patterns, see The Sentinel Object Pattern by Brandon Rhodes.

Real world examples #

The real next() doesn't actually use sentinel values, because it's implemented in C, and things are sometimes different there.

But there are plenty of examples in pure-Python code:

The dataclasses module has two.

The docs even explain what a sentinel is:

[...] the MISSING value is a sentinel object used to detect if the default and default_factory parameters are provided. This sentinel is used because None is a valid value for default. No code should directly use the MISSING value.

(The other one is used in the __init__ of the generated classes to show a default value comes from a factory.)
attrs also has two. One of them (analogous to dataclasses.MISSING) is even included in the API documentation.
Werkzeug has one.
I have one in my feed reader library (originally stolen from Werkzeug). I use it for methods like get_feed(feed[, default]), which either raises FeedNotFoundError or returns default.

Non-private sentinels #

I mentioned before sentinels are private; that's not always the case.

If the sentinel is the default argument of a public method or function, it may be a good idea to expose / document it, to facilitate inheritance and function wrappers.⁵ attrs is a good example of this.

(If you don't expose it, people can still extend your code by using their own sentinel, and then calling either form of your function.)

What's this got to do with typing? #

Let's try to add type hints to our hand-rolled next():

from typing import overload, TypeVar, Union, Iterator

T = TypeVar('T')
U = TypeVar('U')


# We define MissingType in one of two ways:

class MissingType: pass
# MissingType = object

# The second one is equivalent to the original
# `_missing = object()`, but the alias allows us
# to keep the same type annotations.

_missing = MissingType()


# As mentioned before, next() is actually two functions;
# typing.overload allows us to express this.
#
# One that returns an item or raises an exception:

@overload
def next(iterator: Iterator[T]) -> T: ...

# ... and one that takes a default value (of some type U),
# and returns either an item, or that default value
# (of the *same* type U):

@overload
def next(iterator: Iterator[T], default: U) -> Union[T, U]: ...

# The implementation takes all the arguments,
# and returns a union of all the types:

def next(
    iterator: Iterator[T],
    default: Union[MissingType, U] = _missing
) -> Union[T, U]:
    try:
        return iterator.__next__()
    except StopIteration:

        # "if default is _missing" is idiomatic here,
        # but Mypy doesn't understand it
        # ("var is None" is a special case).
        # It does understand isinstance(), though:
        # https://mypy.readthedocs.io/en/stable/casts.html#casts

        if isinstance(default, MissingType):
            # If MissingType is `object`, this is always true,
            # since all types are a subclass of `object`.
            raise

        return default

The isinstance() thing at the end is why a plain object() sentinel doesn't work – you can't (easily) get Mypy to treat your own "constants" the way it does a built-in constant like None, and the sentinel doesn't have a distinct type.

Also, if you use the MissingType = object version, Mypy complains:

next.py:37: error: Overloaded function implementation cannot produce return type of signature 2

If you're wondering if the good version actually worked, here's what Mypy says:

it = iter([1, 2])

one = next(it)
reveal_type(one)
# next.py:62: note: Revealed type is 'builtins.int*'

two = next(it, 'a string')
reveal_type(two)
# next.py:66: note: Revealed type is 'Union[builtins.int*, builtins.str*]'

What's with PEP 661? #

There are many sentinel implementations out there; there are 15 different ones in the standard library alone.

Many of them have at least one of these issues:

non-descriptive / too long repr() (e.g. <object object at 0x7f99a355fc20>)
don't pickle correctly (e.g. after unpickling you get a different, new object)
don't work well with typing

Thus, PEP 661 "suggests adding a utility for defining sentinel values, to be used in the stdlib and made publicly available as part of the stdlib". It looks like this:

>>> NotGiven = sentinel('NotGiven')
>>> NotGiven
<NotGiven>
>>> MISSING = sentinel('MISSING', repr='mymodule.MISSING')
>>> MISSING
mymodule.MISSING

This utility would address all the known issues, saving developers (mostly, stdlib and third party library authors) from reinventing the wheel (again).

How does this affect me? #

Not at all.

If the PEP gets accepted and implemented, you'll be able to create an issue-free sentinel with one line of code.

Of course, you can keep using your own sentinel objects if you want to; the PEP doesn't even propose to change the existing sentinels in the standard library.

Is this worth a PEP? #

PEPs exist to support discussions in cases where the "correct" way to go isn't obvious, consensus or coordination are required, or the changes have a big blast radius. A lot of PEPs get abandoned or rejected (that's fine, it's how the process is supposed to work).

PEP 661 seems to fall under the "requires consensus" category; it follows a community poll where although the top pick was "do nothing", most voters went for "do something" (but with no clear agreement on what that should be).

The poll introduction states:

This is a minor detail, so ISTM most important that we reach a reasonable decision quickly, even if that decision is that nothing should be done.

It's worth remembering that doing nothing is always an option. :)

If you're into this kind of thing, I highly recommend going through the poll thread and the (ongoing) PEP discussion thread – usually, these discussions are API design master classes.

That's all I have for now.

Learned something new today? Share it with others, it really helps! PyCoder's Weekly HN Bluesky linkedin Twitter

The PEP is still in draft status as of 2021-06-10. ^[return]
Here's a 2008 article about it. ^[return]
While Python doesn't support overloading, sometimes it's useful to think about functions in this way. ^[return]
The same applies to using some other "common" value, for example, a "<NotGiven>" string sentinel.

For immutable values like strings, it's probably worse. Because of optimizations like interning, strings constructed at different times may actually result in the same object. The data model specifically allows for this to happen (emphasis mine):

Types affect almost all aspects of object behavior. Even the importance of object identity is affected in some sense: for immutable types, operations that compute new values may actually return a reference to any existing object with the same type and value, while for mutable objects this is not allowed.

^[return]
Thanks to u/energybased for reminding me of this! ^[return]

When to use classes in Python? When your functions take the same arguments

2021-06-04T12:35:00+00:00