Python on Jeff Triplett's Micro.blog

How I Use django-simple-nav for Dashboards, Command Palettes, and More

Thu, 26 Feb 2026 20:56:51 -0600

I first got exposed to django-simple-nav while working with Josh Thomas at the Westervelt Company over the last two or three years. It quickly became a go-to library in my toolkit. django-simple-nav lets you define nav items and groupings in Python, then hand them off to a Django template to render. I use it for sidebars, headers, dashboards, and other spots where I need a menu.

Since then, I have also started using it on a lot of personal projects. It has been a great fit every time.

Defining nav items once and reusing them everywhere

Recently I wanted to reuse parts of a menu nav without having to update a bunch of different files. I had a dashboard with a sidebar full of links, and I also wanted a command palette that could search across all those same options when you hit Command-K from any page. django-simple-nav was a natural fit because you define your items once in Python and render them with whatever template you want.

Here is an example of what that looks like in Python:

from django_simple_nav.nav import Nav
from django_simple_nav.nav import NavGroup
from django_simple_nav.nav import NavItem

NAV_ITEMS = [
    NavItem(title="Home", url="home"),
    NavItem(title="Dashboard", url="dashboard"),
    NavGroup(
        title="Admin & Tools",
        items=[
            NavItem(title="Admin", url="admin:index", permissions=["is_staff"]),
        ],
    ),
    NavItem(title="Login", url="login"),
    NavItem(title="Logout", url="logout"),
]


class DashboardNav(Nav):
    template_name = "nav/dashboard.html"
    items = NAV_ITEMS


class CommandPaletteNav(Nav):
    template_name = "nav/command_palette.html"
    items = NAV_ITEMS

Flexible URLs and named routes

A few things to notice here. When you create a nav item and you have a URL, you can either use the URL’s name or you can give it the path to it and it will figure out what it is. So if I want a nav item for the Django admin, I can use "admin:index" as the URL.

Handling permissions without template logic

Another nice feature is the permissions parameter. The Admin link uses permissions=["is_staff"], which means it only shows up for staff users. You can use this to hide things based on whether someone is staff, a superuser, or has specific Django permissions. This makes it nice to keep all of your nav options in one place without having to write special conditional logic in your templates to show or hide items for different users.

Rendering a dashboard nav with Django templates

The other thing to notice is the two Nav classes at the bottom: DashboardNav and CommandPaletteNav. They both use the same NAV_ITEMS list but point to different templates. This is where the reuse really shines.

Here is an example nav template at templates/nav/dashboard.html:

<nav>
  {% for item in items %}
    {% if item.items %}
      <span>{{ item.title }}:
        {% for subitem in item.items %}
          <a href="{{ subitem.url }}"{% if subitem.active %} class="active"{% endif %}>
            {{ subitem.title }}
          </a>
        {% endfor %}
      </span>
    {% else %}
      <a href="{{ item.url }}"{% if item.active %} class="active"{% endif %}>
        {{ item.title }}
      </a>
    {% endif %}
  {% endfor %}
</nav>

Then in your base template, you can render it with the template tag:

{% load django_simple_nav %}

{% django_simple_nav "myapp.nav.DashboardNav" %}

Building a command palette from the same nav items

Because the CommandPaletteNav class uses the same NAV_ITEMS list but with a different template, we can render the same nav data as JSON for a command palette. Here is an example template at templates/nav/command_palette.html:

{% load django_simple_nav %}

<script type="application/json" id="command-palette-data">
[{% for item in items %}{% if item.items %}{% for subitem in item.items %}
  {"title": "{{ subitem.title }}", "url": "{{ subitem.url }}", "group": "{{ item.title }}"}{% if not forloop.last or not forloop.parentloop.last %},{% endif %}{% endfor %}{% else %}
  {"title": "{{ item.title }}", "url": "{{ item.url }}"}{% if not forloop.last %},{% endif %}{% endif %}{% endfor %}
]
</script>

This also lent itself well when I was trying to do something quickly with Claude. I don’t know how to build a command palette, but I was able to just have Claude create one. I asked it to use HTMX to keep everything pretty lightweight.

Final thoughts

All in all, django-simple-nav is a nice library to have in your toolkit. I like using it for headers, menus, dashboard navigation links, and even footer text. If you need flexible, reusable navigation in Django, give it a look.

Written by Jeff. Edited with Grammarly and Claude Code.

📅 Office Hours for November

Thu, 13 Nov 2025 17:08:22 -0600

I’m hosting office hours on the next two Fridays in November.

Office Hours Schedule

Friday, November 14, 2025, 2:30 pm to 4:30 pm CT
- Starting at 2pm CT, I’ll be working on some community/open source projects if anyone wants to see Claude Code or Claude Code on the web in action
Friday, November 21, 2025, 2:30 pm to 4:30 pm CT
🦃 Friday, November 28th - No office hours due to US Thanksgiving

Join Us

These sessions are open to anyone, especially those working remotely who wants to connect with others. It’s a collaborative space for discussions, catch-ups, or just hanging out while you work.

While we have a bunch of Python and Django developers who regularly attend, anyone is welcome to join even if you don’t use either of these things.

Whether you’re a first-time attendee or a regular, you’re welcome to drop in anytime during the session. No need to stay for the full two hours, come and go as your schedule allows.

How to Join

The Zoom link is available in the gist from our previous sessions. Feel free to reach out if you need the details.

ℹ️ Meeting Guidelines

🔕 Please mute when someone else is talking 👈
👍 Be respectful to other people and mute when not talking
👍 Be mindful of the spirit of the Python Code of Conduct
👍 Be mindful and respectful of everyone’s time.
👍 Join because you want to work on something with a small group
⚠️ No marketing your product or service
⚠️ No meeting/recording bots. Humans are welcome. Meeting bots are not.
⚠️ Mute!

See you there!

Default Apps 2025

Wed, 29 Oct 2025 17:55:03 -0600

Here are my Default Apps 2025, which builds from my Default Apps 2024 and Default Apps 2023 posts.

🤖 AI Coding Assistant: Claude Code + GitHub Copilot CLI + Codex
⚙️ Automation: Hammerspoon
🎒 Backups: Backblaze
🔖 Bookmarks: Raindrop.io
🌐 Browser: Vivaldi
📆 Calendar: Apple Calendar
💬 Chat: Apple Messages (family & friends), Discord (friends), Slack (work and community work), Telegram (weird bots)
📁 Cloud File Storage: iCloud Drive + Syncthing
🧑‍💻 Code Editor: Sublime Text + Zed
🤖 Containers: OrbStack + Docker Compose
✍️ Grammar: Grammarly
🚀 Launcher: Alfred 5
📨 Mail Client: Mimestream (Gmail only and not very often)
📮 Mail Server: Fastmail (I pay for it but still don’t use it enough) + Gmail
👔 Menu Bar: Ice
🎵 Music: Apple Music + Spotify
🔐 Password Management: 1Password
🎤 Podcasts: Apple Podcast
📑 Read It Later: Raindrop.io
🔗 Remote Access: Tailscale
🗓️ Scheduling + Booking: Cal.com
🖼️ Screenshots: Xnapper
🔍 Search: Kagi
🐚 Terminal: iTerm2 + Ghostty
🌐 Terminal Proxy: VibeTunnel
🎙️ Transcription: MacWhisper
🗃️ Version Control: Tower
📹 Video Calls: Zoom + Cal.com + Meeter
✈️ VPN: Mullvad
📝 Writing: Obsidian

Commentary

The biggest changes for me this year were the addition of more AI tooling like coding assistants, VibeTunnel, and the MacWhisper voice translating app. A year ago, I was writing my own AI coding tools with Python. Today, I use voice-to-text to interact with multiple coding assistants that I can access from any browser connected to my Tailscale Tailnet. It’s a mouthful to explain, but coding looks very different now than it did a year or two ago for me.

I’m not using Google Apps as much as I was a few years ago, though I still need to switch between Google Orgs for work and community organizations.

Changes from 2024

New Additions:

AI Coding Assistant (Claude Code, GitHub Copilot CLI, Codex)
VibeTunnel for accessing my terminal through any browser on my Tailnet
I started using Ghostty to run my AI sessions, but I find its lack of find (⌘f) support to be terribly frustrating.
I have used MacWhisper for years, but now it’s a daily driver. I can hold down the globe (fn) key on my keyboard and talk to my computer and quickly reply to people or even write blog posts with my voice. It’s incredibly cool and powerful.

Apps Removed:

I dropped Polypane (Browser category)
I managed to not use Overcast (Podcasts category) for most of the year, but I’m still not happy with Apple’s Podcast app. I might be switching back.

Previously inspired by: “Apps I’ve been using regularly this year.” Heavily inspired by Matt Stein and cataloged by Robb Knight’s App Defaults

🤖 On GitHub Copilot CLI and prompts as code

Sat, 04 Oct 2025 19:57:19 -0600

I checked out William Vincent’s The Secret Prompts in GitHub Copilot CLI tonight, and I wanted to share a few tips and what stood out to me.

GitHub Copilot CLI uses Claude Sonnet 4.5 by default

No luck other than confirming it is using Claude models by default. Apparently, you can change the underlying model, for example, to ChatGPT 5, by updating the environment variable COPILOT_MODEL=gpt-5; however, we will work with the defaults here.

You can Copilot CLI’s default model via the /model command and it will also let you pick between three models.

 Select Model

 Choose the AI model to use for Copilot CLI. The selected model will be persisted and used for future sessions.

 ❯ 1. Claude Sonnet 4.5 (default) (current)
   2. Claude Sonnet 4
   3. GPT-5
   4. Cancel (Esc)

 Use ↑/↓ to navigate, Enter to select, Esc to cancel

I’m somewhat surprised that GitHub CLI isn’t shipping with gpt-5-codex model support yet. From my testing, it appears to be OpenAI’s best coding model, but it may be shipping soon.

The “You are” trick

Well, what we “really” care about are any prompts that start with “you are” since those are instructions from Copilot CLI to the model.

The “you are” tip is one I shared with Will and Simon Willison, which I have used for a few years when trying to find a tool’s system prompt, which is always buried and obfuscated in their JavaScript files.

Simon and I were invited to Microsoft HQ last month for an AI Insider’s summit (I prefer not to use the term influencer), where we were given early access to GitHub Copilot CLI before the public release. The first thing we both did was dive into the system prompt to see what it contained. Every system prompt has “You are” buried in it somewhere, which makes it much easier to find that paging through 10k lines of JavaScript.

Use Xml tags in your prompts

Will posted the Copilot CLI’s system prompt, and one thing that stood out to me was the various XML-tagged sections, such as <tips_and_tricks> and <style>, which Anthropic has encouraged for years. They documented the technique in the Use XML tags to structure your prompts section of their docs.

Natural language prompts are becoming the universal programming interface

I have been experimenting with LLMs since the release of GPT-2. While I understand why it pains many developers to admit this, natural language prompts have become a language-agnostic way to program. Somebody could build the same app I’m building using Django with the same prompts in Laravel, Rails, or any other well-documented web framework and underlying programming language.

For many people who are new to development, this is more obvious:

Prompts are code.

Languages like Python aren’t going away, but it’s much easier and faster to develop when pairing with an LLM assistant who is always online and can instantly answer any questions that you might have.

The more I examine tools like GitHub Copilot CLI, the clearer it becomes: understanding how these tools prompt their models matters not just for curiosity’s sake, but because knowing how the prompt works helps you work better with the tool. Next time you’re using an AI coding assistant, use the “You are” trick to peek under the hood. You might be surprised by what you find.

🗓️ Office Hours for Late August

Thu, 21 Aug 2025 17:54:44 -0600

I’m hosting office hours on the next two Fridays in August.

Office Hours Schedule

Join Us

These sessions are open to anyone, especially those working remotely who wants to connect with others. It’s a collaborative space for discussions, catch-ups, or just hanging out while you work.

While we have a bunch of Python and Django developers who regularly attend, anyone is welcome to join even if you don’t use either of these things.

Whether you’re a first-time attendee or a regular, you’re welcome to drop in anytime during the session. No need to stay for the full two hours, come and go as your schedule allows.

How to Join

The Zoom link is available in the gist from our previous sessions. Feel free to reach out if you need the details.

ℹ️ Meeting Guidelines

🔕 Please mute when someone else is talking 👈
👍 Be respectful to other people and mute when not talking
👍 Be mindful of the spirit of the Python Code of Conduct
👍 Be mindful and respectful of everyone’s time.
👍 Join because you want to work on something with a small group
⚠️ No marketing your product or service
⚠️ No meeting/recording bots. Humans are welcome. Meeting bots are not.
⚠️ Mute!

See you there!

Ditching Cloud APIs: How I Set Up Local Text-to-Speech with Kokoro TTS and Python

Wed, 02 Jul 2025 18:06:57 -0600

Today, I fired up the Voices macOS app, which I occasionally use to convert blog posts or documentation text to audio files that I can take on the go. I usually use one of OpenAI’s Text to speech APIs, but today I noticed there was a new option called Kokoro, which is a local voice model. I was interested in the model and tried out a few voices. Two or three of them were amazing for a local model.

After trying out a few blog posts and listening to them, I decided to find a Python version of the model. It might be slow to run, but at least I can create a pipeline to automate the process of generating voice files from arbitrary text files I have lying around.

That’s when I found the Kokoro TTS application. I downloaded it and noticed that it needed a few things. Things like NumPy required to be installed, and a couple of things were out of sync, but the project mostly just worked. While it wasn’t quite as fast as a native Mac app, it was still good enough for me to run text files through it very quickly.

Tonight, I decided to write up my notes on how to bootstrap and run the project.

Installation

git clone [email protected]:nazdridoy/kokoro-tts.git

cd kokoro-tts

uv add numpy

uv sync 

wget https://github.com/nazdridoy/kokoro-tts/releases/download/v1.0.0/voices-v1.0.bin

wget https://github.com/nazdridoy/kokoro-tts/releases/download/v1.0.0/kokoro-v1.0.onnx

Basic Usage

Once installed, you can use Kokoro TTS from the command line:

echo "Hello, this is a test of Kokoro TTS" | ./kokoro-tts /dev/stdin example-streaming.mp3 \
    --format mp3 \
    --voice af_heart

If you already have an existing text file:

./kokoro-tts textfile.txt example-textfile.mp3 \
    --format mp3 \
    --voice af_heart

Conclusion

Kokoro TTS is a solid local text-to-speech option that runs entirely on your machine without requiring API calls or internet connectivity. While it may not be as fast as native Mac apps, the quality is impressive for a local model, and the ability to automate voice file creation from text files makes it valuable for anyone who wants to convert written content to audio format.

One major advantage I discovered is that unlike most cloud APIs like OpenAI’s Text-to-speech that limit posts to around 3000 characters, Kokoro didn’t seem to have an upper limit on any of the content I tried it with. This makes it particularly useful for longer blog posts and documentation.

The setup process is straightforward once you handle the missing dependencies, and the command-line interface makes it easy to integrate into existing workflows or automation scripts.

Written by Jeff. Edited with Grammarly and Claude Code.

🤷‍♂️ I miss Visual Basic

Sun, 11 May 2025 12:33:01 -0600

I miss Visual Basic. I could build something meaningful with it in 15 to 30 minutes in a way that I have never seen anything since then, even come close to.

“For example, I personally believe that Visual Basic did more for programming than Object-Oriented Languages did,” Torvalds wrote, “yet people laugh at VB and say it’s a bad language, and they’ve been talking about OO languages for decades. And no, Visual Basic wasn’t a great language, but I think the easy DB interfaces in VB were fundamentally more important than object orientation is, for example.”

retool.com/visual-ba…

Visual Basic 3 was the first programming language and application building framework that clicked with me. I never loved the language or the inconsistencies, but it just worked. I started with GW Basic, then Turbo Pascal in high school while prepping for college. My high school business teacher let me take an independent study where I made a simple video game in the spirit of Duck Hunt, where the Energizer bunny would explode into bloody bits when you clicked on it. She wasn’t prepared for that, and told me to spend the next half of the year just doing what I was doing.

Visual Basic 6 was the last version I used before Microsoft effectively killed it with .dot by trying to turn it into something it was not. I had already started writing more web/PHP code to avoid Delphi and .net/C# code.

Trying to build a Mac app today makes me sad. There isn’t a good visual way to create an app, which is both the strength and weakness of Visual Basic’s model. When you made a VB application, it looked like a Windows app. Anything I have ever dabbled with Xcode and other application frameworks looks like an unstyled/blank canvas, making me quickly feel fatigued.

I know several people are trying to build one of these apps using Python, but it still feels like the infamous owl drawing, where step #1 is too basic, and the final step is impossible to get to. I’m hoping that AI can help fill in the “how to draw the owl steps,” but my concern is that modern-day frameworks are an impossibly hard bar to maintain and worth it for someone like myself, who isn’t paid to work on them.

I love Python, but I miss Visual Basic.

💍 Announcing The Great Django Webring

Fri, 18 Apr 2025 09:50:39 -0600

Blogging is back, so why not bring back Webrings, too?

The Great Django Webring is a tribute to the early web. It’s a cozy loop of community sites and projects, all linked together jor the joy of discovery.

Join the ring. Explore the community. Rediscover the magic of a smaller, friendlier internet.

🤖 A better Django version/update command and Prompt-driven-development

Wed, 26 Mar 2025 18:27:39 -0600

Late last year, I wrote django-cli-no-admin, a proof-of-concept Python library whose goal was to shorten Django’s default django-admin command to just django.

I published the package on pypi and it helped create one of the more colorful forum topics of 2024. I ended up writing two blog posts about it:

Ever since I had this idea, the idea of building a better django command has stuck with me. While I don’t see Django embracing the idea, of shortening its default command any time soon, that doesn’t mean we shouldn’t

Lately, I have been doing a lot of vibe coding, and I took some inspiration from the claude-code’s cli app, which has a nice developer experience (DX). The claude-code app is version-aware and can query npm’s servers to see if a newer version is out and can even update itself.

I took the output of a few of claude-code’s commands and fed that back into Claude and I asked it to build a django command that can mimic this behavior with Django using either pip or UV and the PyPI servers to query for the latest Django version.

Here is what the output of my 2-minute vibe coding sessions resulted in or what I prefer to call Prompt-driven-development (PDD):

$ django --version
5.1.6


$ django update
Current version: 5.1.6
Checking for updates...
New version available: 5.1.7 (current: 5.1.6)
Installing update...
Successfully updated from 0.5.1.6 to version 0.5.1.7

This just worked on the first try. I used a few follow-up prompts to make it less generic and to default to only ever checking Django.

One of my favorite features of prompt-driven development is that I create output like this and give it to Claude or ChatGPT, along with a few libraries with which I want to build a new tool. It gives me the working code back in seconds.

It’s incredibly powerful to focus on the end result without worrying about a clever solution and getting lost along the way. With vibe coding or prompt-driven development, we skip the journey and focus on a better result at our destination.

What about funding?

I noticed while updating another npm-based project today that it told me that two projects could be funded.

The Python Packaging User Guide lists sponsor in their Well-known labels which gives us a data point we could use to list all of the projects that may be funded that we have installed within our Django and/or Python application.

We could also be baked into our django command to determine fundable projects.

Thinking outside the box

Simon Willison’s LLM command also ships with the ability to update itself and even install its own plugins by wrapping PIP or UV or whatever magic Simon thought up.

I don’t hate using a DX like django install {package} to install a new package into my application. It would be really cool if we could even add the package into INSTALLED_APPS and make some other suggestion changes if the package could be installed in our urls.py or even explore optional settings that the package might support. I suspect this could even be locally LLM-driven if we wanted to test our docs.

I also recommend you check out Brett Cannon’s python-launcher app, a Python library wrapper. Brett was writing Rust wrappers to improve Python tooling before it was popular.

🤖 Ideas for "The Levels of Claude" Notes

Wed, 05 Mar 2025 18:57:07 -0600

Lately, I have been conversing more with friends, colleagues, clients, and online friends about Claude and coding tools.

It turns out that everyone considers Claude to be something different when they ask, “Have you tried out Claude 3.7 Sonnet?” and that could mean through the website or the mobile app or people who are using it to write code or who are developing against one of Claude’s many REST APIs. Some people primarily interact with Claude by speaking in their voices while walking outside with their pets.

I have even struggled to land at a good starting point, so I have been thinking through the various ways to explain how one might use Claude.

You can access Claude Chat through the website.
You can access Claude through the mobile app and use voice support.
You can copy one or more files into Claude.
You can access Claude through an IDE.
You use Claude Projects to organize your projects and might even have a support document you give Claude for more context.
You can access Claude through the Claude Desktop app and use voice support.
You have enabled MCP plugins/servers within the Claude Desktop app.
You can access the Claude Code CLI/terminal app.
You have enabled MCP with the Claude Code CLI/terminal app.
You are writing your own MCP servers in Python or JavaScript.
You have registered for Claude API access and are writing code against one of their REST APIs. (more on this later)

I originally broke this down into five levels of Claude, but there were too many platforms and features that blurred together.

Accessing Claude

Accessing Claude can be confusing because there are many ways to access it.

Web interface
Mobile app
Desktop app
CLI/Terminal app
REST API (could be a third-party app too, that requires a developer token)

Organization by experience level

If we break these down by user experience or technical proficiency:

Beginner: Web interface, mobile app, IDE integrations
Intermediate: Desktop app, file uploads, REST API integration
Advanced: Claude Code CLI
Really Advanced: You are writing custom MCP implementations.

Claude’s developer APIs

Claude’s Developer APIs have their vibes and are helpful in various situations. I’m not going to deep dive into them, but from a bird’s eye view:

Text Completion API
Embeddings API
Messages API
Claude-in-the-Middle (Tools API and Agents)
Batch API

These APIs can also vary based on passing them text, documents, images, or other types of files.

I’m still amazed that I can send an image to Claude and ask it to give me back an HTML file and CSS, and it does a relatively decent job.

Conclusion: Claude as a Spectrum

I wasn’t sure how to end my notes, but I asked Claude to: “Please write a conclusion about Claude being more of a spectrum than just a service.” and this is what it told me:

When discussing “Claude,” we’re not just talking about a single AI service but rather a spectrum of capabilities, interfaces, and integration points that serve different user needs, technical abilities, and use cases.

What makes this spectrum particularly interesting is how it creates different experiences and understanding of what “Claude” actually is. For a casual user accessing the web interface, Claude is simply a helpful chat assistant. For a developer using the API or building MCP servers, Claude becomes an extensible AI platform with programmable capabilities.

This spectrum explains why conversations about Claude can sometimes feel like people are talking about entirely different products. In a very real sense, they are. The Claude that helps a writer craft a story through the mobile app exists in a different context than the Claude that helps a developer debug code through an IDE integration.

As the Claude ecosystem continues to evolve, this spectrum will likely expand further, with new interfaces, capabilities, and integration points creating even more varied experiences. Understanding Claude as a spectrum rather than a singular service helps contextualize these different experiences and provides a framework for navigating the growing complexity of AI assistants.

The most powerful aspect of this spectrum may be that it allows Claude to meet users where they are - from the casual web user to the advanced developer building custom AI solutions. This flexibility transforms Claude from merely an AI service into an adaptable AI ecosystem that can evolve alongside user needs and technical capabilities.

It’s not half bad.

🤖 "Agents" are Cheat Codes

Sun, 02 Mar 2025 13:34:36 -0600

Lately, I have been trying to wrap my brain around AI Agents, so as a starting point, I have been using Pydantic AI’s Agent class/framework to build “Agents”.

“Agent” is a loaded term. Pydantic AI’s usage is more or less a system prompt and a good API around adding tool calls and working with existing LLMs.

I have written several experimental projects to help me quickly research and find answers to several areas of Django that confuse people, including myself. These ask-one-question bots do their best to fetch the resources they need to answer your questions.

The three I have published publicly are:

None of these are official resources of the Django Software Foundation, nor should they be considered “official” or even “legal” answers to any questions that may arise.

The pattern I landed on for building the system prompts and pulling remote data has been a practical, quick way for me to get feedback and ask questions based on our existing material. I can change a local copy of the bylaws and then ask the Agent questions to see if my potential changes might be comprehensive enough for the Agent to answer.

It effectively feels like running tests on governance to see if the Agent picks up on my changes.

Our Cheat Codes

These are cheat codes for a quick one-file Agent that one can quickly stand up and ask questions.

UV is a cheat code because it can quickly create a one-file Agent with dependencies and the version of Python needed to run the demo baked in.
Pydantic AI’s Agent class is a nice wrapper around a system prompt and can even create a dynamic system prompt. Having a global system prompt has a nice feel to it too.
Pydantic’s BaseModel creates structured data responses as a cheat code for processing unstructured text. If you haven’t seen this pattern yet, you can’t unsee it.
The Jina AI for cleaning up HTML into Markdown is an AI I have wanted for a decade+. I use it in dozens of apps for free, saving me hours of work.
The Python libraries Typer, Rich, and httpx may not seem like they are doing much, and I’m underutilizing them, but their Developer Experience (DX) is great, and they just work.

More areas to explore

Pydantic AI supports dynamic System Prompts, which might save me a few extra templating steps. They didn’t really click for me before I was writing this post.

When I wrote my Django Agents, I had Pydantic AI’s Multi-agent Applications feature in mind. In theory, I want to ask my Django Agents a question and have it route my question to the appropriate Agent to get an answer.

Function Tools or Tool Call is what inspired me to try out Pydantic AI. Function Tools are a way to give LLMs the ability to get information outside of their memory and system prompts when needed. I built one for reading and writing to my work calendar to help me manage my schedule. I didn’t use them for my suite of Django Agents, but when mixed with more real-time data they could be helpful.

We could also refactor each Agent using a reusable tool call so we could assemble one Agent that can gather the information needed to answer common Django Governance questions. I don’t know if that would be effective. In theory, it might not be a bad fit after looking at their DuckDuckGoSearchTool example.

🤖 My big list of AI/LLM tools, notes, and how I'm using them

Wed, 29 Jan 2025 18:46:04 -0600

I have been using, working, and running commercial and local LLMs for years, but I never got around to sharing the tools and applications I use. Here are some quick notes, tools, and resources I have landed on and use daily.

Mac Apps I use:

I do all of my development on Macs. These tools make running local LLMs accessible.

Ollama is the server that can download and run 100s of AI models.
Ollamac is a GUI client for Ollama that lets you write prompts, save history, and allow you to pick models to test out quickly.
Tailscale I use Tailscale on all of my devices, which gives me access to my work M2 Mac Studio and home office Mac Mini Pro, which both run Ollama, from anywhere in the world. This makes prototyping at home quick but then I can run a larger model from my work machine and it’s so fast, it feels like the machine is running in my house.
OpenAI Bundle—I bought this bundle because it was the cheapest way to get a bunch of AI apps, including four of Jordi Bruin’s apps. I have used these for a few years.
- MacWhisper - I use MacWhisper to turn voice notes and podcasts into plain text files for my notes and sometimes blog articles.
- Voices - I use Voices when I find a large blog post and want to listen to it while working.
Claude for Desktop gets a lot of crap for being “yet another Electron app” instead of a custom-built macOS app, but the people saying that don’t know what they are talking about. The Claude Desktop has voice support and keyboard hotkeys, which make the app incredibly useful. More importantly, Claude Desktop also supports Model Context Protocol, which lets Claude access your file system, git, and anything else you want to access. It’s incredibly powerful, and there’s nothing quite like it.

Baseline rules for running a model

While running models locally is possible, consumer hardware is constrained by RAM and GPU for even the most miniature models. The easiest mental model to work with is that Billions of parameters are roughly equivalent to your system’s RAM in Gigabytes. An 8B model needs roughly 8G RAM to fit into memory.

My mental formula is somewhat lossy because 40B models fit 32G of memory, and 72B models fit 64G of memory with some room to spare. This is just the rough estimate that I use.

Even though you can run models locally, even the smallest models with a significant context window will exceed your machine’s available RAM. A 128k context window needs about 64 GB of RAM to load into memory for an 8B parameter model fully, even though the model can easily fit into 8GB of RAM. That doesn’t mean the model won’t run locally, but it will run closer than it would if you have more than 72 GB of RAM, which your model fully needs to fit.

I look for three things when I’m evaluating a model:

A number of parameters are measured in Billions.
Context length
- The input context length, which effectively the model’s memory
- The output context length, which is how big the answer can be
The type of model:
- Default Models are general-purpose models like GPT-4 and Llama 3.3.
- Vision Models and process and read visual data like images and videos.
- Tool Models can call external tools and APIs and perform custom actions to which you give them access.
- Embedding Models can turn text into vectors or tokens, which helps measure your prompts and other RAG operations.

What about quantization? Quantization can help you scale a model down so that it might fit into memory, but there’s always a loss in quality, which defeats the purpose of using the bigger model in my book.

Keeping up

My favorite resource for keeping up is Ollama’s Models page sorted by Newest models. I check it a few times a day and you’ll see new models release single-digit hours to days before press releases can catch up.

I like Matt Williams' YouTube Channel a lot. It’s the one channel I come back to, and I find that I always learn something from it. His videos tend to be ten to twenty minutes long, which is about right since the material is so dense.

Start with his Optimize Your AI Models videos. They’re a lot to fit in your brain, but they’re a great starting point.

Simon Willison’s Weblog is good too.

Python

I’ll have to write a few posts on how I’m using LLMs with code, but Simon’s LLM is a good general-purpose AI hammer if you need one.

As of last week, I’m using Pydantic AI instead of OpenAI’s or Anthropic’s Python libraries. Pydantic AI will install both of those libraries for you, but I find it to be 100% better and easier to switch between models using it than LangChain (not linked) or anything else I have tried.

Python Click, django-click, and Typer notes

Wed, 22 Jan 2025 18:37:45 -0600

One of the most significant Python innovations in my development toolchain was the Click utility, which simplified the creation of Python scripts. Click changed how I approach writing one-off Python scripts and made it easier for me to write better developer experiences around those scripts.

Once I found django-click, writing Django management commands was a breeze, using the same Click API that I was already familiar with.

Arguably, the second most significant innovation was the Typer library, built on Click, making writing Python scripts even easier. I didn’t think it was possible to be easier than Python Click until Typer came out and proved me wrong.

The Python Click library

A typical pattern I use with Click is creating a command that accepts an argument like a URL.

import click


@click.command()
@click.argument("url", type=str)
def command(url):
	do_something(url=url)

The Python django-click library

This works well, but one improvement would be to accept as many url arguments as I can pass to it. Thankfully, Click solves this with the nargs argument. I always have to look this up, which is why we are here.

import djclick as click


@click.command()
@click.argument("urls", nargs=-1, type=str)
def command(urls):
    for url in urls:
        do_something(url=url)

The Python Typer library

Suppose we were writing this example with Typer. In that case, we could simplify it to using Python’s native data types, which would make it feel more like I’m writing native code and less like I’m using a library.

import typer


app = typer.Typer() 


@app()
def command(urls: list[str]): 
    for url in urls:
        do_something(url=url)

Conclusion

There is also a django-typer library, bringing the Typer library to Django. I suspect I’ll switch to django-typer the next time I start a new project to give it a good test drive. I can speculate on what that looks like, but I’ll leave that for another day.

📩 Email, Calendars, and the Chaos of Modern Workflows

Wed, 15 Jan 2025 20:51:45 -0600

I was feeling overloaded with emails this week, and then I remembered that my out-of-office auto-responder told people they should contact me after the first of the year if they needed me to reply.

Thankfully, I could select and archive all of my 2024 emails with this rule label:inbox after:2023/12/31 before:2025/01/01, which reconciled my old emails.

Calendars and shared Documents

With each Google organization, I’m a member with another Google Calendar, Google Drive, and Google Contacts to manage. That document someone wants feedback on sometimes feels like spinning a wheel, and I need to guess which inbox and account the message might land in.

The best solution that I have found for juggling meeting invites is Reclaim, which is terrific for merging multiple calendars into one calendar so I can at least keep on top of meeting invites and scheduling. Dropbox recently bought them, but I’m hoping that Dropbox will leave them alone.

Email and calendars have become more challenging since I switched to a Mac Studio at the office. While we were returning to work during a blizzard last week, I realized that my personal Mac Mini in my home office had no concept of my work calendar or the 4 or 7 Vivaldi profiles with syncing that I use to jump between orgs all day.

With 1Password, this is a straightforward process to set up and authorize, but it still takes time.

Tonight, I’m pretty sure I even locked myself out of one service because it’s probably not a typical usage pattern to jump between three Macs over two locations with a half dozen profiles to juggle.

Calendar Agent

Over the Thanksgiving break, I wrote my first Calendar Agent, who can read and write to my work calendar. It’s not fully baked yet, but it works well enough to tell me about my upcoming meetings and to create a meeting for me. Sometimes.

The biggest downside to using my Calendar Agent is that I have to run it from my terminal, which isn’t always the most convenient place.

Side note: I might rewrite my agent using PydanticAI as an excuse to learn about the Python agent framework, streamline tool-calling, and play with more local agents using Ollama.

The better email solution

The better email solution was a Django email app called Concorde, one of Adam Fast’s creations. It was Django querysets for managing email rules, which I modified and ran over the years. It quickly created better rules than Gmail supported, like deleting old messages in specific folders after x-days. When I kept my fork running and updated, the tool was invaluable. When I kept my Concorde up and running, my email life was healthier than when I was slower to fix it after an upgrade.

Conclusion

I’m annoyed that the best solutions for these problems are to either pay a company to make a Google Suite usable or you must be a developer to build tools to manage it all.

This stuff sucks.

django-templated-email-md notes aka if you want to format emails with Markdown, use it

Sat, 11 Jan 2025 10:22:36 -0600

I launched Django News Jobs two DjangoCon USs ago, and I somehow put off deploying emails until this week. So, every day or two, I check my Review Jobs queue to see if there’s anything new to approve or reject.

✅ Sending emails with Django is straightforward and not very painful.

🤔 Working with most email providers and troubleshooting issues is painful.

🤔 Starting with a blank page and styling emails is painful.

I tried a few third-party apps that I fought with before landing on Jack Linke’s django-templated-email-md (DTEM), which just worked. DTEM doesn’t re-invent the wheel, but it does let me write my email messages with Markdown, which turns out to be all I need.

To add email support, I followed the Usage Guide and then I added one send email function per email that I wanted to send. I’ll eventually refactor this, but it was good enough to get started.

`jobs/models.py`

For the curious, the code looks like:

# jobs/models.py

# a bunch of imports ... 

from templated_email import send_templated_mail

...

class Job(models.Model):	
    ...

    def send_job_new_email(self):
        send_templated_mail(
            template_name="job_new",
            from_email=settings.DEFAULT_FROM_EMAIL,
            recipient_list=settings.ADMINS,
            context={
                "job": self,
            },
        )

send_templated_mail does the actual sending, and template_name will look for a template file called job_new.md, which will contain our Markdown message.

You can put anything you want in context, but I will include the job so we can include as many details from our job submission as possible.

To send a Job, I can call job.send_job_new_email() via a signal, cron job, or after someone submits a Job for approval.

`templates/emails/job_new.md`

My emails contain both a “subject” and “content” block, and DTEM figures out the rest for me.

<!-- templates/emails/job_new.md -->

{% block subject %}[{{ site_name }}] New Job Posting: {{ job.title }} at {{ job.employer_name }}{% endblock %}

{% block content %}
# New Job Listing Posted

A new job has been posted on the website:

- **Title:** {{ job.title }}
- **Company:** {{ job.employer_name }}
- **Location:** {{ job.location }}
- **Remote:** {{ job.get_remote_display }}

You can view the full job listing here: <a href="{{ job.get_absolute_url_with_domain }}">{{ job.get_absolute_url_with_domain }}</a>
{% endblock content %}

My get_absolute_url_with_domain calls in my templates are my workaround for Django’s existential crisis of not making it easy to include the domain name in my urls.

Bonus: Django-Q2

I paired DTEM with Django-Q2, my favorite Django task queue. It can work with just the Django ORM, which is good enough for projects like Django News Jobs, which are relatively low traffic but spiky traffic.

If my email-sending provider times out or I have an issue, like my credit card expiring, I never want a user to see it. So, I use a task queue to handle all potentially blocking processes, like sending emails.

Django-Q2 is painless to configure. Using it involves importing async_task and modifying our send_templated_mail method to be an argument to the async_task method.

# jobs/models.py

# a bunch of imports ... 

from django_q.tasks import async_task
from templated_email import send_templated_mail

...

class Job(models.Model):	
    ...

    def send_job_new_email(self):
        async_task(
            send_templated_mail,
            template_name="job_new",
            from_email=settings.DEFAULT_FROM_EMAIL,
            recipient_list=settings.ADMINS,
            context={
                "job": self,
            },
        )

If a Job was sent successfully, I can now check the Django Admin to see if there were any failures.

Forward Email

Now that we have an email confirmed to send, my go-to provider for side projects is Forward Email. They allow outbound SMTP and even support webhooks, which I might write about some other time.

I like them over Mailchimp, Sendmail, and Gmail because they are cheap ($3 a month) and let me have unlimited domains and aliases. I have used them for a dozen side projects for several years now, and they just work. I gave up on Sendmail because I spent more time fighting with them to not turn off my account because the volume was too low. It’s worth $36 a year to have to fight this fight again.

Forward Email’s products and services are fully open-source if you care about such things.

🤔 Rethinking Django's Command-Line Tool: Why We Should Rename `django-admin`

Wed, 08 Jan 2025 19:15:02 -0600

Django has been a key tool for Python web developers for many years. But as new frameworks like FastAPI become prevalent, it’s important to ensure Django stays easy for new and experienced developers. Recently, a discussion thread received over 60 comments about changing Django’s main command from django-admin to something else, like django. The thread also explored other django-cmd possibilities, showcasing many ideas. While the conversation was broad, I want to focus on why renaming django-admin is a good idea.

Why Rename `django-admin`?

Keep It Simple and Pythonic

When I chaired DjangoCon US, a common question I asked attendees during our opening session was whether they learned Django first or Python first. It surprised me to see the majority of hands raised for “Django first,” which meant that learning Django taught them Python. This showed me how important Django is for teaching Python.

Because Django helps so many people learn Python, it should follow Python’s simple and clean style. Changing django-admin to django makes the command easier to remember and use. New developers won’t have to remember an extra non-off command, making their experience smoother.

Easy Transition for Everyone

One great thing about django-admin is that it has been the same for many years. If we rename it to django, we don’t have to remove django-admin. Instead, we can make django an alias for django-admin. This way, old projects can keep using django-admin, and new projects can use django. This change won’t disrupt anyone’s work.

Real-Life Benefits

This change is prompted by common problems in the Django community. For example, a forum post showed that some developers get confused when trying to run django-admin as a Python module. They saw errors like ModuleNotFound because it wasn’t obvious why python -m django-admin didn’t work.

Improving Compatibility with Tools Like UV

Another issue is that commands like uv tool run django don’t work as expected, but they could. Python’s best practices support using python -m django, which would work smoothly with tools like UV if Django updated its command structure. Instead, the “correct” answer is to run uv tool run --from django django-admin to bootstrap a Django project.

Renaming django-admin to django and aligning with Python’s module execution standards can make integrating Django with such tools just work. This change would help developers avoid errors and follow best practices, enhancing overall compatibility and workflow.

As it exists today, a new user has to learn to use django-admin to start a project, and then later, once they learn from seeing python -m pip used.

Balancing Old and New

Django has been around for a long time and has a strong history. It’s important to keep what makes Django great while also making improvements. Renaming django-admin to django respects Django’s past and helps it move forward. This change keeps Django reliable for long-time users while improving it for new developers.

Is this change zero work?

No, it requires a one-line config change and replacing every instance of django-admin in the docs. Any changes in Django’s docs will inevitably trigger translation changes, but these should be small and tightly scoped. As of today, there are 39-page instances to update.

Conclusion

Renaming django-admin to django improves Django’s developer experience. This new name makes the command more straightforward to remember and follows Python’s best practices. It also makes it simpler for new developers to start with modern tools like UV.

Although this change means updating some configuration files and documentation, the long-term benefits of having a clearer and more Python-like command are much greater than the initial work needed. Keeping django-admin as an alias also ensures that existing projects continue to work without problems.

Join the conversation here and share your ideas on making Django even better for everyone.

PS: This piece was written quickly and proofed even more quickly with Grammarly.

🐚 My most used commands in my terminal history

Thu, 02 Jan 2025 09:02:42 -0600

This post was inspired by Andrea Grandi’s My ZSH history post, but I modified it back to work with my customized BASH output instead

➜ history | awk '{print $4}' | sort | uniq --count | sort --numeric-sort --reverse | head -10
11063 git
7636 just
3280 cd
2575 workon
1512 ls
1061 subl
 967 docker
 887 cat
 703 python
 700 gittower

I guess you could say I use git a lot.

just is my main workflow driver.

workon - I switched between projects 2575 times.

subl and gittower are aliases to open Sublime Text and to open the existing project in Git Tower, respectively.

I used docker more than I would have guessed, but just tends to wrap most of that workflow, or it would have given git a run for its money.

🎊 Year in review (short version) - It was a good year

Wed, 01 Jan 2025 22:09:10 -0600

I love end-of-year posts and reading everyone’s posts. Every year, I draft one, and then I give up on publishing it around March or April after a comically long time. I’m going for shorter than longer this year because a lot happened this year, but most important is how I felt closing 2024 out. Overall, 2024 was a good year despite its many ups and downs.

Family

My kids grew a ton; we ended the year with a three and seven-year-old. Three-year-olds have big personalities, and my first-grader decided he could read anything after the first two weeks of school. We had several family trips, including a big family reunion in Chicago(ish) and a trip with my family to Colorado, so we got to see a lot of both families.

Each of our parents went through one hip surgery, and both are doing well. One of my parents had some more serious medical issues that we are hoping are behind everyone now. We buried two aunts in 2024, which brings me down to just one aunt and uncle (both by marriage).

Because time is so short, my new metric of a good year is measured by the health and well-being of our family and parents.

Writing

I published 230 posts this year on my Micro Blog, which ranged from short thoughts to longer posts. I have always wanted to blog, but 2024 is the year I finally made it a priority and figured out what worked for me to ease the publishing friction. I wrote about everything from Python to Django to UV to what I was watching, and I was surprised to see it inspire other blog posts and podcasts. So writing works, friends. Even if it’s just about what show you watched last night.

I’m also pleased with how my Now turned out. The page shows the latest movies, series, and games we are playing. It’s rapidly becoming a dashboard for me, which might be what my homepage turns into.

Conferences

I attended PyCon US in person for the first time since 2019 and DjangoCon US.

Volunteer/Community Work

2024 was a break for me after a decade of leadership roles in several non-profits. After five years on the Python Software Foundation board, 2024 was my first full year off. I stayed in one workgroup, but I stepped down on everything else.

After serving as President or Vice President, 2024 was my first full year as only a Django Events Foundation of North America (DEFNA) director and a DjangoCon US organizer. I had room to volunteer during DjangoCon US without my time being spoken for. It felt more or less like an emeritus role.

I mentored a small cohort for Djangonaut Space this year for the Django Packages project. It was a rewarding process, and I look forward to helping in a future session now that I know what to do. (If you were in my cohort, I promise to post that follow-up blog post soon.)

I joined the Django Software Foundation (DSF) in December, intending to hire an Executive Director. I was so happy to have my time back in 2024 that I decided to give the DSF one more year to do this on their own, but a small mob of friends asked me to run at DjangoCon US this year. So, I ran for the board, and now I have less than two years to hire someone so I can go back to normal life.

Office Hours

I hosted several Office Hours every month, letting them grow into what they needed to be. Some weeks were more structured than others, but watching this turn into a community on its own with lots of new and regular faces joining and sticking around. It’s fun and something I want to keep doing in 2025.

Side Quests

I enjoyed a ton of side quests this year, which I have already written about over the last year. From the newsletter to a job board to even a project to try to make Django talks more discoverable to a bunch of random AI projects. I had a lot of fun working on side projects and side quests.

Going forward

I left a ton of details out, but that’s alright. After 230 posts last year, it’s hard to summarize the journey, but I plan to keep removing the friction that keeps my drafts above 200 posts, which may never see the light of day.

🎄 Weeknotes for Week 51: December 15 to December 21

Sun, 22 Dec 2024 21:34:07 -0600

I mostly finished my holiday shopping this week. I picked up a few gifts for my family throughout the year and then randomly stressed out last week, thinking I didn’t have enough. I picked up a few last-minute items, and everyone was in good shape outside of a few gift cards. Now, to wrap everything…

Family

December is LEGO advent calendar month in my household. Our recently turned three-year-old participates for the first time with the LEGO® ǀ Disney Advent Calendar 2024, and our seven-year-old picked the LEGO® Star Wars™ Advent Calendar 2024. Even if you are a grownup, these are fun, and you get some cool LEGO mini-figures.

House

If you ask our three-year-old what she wants for Christmas, she defaults to “Rainbow lights” because I didn’t put any lights up outside. Between our Thanksgiving trip to Chicago and two weeks of everyone in the house cycling through the crud, here we are, but at least the Christmas tree got put up.

Community Work

My community cups ran over a bit this week. This is due to my being new to the DSF Board and various end-of-year and before-holidays “things” that pop up.

Community work cut into some of my open-source projects, but I’d like to catch up over the holidays.

I also ended my week with my last Office Hours of 2024. This week was a little too non-profit and community-heavy, so I will balance that better. With the DSF having its own office hours, I want to keep solving those problems in the proper space.

Side projects

Django News Newsletter: We shipped issue #264.
Django Packages, Django News Jobs, DjangoTV, and Upgrade Django all had minor updates over the last month. I need to do a better job aka write a tool to help me keep up since that’s mostly watching git commits or an activity feed for the more content-heavy websites.

Side Quests

Writing

2024-12-20: 🗓️ December 21, 2024, is Volunteer Responsibility Amnesty Day

2024-12-19: Default Apps 2024 - Here are my Default Apps 2024, which builds from my Default Apps 2023 post.

2024-12-17: 🤷 Why do the Django and Python communities use so many Google Forms?

2024-12-14: New project to shorten django-admin to django because we are not monsters - I didn’t realize this idea would kick a hornet’s nest, and yet somehow it did.

Entertainment

We picked up Mighty Morphin Power Rangers: Rita’s Rewind and finished it last week. It was fun, and we like these retro games, but it’s not TMNT’s level of replayability. The game has three hours of gameplay. I’m hoping future updates address this.

I read mixed reviews on G.I. Joe: Wrath of Cobra, but my son was pretty excited about the new Power Rangers game, so we picked it up as a beat ‘em up game we could play over the weekend. It’s buggy in ways that make me wonder how this game has been out for a few months and still has this level of bugs. The controls are bad, but we somehow played through to the last stage before we both shrugged and decided to call it a night. I would give it a two out of five stars type of game. If it weren’t for the nostalgia from my youth, I’d give it one or one and a half stars.

New gear

My NanoKVMs arrived a few weeks ago. I quickly ran out of ports, so I ordered some short HDMI cables, Cat6 cables, more UBC-C cables, and a cheap 8-port switch.

I also lost my Home Assistant machine again, so I swapped out RPis and still ran into issues. As impressive as Home Assistant, running, maintaining, and keeping running is a pain. I have been debating switching to one of their yellow box hardware solutions to support them financially and hoping that I won’t lose my box once a year because it’s so hard to troubleshoot and fix.

I also picked up a 50-foot sewer cam (for looking in walls and vents), an under-desk walking treadmill, and a smart garage door opener to replace our smart Chamberlain garage door opener because they dropped their API.

Next week

This week is Christmas, which means a little bit of travel. Both kids are out of school and preschool, and we both are juggling jobs and deadlines.

🗓️ December 21, 2024, is Volunteer Responsibility Amnesty Day

Fri, 20 Dec 2024 13:43:01 -0600

December 21 is Volunteer Responsibility Amnesty Day, one of two days every year that allows you to build out your responsibility inventory and decide what you have time for.

Volunteer Responsibility Amnesty Day is about checking with yourself, and ending the commitments you need to end – maybe by taking a break, or by rotating it on to someone else, or by sunsetting a project.

It’s one of the best ideas in tech because it creates space to de-commit from projects we might not otherwise have time for, but we feel like we’d be letting people down by stepping back from. It’s a wonderful excuse to step back from projects that have been weighing on your mind.

What I’m stepping down from

This year, I decided it was time to step down from the Paramiko, Fabric, and Invoke Triage Teams because I had not been active there in years. I volunteered to help triage issues that were always a bit over my head.

I joined when Jeff Forcier needed some extra hands, and it might have helped get another maintainer signed up who could help out more. At PyCon US this year, I got to hang out with Jeff. I gave him a heads-up that I felt like I was excess baggage to the project because my good intentions were no longer turning into helping out the project. I have much respect for Jeff (#teamjeff) and all three of these projects.

If you have free cycles and can help with these projects, please check out the open issues and see if you can help. I use all three beneficial projects in one way or another every week.

What you can do

Suppose you are signing up for projects but are not fulfilling your commitment. In that case, I encourage you to use Volunteer Responsibility Amnesty Day as motivation to step down from whatever project or responsibility you may have.

I also want to call out some members of the Python and Django communities. If you are signing up to help with projects you know you won’t have time for, please step back and encourage someone else to help with those projects and roles. If you know that famous face you see plastered on every website and they are friends, please ask them if they know about Volunteer Responsibility Amnesty Day and share your list. That nudge may help them decide if they are taking on too much.

🤷 Why do the Django and Python communities use so many Google Forms?

Tue, 17 Dec 2024 11:09:32 -0600

Last week, I wrote but didn’t publish some notes about why the Django and Python communities use so many Google Forms.

The simple answer is that Google Forms are quick and easy to set up, and data can be easily exported via Google Sheets or a CSV file.

Getting data in and out of Django isn’t hard, but why isn’t it as easy as using Google Forms?

I’d love to see a Django version of Google Forms that we can all use.

Has anyone solved this problem?

From a technical perspective, Django ships with a JSONField, which can store any form data we want without having to migrate databases. Turning a flat JSONField into CSV is already a solved problem.

Please note: I am not looking for a Django forms wrapper or your opinions about why Django should use your fancy Django forms tool.

New project to shorten django-admin to django because we are not monsters

Sat, 14 Dec 2024 21:18:43 -0600

One of the biggest mysteries in Django is why I have to run django-admin from my terminal instead of just running django. Confusingly, django-admin has nothing to do with Django’s admin app.

If you have ever wondered why and wanted to type django from your terminal, my new project, django-cli-no-admin solves this problem for you.

I looked at several package names on PyPI, including django-cli, which I liked the best (someone is name squatting this package.)

I gave up and went with django-cli-no-admin for lack of a better name.

# new school
uv pip install django-cli-no-admin

# old school
pip install django-cli-no-admin

# to use it...
django --version

This tool aliases Django’s django-admin script does but we shorted the name by 50%:

[project.scripts]
django = "django.core.management:execute_from_command_line"

Should Django adopt this?

Yes. But we can leave django-admin alone since we have ~20 years of history referencing it.

How long has this lived in your head?

Almost two decades.

Where is the code?

https://github.com/jefftriplett/django-cli-no-admin

📓 Weeknotes for Week 48: November 25 to December 1

Sun, 01 Dec 2024 17:48:17 -0600

Family

We drove up to Chicago to see family for five days. It’s a 7.25-hour drive if not stopping were an option, but we usually have to stop at least two or three times to re-fuel, eat, and have a few bathroom breaks. Both of my kids are pro-travelers. Give them an iPad with an offline cache of their favorite Disney and Netflix movies and series, plus some snacks, and they are good to go for 8 to 16 hours. On our last trip back from Omaha, they complained that it was too short because we didn’t stop on our 2.5-hour drive back.

We take turns driving, and through the magic of tethering, I’m surprised that I can comfortably use my laptop from the car for half the trip.

Seeing family was good. There are five kids, ranging from two to nine, but this year everyone is out of diapers, everyone can communicate their needs, and everyone plays together nicely.

We decided to avoid dinner prep drama for Thanksgiving and go out for Mexican food and margaritas. This was an excellent idea. The staff was super friendly, the food was excellent, the margaritas were the right amount of salty and sweet, and everyone got to pick something they enjoyed. There was no food prep or cleanup stress. Overall, our bill for 10 people plus a very generous tip, even after a service fee, was much less than it costs to try to feed 10 people a traditional spread.

Work

It was a short, two-day workweek for me. I helped a client with an inventory project running a week or two behind on their side. The timing wasn’t great because it all landed on my lap the day before I was heading out of town, and it took a few days to run. I hate k8s even more than before.

My main client is missing a tool or two to nail the pulse. I have thought about this because everything takes less time to complete but more time to work on the problem, so things are stretched out. The holiday break was the mental reset I needed to know how to manage this for the next month until our winter break.

Community Work

I skipped all Office Hours this week, but it was a busier few days for my DSF CoC WG card. We reset/rebooted the WG a month ago and have more members and communication. It’s taking more time to reset and settle into a healthy normal.

Side projects

Django News Newsletter: We shipped issue #260.
Django News Jobs: More jobs.
DjangoTV I fixed a bug where non-public videos were being counted in the video counts. I also made several small updates.

Side Quests

Calendar Agent

I spent some free time over the break working on an app that lets ChatGPT control my work calendar. I got the basics up quickly, and my Calendar Agent (Bot?) can look at what’s scheduled on my calendar and create new events for me. The hardest part was navigating Google permissions (I hate them) so that I could access my calendar through Python. Once I got that setup, I used Claude Projects to help me write most of the code.

I ran into an issue with tool calling because I wanted to let the Agent query the time for a day before creating the event. I shifted this logic into my create event function as a fix and realized I was way overthinking it. Now, I have an interesting agent. Still, I have yet to figure out how to run it outside my terminal, which isn’t helpful.

Entertainment

I mostly watched football and basketball this week, but I started a few shows from jefftriplett.com/now/

New gear

Since it was Black Friday, I did some shopping, but not as much as in previous years. I noticed Mastodon vibes are anti-Black Friday, whereas Blue Sky and X are more about saving money and getting a deal. More of my friends shared deals over Slack and Discord than in previous years.

I picked up a five-pack of 32 GB microSD drives for my NanoKVMs, which I hope will be delivered next week. I also bought the family an Air Fryer convection oven combo unit, which may replace our old toaster and be helpful.

Next week

We get to start our LEGO Advent Calendars. I started this tradition with my son when he was two or three. My daughter will be three in a few weeks, so she is now old enough to join in the tradition, too. She requested the Disney Princess calendar, and my son has the Star Wars set. We skipped the traditional set because one can only have so many Santa Clauses.

🤷 UV does everything or enough that I'm not sure what else it needs to do

Sun, 03 Nov 2024 19:36:23 -0600

UV feels like one of those old infomercials where it solves everything, which is where we have landed in the Python world.

I have had several discussions with friends about UV, and even when we talk about it during my weekly(ish) office hours, the list has grown to an ever-growing number of options.

UV started as a quicker way of installing Python packages, and now it’s easier to tell people that UV does everything and to focus on what it doesn’t do.

My favorite feature is that UV can now bootstrap a project to run on a machine that does not previously have Python installed, along with installing any packages your application might require.

Here is my incomplete list of what UV does today:

uv pip install replaces pip install
uv venv replaces python -m venv
uv pip compile replaces pip-tools compile
uv pip sync replaces pip-tools sync
uv run replaces pipx
uv tool run replaces pipx
uv python replaces pyenv, asdf, mise, and several other like-minded tools
uv build - Build your Python package for pypi
uv publish - Upload your Python package to pypi
astral-sh/setup-uv brings UV to GitHub Actions
ghcr.io/astral-sh/uv:latest brings UV and Python to Docker

I copied these four from uv --help, which feels like poetry features.

uv add - Add dependencies to the project
uv remove - Remove dependencies from the project
uv sync - Update the project’s environment
uv lock - Update the project’s lockfile

So what doesn’t UV do?

UV does a lot, but it still needs to do everything.

UV doesn’t run custom scripts defined in our pyproject.toml like npm-run-script allows. Thank you to [email protected]/113422203525133302">@command_tab for jogging my memory.
UV doesn’t convert my non-UV-based projects to UV. Converting is more about prefixing and replacing my commands to switch over.
UV doesn’t manage, and bump version numbers like the BumpVer, and others do.
UV doesn’t manage pre-commit like hooks. This is a long shot, but I’d love to see support via pyproject.toml.
UV doesn’t replace Python, nor should it.

📓 My notes on publishing a Python package with UV and building a custom GitHub Action for files-to-claude-xml

Wed, 16 Oct 2024 22:07:08 -0600

My new Python application files-to-claude-xml is now on PyPI, which means they are packaged and pip installable. My preferred way of running files-to-claude-xml is via UV’s tool run, which will install it if it still needs to be installed and then execute it.

$ uv tool run files-to-claude-xml --version

Publishing on PyPi with UV

UV has both build and publish commands, so I took them for a spin today.

uv build just worked, and a Python package was built.

When I tried uv publish, it prompted me for some auth settings for which I had to log in to PyPI to create a token.

I added those to my local ENV variables I manage with direnv.

export UV_PUBLISH_PASSWORD=<your-PyPI-token-here>
export UV_PUBLISH_USERNAME=__token__

Once both were set and registered, uv publish published my files on PyPI.

GitHub Action

To make files-to-claude-xml easier to run on GitHub, I created a custom action to build a _claude.xml from the GitHub repository.

To use this action, I wrote this example workflow, which runs from files-to-claude-xml-example

name: Convert Files to Claude XML


on:
  push


jobs:
  convert-to-xml:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Convert files to Claude XML
      uses: jefftriplett/files-to-claude-xml-action@main
      with:
        files: |
          README.md
          main.py          
        output: '_claude.xml'
        verbose: 'true'
    - name: Upload XML artifact
      uses: actions/upload-artifact@v4
      with:
        name: claude-xml
        path: _claude.xml

My GitHub action is built with a Dockerfile, which installs files-to-claude-xml.

# Dockerfile
FROM ghcr.io/astral-sh/uv:bookworm-slim


ENV UV_LINK_MODE=copy


RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    uv sync --frozen --no-install-project


WORKDIR /app


ENTRYPOINT ["uvx", "files-to-claude-xml"]

To turn a GitHub repository into a runnable GitHub Action, an action.yml file needs to exist in the repository. This file describes the input arguments and which Dockerfile or command to run.

# action.yml
name: 'Files to Claude XML'
description: 'Convert files to XML format for Claude'
inputs:
  files:
    description: 'Input files to process'
    required: true
    type: list
  output:
    description: 'Output XML file path'
    required: false
    default: '_claude.xml'
  verbose:
    description: 'Enable verbose output'
    required: false
    default: 'false'
  version:
    description: 'Display the version number'
    required: false
    default: 'false'
runs:
  using: 'docker'
  image: 'Dockerfile'
  args:
    - ${{ join(inputs.files, ' ') }}
    - --output
    - ${{ inputs.output }}
    - ${{ inputs.verbose == 'true' && '--verbose' || '' }}
    - ${{ inputs.version == 'true' && '--version' || '' }}

Overall, this works. Claude’s prompting helped me figure it out, which felt fairly satisfying given the goal of files-to-claude-xml.

🤖 I released files-to-claude-xml and new development workflows

Sat, 12 Oct 2024 13:00:31 -0600

After months of using and sharing this tool via a private gist, I finally carved out some time to release files-to-claude-xml.

Despite my social media timeline declaring LLMs dead earlier today, I have used Claude Projects and Artifacts.

My workflow is to copy a few files into a Claude Project and then create a new chat thread where Claude will help me write tests or build out a few features.

My files-to-claude-xml script grew out of some research I did where I stumbled on their Essential tips for long context prompts which documents how to get around some file upload limits which encourages uploading one big file using Claude’s XML-like format.

With files-to-claude-xml, I build a list of files that I want to import into a Claude Project. Then, I run it to generate a _claude.xml file, which I drag into Claude. I create a new conversation thread per feature, then copy the finished artifacts out of Claude once my feature or thread is complete.

After the feature is complete, I delete the _claude.xml file from my project and replace it with an updated copy after I re-run files-to-claude-xml.

Features on the go

One bonus of using Claude Projects is that once everything is uploaded, I can use the Claude iOS app as a sort-of notes app and development tool. I can start parallel conversation threads and have it work on new ideas and features. Once I get back to my desktop, I can pull these chat conversations up, and if I like the direction of the feature, I might use them. If not, I have wasted no time or effort on them. This also serves as a nice ToDo list.

New workflows

I am working on side projects further using this methodology. Sometimes, I would like to work on something casually while watching Netflix, but my brain shuts off from coding during the day. Instead of feeling bad that I haven’t added share links to a website or some feature I meant to add last week, I can pair Claude to work on it with me.

I can also get more done with my lunch hours on projects like DjangoTV than I could have otherwise. Overall, I’m happy to have an on-demand assistant to pair with and work on new features and ideas.

It’s also quicker to try out new ideas and projects that I would have needed to make time for.

Alternatives

Simon Willison wrote files-to-prompt, which I think is also worth trying. I contributed to the discussion, feedback, and document structure for the --cxml feature.

I wrote files-to-claude-xml before Simon had cxml support and hoped to not release my version.

However, after trying it out on several projects, my ignore/exclude list grew more significant than the files that I wanted to include in my project to send to Claude. I found it easier to generate a list of files to pass to mine instead of maintaining a long list to exclude.

⚙️ UV with GitHub Actions to run an RSS to README project

Sat, 05 Oct 2024 11:43:54 -0600

For my personal GitHub profile, I list my activities, affiliations, and the latest updates from some of my projects.

Historically, I have used JasonEtco/rss-to-readme GitHub Action to fetch a few RSS feeds or two and to update my README a few times a day.

Overall, I’m happy with this setup. I used it on the Django News GitHub Organization to pull in newsletter issues, jobs, and the latest videos from our various projects. When I tried to install rss-to-readme in our repo, I was getting node12 errors. (Have I mentioned how much I loathe node/npm?).

Instead of forking rss-to-readme and trying to figure out how to upgrade it, I used this as an excuse to “pair program” with Claude. We quickly built out a prototype using Python and the feedparser library.

I would share the chat log, but it’s mostly me trying out a few different ways to invoke it before I settle on the finished approach. See the source code over on GitHub if you are curious: https://github.com/django-news/.github/blob/main/fetch-rss.py

Once I had a working Python script that could fetch an RSS file and modify the README, I decided to run/deploy it using UV to see how minimal I could build out the GitHub Action.

GitHub Action

To run our fetch-rss.py script, we have four steps:

actions/checkout Get a git checkout of our project.
astral-sh/setup-uv Setup UV also installs Pythons for us. As a bonus, we enabled UV’s cache support, which will run much faster in the future unless we change something in our fetch-rss.py file.
Run uv run fetch-rss.py ... to fetch our RSS feeds and write them to disk. uv run installs any dependencies and caches them before our fetch-rss.py runs.
stefanzweifel/git-auto-commit-action If our README.md file has changed, save our changes and commit them back to git and into our README.

Our schedule.yml GitHub Action workflow runs twice daily or whenever we push a new change to our repo. We also set workflow_dispatch, which gives us a button to run the script manually.

# .github/workflows/schedule.yml
name: Update README

on:
  push:
    branches:
      - main
  schedule:
    # Once a day at 12 AM
    - cron: 0 12 * * *
  workflow_dispatch:

jobs:
  update:
    runs-on: ubuntu-latest

    permissions:
      contents: write

    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v3
        with:
          enable-cache: true
          cache-dependency-glob: |
                        *.py

      - name: Fetch our Feeds
        run: |
          # Fetch latest Django News Newsletter entries
          uv run fetch-rss.py \
              --section=news \
              --readme-path=profile/README.md \
              https://django-news.com/issues.rss          

      - uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: ":pencil: Updates README"

Results

Overall, I’m pleased with this solution. If I wanted to spend more time on it or re-use this workflow, I might turn it into a GitHub Action workflow so that we can call: django-news/rss-to-readme to use in other projects. For now, this is fine.

I’m happy with the astral-sh/setup-uv and uv run steps because they save me from having to set up Python and then install our project dependencies as separate steps.

I normally shy away from running Python workflows like this in GitHub Actions because they involve a lot of slow steps. This entire workflow takes 16 to 20 seconds to run, which feels fast to me.

🎉 Announcing DjangoTV

Fri, 27 Sep 2024 16:04:31 -0600

Friends, I’m launching a half-finished website idea that I have been playing around with for a while. djangotv.com is focused on promoting and searching Django videos to make discovery easier.

I wanted to launch DjangoTV before I left DjangoCon US 2024, so I’m technically announcing it from the airport. Last year, I launched Django News Jobs during the Django sprints because I was annoyed by the state of the various Django job boards.

After a year of hearing people complain about Django and Python YouTube videos not getting enough views, I decided to address the problem by building out a website to help organize and promote them.

DjangoTV is not competing with pyvideo.org. PyVideo is a fantastic resource for the Python community, and one of my goals is to make it easier to get Django content on PyVideo, too.

DjangoTV is incomplete, and I have many ideas to improve it, including backfilling some of our older conferences, adding presenters, and adding some other metadata to make it a useful community resource.

I focused on DjangoCon videos to bootstrap the project, and we’ll slowly expand our archives. I have been extracting good quotes, adding transcriptions, and adding better summaries. Please expect this to change a lot. I even launched the website with DEBUG turned on for a few days before I turned it off.

Your feedback is crucial to us. If you have any ideas or suggestions, please don’t hesitate to share them.

Most importantly, names change, people change, and bodies change. It’s essential to me that we respect each other and handle this with care. If you have a concern or need us to pull a video, please don’t hesitate to reach out; we will take care of it.

🧳 DjangoCon US, Black Python Devs Leadership Summit, and Django Girls Durham

Fri, 20 Sep 2024 06:30:00 -0600

I’m heading to Durham, NC, for seven days of DjangoCon US this Friday. This is my 10th year volunteering and the 9th year that DEFNA, the non-profit I co-founded, has run a DjangoCon US event. Here is an overview of the week.

Black Python Devs Leadership Summit (Saturday)

I’m attending and speaking on a discussion panel on Saturday at the Black Python Devs Leadership Summit. Tickets are free, and they will be streaming online in the afternoon. Donations are accepted and appreciated.

Django Girls Durham (Saturday)

Django Girls are hosting a Django workshop and teaching beginners a crash course on building their first website using Django.

DjangoCon US Tutorials (Sunday)

On Sunday morning, I’ll be volunteering and helping out at the tutorials. In the afternoon, we have a tradition of stuffing swag bags, which takes a big group and is a fun way to kick off the conference. You do not need a tutorial ticket or an organizer to help out. Ask at the registration desk, and they can direct you to when and where we are doing this.

My company REVSYS is sponsoring a DjangoSocial Raleigh/Durham Pre-DjangoCon Special meetup on Sunday evening before the conference kicks off. The meetup will be great for meeting other attendees the night before the conference.

DjangoCon US Talks (Monday through Wednesday)

The talks are great, but the busiest three days of the conference are also the busiest. There is always a lot going on, from sun up to sun down.

DjangoCon US Sprints (Thursday and Friday)

The sprints are one of my favorite parts of the conference. In past years, I have been so exhausted by the sprints that it’s hard to sit down and focus. It’s one of the best times to discuss Django and the Django ecosystem. If you have a project or want to find a project to help with, the sprints are great for getting your feet wet.

Outro

Tickets are still available if you live near Durham and want to attend. Both events have online and in-person options, so there is no pressure to make last-minute travel plans.

If you live around Durham and want to meet up, please reach out. Let’s see if we can meet for coffee.

🤠 UV Roundup: Five good articles and a pre-commit tip

Thu, 19 Sep 2024 13:18:30 -0600

I have written quite a bit about UV on my micro blog, and I am happy to see more and more people adopt it. I have stumbled on so many good articles recently that I wanted to share them because every article points out something new or different about why UV works well for them.

If you are new to UV, it’s a new tool written by Astral, the creators of Ruff.

I like UV because it replaces, combines, or complements a bunch of Python tools into one tool and user developer experience without forcing a UV way of doing it. UV effectively solves the question, “Why do I need another Python tool?” to do everyday Python tasks.

Some reason I like UV after using it for months:

It’s a faster pip and is really, really fast
It can install and manage Python versions
It can run and install Python scripts
It can run single-file Python scripts along with their dependencies
It can handle project lock files

While some people don’t care about UV being fast, it’s shaved minutes off my CI builds and container rebuilds, which means it has also saved me money and energy resources.

Overall thoughts on UV

Oliver Andrich’s UV — I am (somewhat) sold takes the approach of only using UV to set up a new Python environment. Oliver uses UV to install Python, aliases to call Python, and UV tool install to set up a few global utilities.

Using UV with Django

Anže Pečar’s UV with Django shows how to use UV to set up a new project with Django.

Switching from pyenv to UV

Will Guaraldi Kahn-Greene’s Switching from pyenv to uv was relatable for me because I also use pyenv, but I plan to slowly migrate to using only UV. I’m already halfway there, but I will have pyenv for my legacy projects for years because many aren’t worth porting yet.

Using UV and managing with Ansible

Adam Johnson’s Python: my new uv setup for development taught me to use uv cache prune to clean up unused cache entries and shows how he manages his UV setup using Ansible.

Some notes on UV

Simon Willison’s Notes on UV is an excellent summary of Oliver’s notes.

A parting UV tip

If you are a pre-commit fan hoping for a version that supports UV, the pre-commit-uv project does just that. I started updating my justfile recipes to bake just lint to the following uv run command, which speeds up running and installing pre-commit significantly.

$ uv run --with pre-commit-uv pre-commit run --all-files
pre-commit-uv

If you are attending DjangoCon US…

If you are attending DjangoCon US and want to talk UV, Django, Django News, Django Packages, hit me up while you are there.

I’ll be attending, volunteering, organizing, [email protected]/113165161818979000">sponsoring, and sprinting around the venue in Durham, NC, for the next week starting this Friday.

We still have online and in-person tickets, but not much longer!

📅 Office Hours Fall Update

Thu, 12 Sep 2024 06:00:00 -0600

This Friday, I am hosting Office Hours before I travel to DjangoCon US (organizer) and the Black Python Devs Leadership Summit (speaker) in Durham, NC.

This Friday will be my last session before a two-week break, but I will resume Office Hours again on October 4th.

Friday, September 13th, 2024, 2:30 pm to 4:30 pm
Friday, September 20th, 2024 - No office hours
Friday, September 27th, 2024 - No office hours
Friday, October 4th, 2024, 2:30 pm to 4:30 pm

High-level details

ℹ️ Anyone can join office hours. Many join because they work remotely, miss seeing faces, and miss the random conversations when a small group hangs out.

ℹ️ Our office hours are a collaborative space where we can discuss our ongoing projects, catch up, and work together to wrap up our week on a productive note.

🙏 As always, everyone is welcome to join, whether you’re a regular attendee or joining for the first time. If you are curious, reach out.

✅ If you need any additional details, feel free to send me a message or check out the gist from our previous sessions, where you’ll always find the most recent Zoom link ⚠️

🚫 Stop scheduling security updates and deprecating major features over holidays

Tue, 03 Sep 2024 08:24:54 -0600

I know people outside the US 🙄 at this, but please stop releasing major security updates and backward incompatible changes over major US, international, and religious holidays.

Given that major security updates are embargoed and scheduled weeks and months in advance, it’s essential to coordinate and avoid conflicts. A simple check of the calendar before scheduling announcements can prevent such issues.

Even if you give everyone two weeks' notice, aka what GitHub just did, wait to schedule them for release over a holiday weekend.

Historically, the Python and Django communities have also been guilty of this, so I’m not just finger-pointing at GitHub. We can all do better here.

Update: 100% unrelated to this: Django security releases issued: 5.1.1, 5.0.9, and 4.2.16. Thank you, Natalia (and Sarah) for scheduling this after the US is back from a major holiday.

🚜 Using Claude 3.5 Sonnet to refactor one of Brian Okken's Python projects

Sun, 25 Aug 2024 13:46:23 -0600

Brian Okken [email protected]/113023568032444293">posted and published his Top pytest Plugins script and then a follow-up post, Finding the top pytest plugins, which was pretty cool.

I have written a few throw-away scripts, which William Vincent wrote about and updated a few times in the Top 10 Django Third-Party Packages (2024) and The 10 Most-Used Django Packages (2024).

These efforts are powered by Hugo van Kemenade’s excellent Top PyPI Packages.

This inspired me to fork Brian’s top-pytest-plugins project, which I updated to support passing in other package names like “django” to get a rough estimate of monthly package downloads.

The refactored project is jefftriplett/top-python-packages.

Please note: Looking at the package name doesn’t scale as well for projects that have their own Trove classifiers. For a project like pytest, it works well. Many of the top packages may not even have Django in their name for a project like Django. Some projects may even actively discourage a project from using their project in their package’s name for trademark reasons. So, YMMV applies here.

Prompts

I added uv run support, which I have written about a lot lately.

I also copied the top_pytest.py file into a Claude 3.5 Sonnet session, and I let it handle the whole refactor. It even handled adding the PEP 723 new package dependencies without me asking it to.

In case it’s useful to anyone, here are my prompts:

## Prompt:
Please update this script to use a rich table.


## Prompt:
Please update the table styles to be ascii so I can copy and paste it into a markdown doc


## Prompt:
Please remove the description column


## Prompt:
Please change all PyTest and pytest references to Django and django


## Prompt:
Please add back `if 'django' in project.lower() and 'django' != project.lower():`


## Prompt:
please remove the \*# Export to markdown section. I can just pipe the output \*


## Prompt:
Please add the typer library.


## Prompt:
Please remove days and limit


## Prompt:
Please refactor the script to allow me to pass the package name instead of django. You can default to django though.


This way I can pass pytest or flask or other projects.


## Prompt:
Please change the default Table box type to MARKDOWN

Outro

I don’t usually write about Claude or prompts, but the tool has been handy lately.

If you have had some similar successes, let me know. I have been exploring some rabbit holes, and it’s changing the way I approach solving problems.

📓 UV Run Django Notes

Fri, 23 Aug 2024 06:45:00 -0600

I wanted to know how hard it would be to turn one of my django-startproject projects into a uv run friendly project. As it turns out, it worked, and the steps were more than reasonable.

Before the PEP 723’ing…

I started with a fairly vanilla manage.py that Django will give you after running python -m manage startproject.

"""Django's command-line utility for administrative tasks."""

import os
import sys


def main():
    """Run administrative tasks."""
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == "__main__":
    main()

shebang

Then we add #!/usr/bin/env -S uv run to the top of our manage.py file.

Next, we make our manage.py executable and try to run it.

$ chmod +x manage.py
$ ./manage.py
ModuleNotFoundError: No module named 'django'

Our script ran, but Python couldn’t find Django. To tell our script to install Django, we can use uv add—- script to add it.

$ uv add --script manage.py django
Updated `manage.py`
$ ./manage.py
...

Type 'manage.py help <subcommand>' for help on a specific subcommand.

Available subcommands:

[django]
    check
    compilemessages
    createcachetable
    dbshell
    diffsettings
    dumpdata
    flush
    inspectdb
    loaddata
    makemessages
    makemigrations
    migrate
    optimizemigration
    runserver
    sendtestemail
    shell
    showmigrations
    sqlflush
    sqlmigrate
    sqlsequencereset
    squashmigrations
    startapp
    startproject
    test
    testserver
Note that only Django core commands are listed as settings are not properly configured (error: No module named 'environs').

Django worked as expected this time, but Python could not find a few third-party libraries I like to include in my projects.

To add these, I passed the other four to uv add --script which will add them to the project.

$ uv add --script manage.py django-click "environs[django]" psycopg2-binary whitenoise
Updated `manage.py`
...
$ ./manage.py
...

Our Django app’s manage.py works when we run it.

After the PEP 723’ing…

After we installed our dependencies in our manage.py file, they were added to the top of the file between the /// blocks.

#!/usr/bin/env -S uv run
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "django",
#     "django-click",
#     "environs[django]",
#     "psycopg2-binary",
#     "whitenoise",
# ]
# ///
"""Django's command-line utility for administrative tasks."""

import os
import sys


def main():
    """Run administrative tasks."""
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == "__main__":
    main()

🐍 Python UV run with shebangs

Thu, 22 Aug 2024 12:54:45 -0600

This UV shebang trick that Simon Willison linked up is a nice pattern, and I plan to rebuild some of my one-off scripts in my dotfiles using it.

Here is a demo that will print “hello python” using the Python Branding colors using the Rich library while letting UV install and manage rich for you.

#!/usr/bin/env -S uv run
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "rich",
# ]
# ///

from rich.console import Console
from rich.theme import Theme

python_theme = Theme(
    {
        "pyyellow": "#ffde57",
        "pyblue": "#4584b6",
    }
)

console = Console(theme=python_theme)

console.print("[pyyellow]hello[/pyyellow] [pyblue]python[/pyblue]", style="on #646464")

Assuming you have UV installed, and you save and chmod +x this file as hello-python.py, then you should be able to run it via ./hello-python.py.

I suspect I can more easily bootstrap new machines using this trick without fewer worries about polluting my global system packages.

🐍 UV Updates and PEP 723: Simplifying Python Packaging and Scripting

Wed, 21 Aug 2024 21:57:49 -0600

The uv: Unified Python packaging update brings fresh air to the Python community, with several improvements streamlining the development process. One exciting addition is an early preview of PEP 723, also known as Single-file scripts.

The Single-file scripts feature particularly caught my attention due to its potential to simplify the distribution and execution of small Python projects. Streamlining the process is highly appealing to someone who frequently creates GitHub Gists and shares them privately and publicly.

With this new feature, I can now instruct users to run uv run main.py without explaining what a venv or virtualenv is, plus a long list of requirements that need to be passed to pip install.

I had the opportunity to test this feature over lunch today. While adding libraries to the script was straightforward, I encountered a few hurdles when I forgot to invoke uv run in my virtual environment (venv). This makes sense, given that it’s a new habit, but it highlights the importance of adapting to changes in our development workflow.

Overall, the UV: Unified Python packaging update and the introduction of Single-file scripts mark a significant step in simplifying Python development. As developers become more familiar with these improvements, we expect increased adoption and smoother collaboration on small-scale projects.

Bonus Example

I looked through some of my recent visits, and one I recently shared with a few conference organizer friends was a one-off script I used to read several YouTube video JSON files that I’m using to bootstrap another project. It was the first time I used DuckDB to make quick work of reading data from a bunch of JSON files using SQL.

Overall, I was happy with DuckDB and what PEP 723 might bring to the future of Python apps, even if my example only does a little.

# To run this application, use:
#   uv run demo-duckdb.py
#
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "duckdb",
#     "rich",
#     "typer",
# ]
# ///
import duckdb
import typer

from rich import print


def main():
    result = duckdb.sql("SELECT id,snippet FROM read_json('json/*.json')").fetchall()

    for row in result:
        id, snippet = row
        print("-" * 80)
        print(f"{id=}")
        print(f"{snippet['channelTitle']=}")
        print(f"{snippet['title']=}")
        print(f"{snippet['publishedAt']=}")
        print(snippet["description"])
        print(snippet["thumbnails"].get("maxres") or snippet.get("standard"))
        print()


if __name__ == "__main__":
    typer.run(main)

Overall, the future is bright with UV and PEP 723 may bring us. I’m excited to have more one-file Python apps that are easier to share and run with others.

PEP 723 also opens the door to turning a one-file Python script into a runnable Docker image that doesn’t even need Python on the machine or opens the door for Beeware and Briefcase to build standalone apps.

💼 Office Hours this Friday, August 9th

Thu, 08 Aug 2024 15:26:15 -0600

I am hosting Office Hours this Friday, August 9th, 2024, from 2:30 pm to 4:30 pm Central Time.

💼 I plan to upgrade some projects to work with Django 5.1 and possibly Python 3.13 as time permits. I have a few Django Packages tasks to finish too.

ℹ️ Anyone can join office hours.

ℹ️ Our office hours are a collaborative space where we can discuss our ongoing projects, catch up, and work together to wrap up our week on a productive note.

🙏 As always, everyone is welcome to join, whether you’re a regular attendee or joining for the first time.

✅ If you need any additional details, feel free to send me a message or check out the gist from our previous sessions, where you’ll find the Zoom link ⚠️

I look forward to seeing everyone.

⬆️ Which Django and Python versions should I be using today?

Wed, 07 Aug 2024 21:05:56 -0600

Django 5.1 was released, and I was reminded of the article I wrote earlier this year about Choosing the Right Python and Django Versions for Your Projects.

While I encouraged you to wait until the second, third, or even fourth patch release of Django and Python before upgrading, I received a bit of pushback. One interesting perspective claimed that if everyone waits to upgrade, we don’t find critical bugs until the later versions. While that may be plausible, I don’t believe that the dozens of people who read my blog will be swayed by my recommendation to wait for a few patch releases.

I could have emphasized the potential risks of not testing early. Please start testing during the alpha and release candidate phase so that when Django 5.1 is released, your third-party applications will be ready and working on launch day, minimizing the risk of last-minute issues.

Today, I tried to upgrade Django Packages to run on Django 5.1 to see if our test suite would run on Django 5.1, and it very quickly failed in CI due to at least one package not supporting 5.1 yet. Even if it had passed, I’m 90% sure another package would have failed because that’s the nature of running a new major Django or Python release on day one. Even if the third-party package is ready, the packaging ecosystem needs time to catch up.

Which version of Django should I use today?

I’m sticking with Django 5.0 until Django 5.1’s ecosystem has caught up. I plan to update the third-party packages I help maintain to have Django 5.1 support. After a few patch releases of Django 5.1 have come out and the ecosystem has time to catch up, I will try to migrate again.

Which version of Python should I use today?

I’m starting new projects on Python 3.12, with a few legacy projects still being done on Python 3.11. While I am adding Django 5.1 support, I plan to add Python 3.13 support in my testing matrixes to prepare everything for Python 3.13’s release this fall.

Office hours

I plan to spend some of my Office Hours this week working on Django 5.1 and Python 3.13 readiness for projects I maintain. Please join me if you have a project to update or would like some light-hearted banter to end your week.

🦙 Ollama Tool Calling Loose Notes

Fri, 26 Jul 2024 22:30:18 -0600

I spent a few hours this week working with the Ollama project and trying to get tool calling to work with the LangChain library.

Tool calling is a way to expose Python functions to a language model that allows them to be called. This will enable models to perform more complex actions and even call the outside world for more information.

I haven’t used LangChain before, and I found the whole process frustrating. The docs were full of errors. I eventually figured it out, but I was limited to one tool call per prompt, which felt broken.

Earlier today, I was telling a colleague about it, and when we got back from grabbing coffee, I thought I would check the Ollama Discord channel to see if anyone else had figured it out. To my surprise, they added and released Tool support last night, which allowed me to ditch LangChain altogether.

The Ollama project’s tool calling example was just enough to help get me started.

I struggled with the function calling syntax, but after digging a bit deeper, I found this example from OpenAI’s Function calling docs, which matches the format the Ollama project is following. I still don’t fully understand it, but I got more functions working and verified that I can make multiple tool calls within the same prompt.

Meta’s Llama 3.1 model supports tool calling, and the two work quite well together. I am also impressed with Llama 3.1 and the large context window support. I’m running the 8B and 70B models on a Mac Studio, and they feel very close to the commercial APIs I have worked with, but I can run them locally.

Embedding models

Tonight, I tried out Ollama’s Embedding models example, and while I got it working, I still need to put practical data into it to give it a better test

One more tip

If you did not know Ollama can parse and return valid JSON, check out How to get JSON response from Ollama. It made my JSON parsing and responses much more reliable.

🦙 Ollama Llama 3.1 Red Pajama

Wed, 24 Jul 2024 12:02:09 -0600

For a few weeks, I told friends I was excited to see if the new Llama 3.1 release was as good as it was being hyped.

Yesterday, Llama 3.1 was released, and I was impressed that the Ollama project published a release to Homebrew and had the models ready to use.

➜ brew install ollama

➜ ollama serve

# (optionally) I run Ollama as a background service
➜ brew services start ollama

# This takes a while (defaults to the llama3.1:8b model)
➜ ollama pull llama3.1:latest 

# (optional) This takes a longer time
➜ ollama pull llama3.1:70b

# (optional) This takes so long that I skipped it and ordered a CAT6 cable...
# ollama pull llama3.1:405b

To use chat with the model, you use the same ollama console command:

➜ ollama run llama3.1:latest
>>> how much is 2+2?
The answer to 2 + 2 is:
4!```

## Accessing Ollama Llama 3.1 with Python

The Ollama project has an [`ollama-python`](https://github.com/ollama/ollama-python) library, which I use to build applications. 

My demo has a bit of flare because there are a few options, like `--stream,` that improve the quality of life while waiting for Ollama to return results. 

```python
# hello-llama.py
import typer

from enum import Enum
from ollama import Client
from rich import print


class Host(str, Enum):
    local = "http://127.0.0.1:11434"
    the_office = "http://the-office:11434"


class ModelChoices(str, Enum):
    llama31 = "llama3.1:latest"
    llama31_70b = "llama3.1:70b"


def main(
    host: Host = Host.local,
    local: bool = False,
    model: ModelChoices = ModelChoices.llama31,
    stream: bool = False,
):
    if local:
        host = Host.local

    client = Client(host=host.value)

    response = client.chat(
        model=model.value,
        messages=[
            {
                "role": "user",
                "content": \
                    "Please riff on the 'Llama Llama Red Pajama' book but using AI terms like the 'Ollama' server and the 'Llama 3.1' model."
                    "Instead of using 'Llama Llama', please use 'Ollama Llama 3.1'.",
            }
        ],
        stream=stream,
    )

    if stream:
        for chunk in response:
            print(chunk["message"]["content"], end="", flush=True)
        print()

	else:
        print(f"[yellow]{response['message']['content']}[/yellow]")

if __name__ == "__main__":
    typer.run(main)

Some of my family’s favorite books are the late Anna Dewdney’s Llama Llama books. Please buy and support their work. I can’t read Llama 3.1 and Ollama without considering the “Llama Llama Red Pajama” book.

To set up and run this:

# Install a few "nice to have" libraries
➜ pip install ollama rich typer

# Run our demo
➜ python hello-llama.py --stream

Here's a riff on "Llama Llama Red Pajama" but with an AI twist:

**Ollama Llama 3.1, Ollama Llama 3.1**
Mama said to Ollama Llama 3.1,
"Dinner's done, time for some learning fun!"
But Ollama Llama 3.1 didn't wanna play
With the data sets and algorithms all day.

He wanted to go out and get some rest,
And dream of neural nets that were truly blessed.
But Mama said, "No way, young Ollama Llama 3.1,
You need to train on some more NLP."

Ollama Llama 3.1 got so mad and blue
He shouted at the cloud, "I don't wanna do this too!"
But then he remembered all the things he could see,
On the Ollama server, where his models would be.

So he plugged in his GPU and gave a happy sigh
And trained on some texts, till the morning light shone high.
He learned about embeddings and wordplay too,
And how to chat with humans, that's what he wanted to do.

**The end**

Connecting to Ollama

I have two Macs running Ollama and I use Tailscale to bounce between them from anywhere. When I’m at home upstairs it’s quicker to run a local instance. When I’m on my 2019 MacBook Pro it’s faster to connect to the office.

The only stumbling block I ran into was needing to set a few ENV variables setup so that Ollama is listening on a port that I can proxy to. This was frustrating to figure out, but I hope it saves you some time.

➜ launchctl setenv OLLAMA_HOST 0.0.0.0:11434
➜ launchctl setenv OLLAMA_ORIGINS http://*

# Restart the Ollama server to pick up on the ENV vars
➜ brew services restart ollama

Simon Willison’s LLM tool

I also like using Simon Willison’s LLM tool, which supports a ton of different AI services via third-party plugins. I like the llm-ollama library, which allows us to connect to our local Ollama instance.

When working with Ollama, I start with the Ollama run command, but I have a few bash scripts that might talk to OpenAI or Claude 3.5, and it’s nice to keep my brain in the same tooling space. LLM is useful for mixing and matching remote and local models.

To install and use LLM + llm-ollama + Llama 3.1.

Please note that the Ollama server should already be running as previously outlined.

# Install llm
➜ brew install llm

# Install llm-ollama
➜ llm install llm-ollama

# List all of models from Ollama
➜ llm ollama list-models

# 
➜ llm -m llama3.1:latest "how much is 2+2?"
The answer to 2 + 2 is:

4

Bonus: Mistral Large 2

While I was working on this post, Mistral AI launched their Large Enough: Mistral Large 2 model today. The Ollama project released support for the model within minutes of its announcement.

The Mistral Large 2 release is noteworthy because it outperforms Lllama 3.1’s 405B parameter model and is under 1/3 of the size. It is also the second GPT-4 class model release in the last two days.

Check out Simon’s post for more details and another LLM plugin for another way to access it.

📓 Weeknotes for Week 28: July 8 to 14

Sat, 20 Jul 2024 12:13:38 -0600

I’m running a week behind on this.

This week was our first week back home without traveling in a month, and it felt good to be home. I had time to catch up on mowing the yard, and I treated the yard with an eco-safe mosquito repellent. Despite the hot weather, Sunday felt nice outside to be mosquito-free.

I rolled my above-ground sprinkler system A few years ago, and I still need to install and run it this year. I wanted to get it this weekend, and here we are.

Family

I converted my daughter’s crib to a daybed over the weekend, and we have been using it for two nights and two naps without any issues. My son took to the board game Risk in Chicago, so I installed the iPad version and walked him through it. It was a pizza and tacos weekend because it was a long week.

Work

Occasionally, a project feels like you signed up for a race, but the distance keeps changing whenever you are within sight of the finish line. A project we have been finishing up keeps growing.

Community Work

Djangonaut Space: All three of our Djangonauts completed their first tasks, and we are moving on to our first big group project.
Conference Chats meetup

Side projects

Django News Newsletter: We shipped issue #241.
Django News Jobs: This week, we picked up more jobs that weren’t from one source. I need to write a tool to help maintain this, but it’s a manageable load.
I bought a new domain name for a project this weekend. More on that soon.

Side Quests

I dusted off my YouTube-to-Frontmatter tool and added the ability to pull playlists from a given username. I wrote the files out as JSON and used DuckDB to query them, which worked amazingly well.
I wrote an Amazon product image downloader for a few blog posts. When the product API did not work, I punted and had ChatGPT write a playwright scraper. It was faster and much less frustrating. I need this for several projects.
I cleaned up my sitemaps research tool.
I tried out a screenshots-to-code project and ran some 00s-era websites through it that I wish still existed. If someone wants to give me a few years of funding, I think we can make the web not suck again.

Writing

2024-07-14: 🔥 Why I deleted Firefox from my machines this weekend I no longer trust or believe in Mozilla, so I deleted Firefox from my machines this weekend. ➜ brew …

2024-07-13: 🦆 DuckDB may be the tool you didn’t know you were missing 🤔 I haven’t fully figured out DuckDB yet, but it’s worth trying out if you are a Python …

2024-07-12: 🚜 macOS Bartender app to Ice app I upgraded my Macs to macOS Sonoma a few weeks ago. While everything has been uneventful, the …

2024-07-11: 🎮 8BitDo Golden/Silver Limited Edition controllers My favorite third-party video game hardware company, 8BitDo, announced its 11th-anniversary limited …

2024-07-10: 📅 Office Hours for July 12th Office Hours returns this Friday, July 12th, 2024, from 2:30 pm to 4:30 pm Central Time. ℹ️ Anyone …

2024-07-09: 🔓 Sharing is Caring: How a Simple Sudo Question Led to Better Solutions One of the fun discoveries of blogging is finding your article in search results while trying to …

2024-07-08: 📓 Weeknotes for Week 27: July 1 to 7 The last week was a blur between the holiday, travel, and cramming a lot of work. My notes this week …

Entertainment

📺 Vikings: Valhalla

📺 The Marvels (2023) - This movie was better than people gave it credit for. It wasn’t my favorite, but it was fun to watch.

📺 Defending Jacob - I skipped to the end of this series.

📺 The Last Thing He Told Me - I skipped to the end of this series.

📺 Presumed Innocent - I surprised myself that I’m still keeping up with this series, but there are only a few weeks left.

📺 The Acolyte - We are ready for the last episode.

📺 Atlas (2024) - I didn’t go into this movie with any expectations, and I immensely enjoyed it.

Next week

I’m solo-parenting next weekend. I’m looking forward to hanging out with my kids and another weekend of being home.

🗳️ My thoughts on the PSF Election results

Tue, 16 Jul 2024 21:37:50 -0600

A few weeks ago, I wrote about this year’s PSF Election, three proposed bylaws changes, and how I intended to vote. I’m happy that the membership overwhelmingly approved all three proposed bylaw changes. Here is this year’s results.

Merging Contributing and Managing member classes

This change is a good step toward consolidating two membership classes and a commitment to acknowledging that all community contributions are important, not just code contributions.

Simplifying the voter affirmation process by treating past voting activity as intent to continue voting

If you voted in last year’s election, there are fewer barriers to voting in the next election. With a 76% turnout this year, I suspect next year will still yield over a 50% voter turnout, and I suspect turnout will continue to be high.

Allow for removal of Fellows by a Board vote in response to Code of Conduct violations, removing the need for a vote of the membership

This one means the most to me. When I joined the board, our Code of Conduct was barely two paragraphs long and said little. We rewrote it and formed the PSF Code of Conduct workgroup. From today forward, we can appreciate that the Python Code of Conduct applies to everyone.

Overall

We also gained three new directors, including two returning directors. This election may be the first time we have had an election in which no one running from North America made it on the board. (Possibly Europe, too, but I didn’t dive as deep to verify that.) Either way, this is a noteworthy milestone.

I’m proud of the Python community for embracing our Code of Conduct and membership changes. A few of these were overdue, but updating the voter affirmation process is an excellent proactive step and a shift for the board.

I also want to thank Débora Azevedo, the PSF’s vice chair-elect and our outbound director. I was impressed with Débora when we served on the board together, and I thought she brought valuable insights. When she put her name forward to run for vice chair, I was impressed because it’s an intimidating group to put yourself out there, and I thought Débora managed it well.

Resources

🐘 Django Migration Operations aka how to rename Models

Mon, 15 Jul 2024 11:20:41 -0600

Renaming a table in Django seems more complex than it is. Last week, a client asked me how much pain it might be to rename a Django model from Party to Customer. We already used the model’s verbose_name, so it has been referencing the new name for months.

Renaming the model should be as easy as renaming the model while updating any foreign key and many-to-many field references in other models and then running Django’s make migrations sub-command to see where we are at.

The main issue with this approach is that Django will attempt to create a new table first, update model references, and then drop the old table.

Unfortunately, Django will either fail mid-way through this migration and roll the changes back or even worse, it may complete the migration only for you to discover that your new table is empty.

Deleting data is not what we want to happen.

As it turns out, Django supports a RenameModel migration option, but it did not prompt me to ask if we wanted to rename Party to Customer.

I am also more example-driven, and the Django docs don’t have an example of how to use RenameModel. Thankfully, this migration operation is about as straightforward as one can imagine: class RenameModel(old_model_name, new_model_name)

I re-used the existing migration file that Django created for me. I dropped the CreateModel and DeleteModel operations, added a RenameField operation, and kept the RenameField operations which resulted in the following migration:

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('resources', '0002_alter_party_in_the_usa'),
    ]

    operations = [
        migrations.RenameModel('Party', 'Customer'),
        migrations.RenameField('Customer', 'party_number', 'customer_number'),
        migrations.RenameField('AnotherModel', 'party', 'customer'),
    ]

The story’s moral is that you should always check and verify that your Django migrations will perform as you expect before running them in production. Thankfully, we did, even though glossing over them is easy.

I also encourage you to dive deep into the areas of the Django docs where there aren’t examples. Many areas of the docs may need examples or even more expanded docs, and they are easy to gloss over or get intimidated by.

You don’t have to be afraid to create and update your migrations by hand. After all, Django migrations are Python code designed to give you a jumpstart. You can and should modify the code to meet your needs. Migration Operations have a clean API once you dig below the surface and understand what options you have to work with.

🦆 DuckDB may be the tool you didn't know you were missing

Sat, 13 Jul 2024 08:31:37 -0600

🤔 I haven’t fully figured out DuckDB yet, but it’s worth trying out if you are a Python dev who likes to work on data projects or gets frequently tasked with data import projects.

DuckDB is a fast database engine that lets you read CSV, Parquet, and JSON files and query them using SQL. Instead of importing data into your database, DuckDB enables you to write SQL and run it against these file types.

I have a YouTube to frontmatter project that can read a YouTube playlist and write out each video to a markdown file. I modified the export script to save the raw JSON output to disk.

I used DuckDB to read a bunch of JSON files using the following script:

import duckdb

def main():
    result = duckdb.sql("SELECT id,snippet FROM read_json('data/*.json')").fetchall()

    for row in result:
        id, snippet = row
        print(f"{id=}")
        print(snippet["channelTitle"])
        print(snippet["title"])
        print(snippet["publishedAt"])
        print(snippet["description"])
        print()


if __name__ == "__main__":
    main()

This script accomplishes several things:

It reads over 650 JSON files in about one second.
It uses SQL to query the JSON data directly.
It extracts specific fields (id and snippet) from each JSON file.

Performance and Ease of Use

The speed at which DuckDB processes these files is remarkable. In traditional setups, reading and parsing this many JSON files could take significantly longer and require more complex code.

When to Use DuckDB

DuckDB shines in scenarios where you need to:

Quickly analyze data in files without a formal import process.
Perform SQL queries on semi-structured data (like JSON)
Process large datasets efficiently on a single machine.

Conclusion

DuckDB is worth trying out in your data projects. If you have a lot of data and you need help with what to do with it, being able to write SQL against hundreds of files is powerful and flexible.

📓 Weeknotes for Week 27: July 1 to 7

Mon, 08 Jul 2024 11:37:40 -0600

The last week was a blur between the holiday, travel, and cramming a lot of work. My notes this week are more glossed over than most

Family

We drove to Illinois (near Chicago) to see family this week. We had family from both coasts who we don’t see very often, and it was the first time much of the family met my youngest, Nora. It takes us 8 to 8.5 hours to drive there, and people are always amazed at how good my kids are at traveling these distances. They both love the extra screen time and are easy to travel with. We can hand each one an iPad, and they are set for hours, given enough snacks and a rest stop every few hours.

Work

It was a short two-day week, but I got a bonus third day of work between fitting a few hours in the car and half a day on Friday. This was nice because we are wrapping up an existing client project while ramping up on another project.

This has also made me realize that Django has many shopping cart projects, but they all seem to be outdated. We struggled to find a project we could use, which made me realize that the Django community is sorely missing a good shopping cart and checkout experience that works.

Community Work

Djangonaut Space: All three Djangonaut Space Team Neptune members have made meaningful contributions early in the process, and now I’m feeling the pressure to detail and share some more advanced projects so they can have tasks for the duration of the project.

Django Code of Conduct WG: We have prioritized keeping up as we get new members used to contributing. This week, was a busy week.

Side projects

Django News Newsletter: We shipped issue #240.

Django News Jobs: We have had code to aggregate some other Django job boards, and this week, we started allowing those to come through.

Writing

This week, I was back on track, and I wrote and published something every day. It was my first day of giving myself a buffer so that I am always writing tomorrow’s post, which took the edge and pressure off. I am still writing every day, but with the holiday, it was nice knowing I didn’t have to fit it in on our last day in Chicago, which ended up being a really long, full day with family.

2024-07-07: 🧰 More fun with Django Extensions using shell_plus and graph_models Yesterday, I wrote about Django Extensions show_urls management command because it’s useful. I …

2024-07-06: Django Extensions is useful even if you only use show_urls Yes, Django Extensions package is worth installing, especially for its show_urls command, which can …

2024-07-05: 📅 No Office Hours on July 5th, but… No Office Hours this week (July 5th), but we will return next Friday, July 12th, 2024, 2:30 pm to …

2024-07-04: 🎆 🤖 Happy AIndependence Day To everyone in the United States, Happy Independence Day and Happy AIndependence Day to everyone …

2024-07-03: 🗳️ PSF Elections how I am voting This was written while driving to Chicago (technically from the passenger seat). Still, a few people …

2024-07-02: 💬 On the PSF Bylaw changes The Python Software Foundation has three bylaw changes up for a vote in this year’s election. …

2024-07-01: 📓 Weeknotes for Week 26: June 24 to 30 Family I took two days off for a funeral and some other family stuff. I saw cousins and other …

Entertainment

📺 Sweet Tooth - I really liked this series.

📺 Presumed Innocent - I watched the first five episodes, and it’s okay.

📺 The Last Thing He Told Me - I just started this series on Sunday night and am unsure if I’ll finish it.

Next week

It’s my first whole week back in a while. I’m looking forward to catching up and mowing my yard while listening to podcasts. I need to catch up on house stuff, including running my homemade above-ground sprinkler system. (Chill; we live near the Kansas River.)

Django Extensions is useful even if you only use show_urls

Sat, 06 Jul 2024 08:00:00 -0600

Yes, Django Extensions package is worth installing, especially for its show_urls command, which can be very useful for debugging and understanding your project’s URL configurations.

Here’s a short example of how to use it because I sometimes want to include a link to the Django Admin in a menu for staff users, and I am trying to remember what name I need to reference to link to it.

First, you will need to install it via:

pip install django-extensions

# or if you prefer using uv like me:
uv pip install django-extensions

Next, you’ll want to add django_extensions to your INSTALLED_APPS in your settings.py file:

INSTALLED_APPS = [
    ...
    "django_extensions",
]

Finally, to urn the show_urls management command you may do some by running your manage.py script and passing it the following option:

$ python -m manage show_urls

Which will give this output:

$ python -m manage show_urls | grep admin
...
/admin/	django.contrib.admin.sites.index	admin:index
/admin/<app_label>/	django.contrib.admin.sites.app_index	admin:app_list
/admin/<url>	django.contrib.admin.sites.catch_all_view
# and a whole lot more...

In this case, I was looking for admin:index which I can now add to my HTML document this menu link/snippet:

... 
<a href="{% url 'admin:index' %}">Django Admin</a>
...

What I like about this approach is that I can now hide or rotate the url pattern I’m using to get to my admin website, and yet Django will always link to the correct one.

🗳️ PSF Elections how I am voting

Wed, 03 Jul 2024 07:30:00 -0600

This was written while driving to Chicago (technically from the passenger seat). Still, a few people contacted me and asked me how I vote for PSF directors, so I wanted to share.

If you can vote in the PSF election, please do so before Tuesday, July 16th, 2024, 2:00 p.m. UTC. For more details, check out their blog post, The 2024 PSF Board Election is Open! (The blog post lists the date incorrectly as ending on a Friday.)

I served on the PSF board for five years, from 2018 to 2023, and here is what I am looking for when I research who to vote for this year.

It’s a harder-than-normal slate of candidates because only three open seats are available, compared to four in the last few years. More candidates are new to me this year than most years.

We also have some solid candidates running, which makes it even harder.

Existing directors

For existing directors, I look at:

How long have they served on the board?
What positions/roles did they serve while on the board?
What has the PSF accomplished during its term?
What was their meeting attendance like?
I read their previous candidate statements to see if the PSF accomplished what they said they wanted.
How did they treat the PSF’s staff in meetings, at events, and behind the scenes?

All candidates

For everyone, including existing directors, I read their candidate statements, and then I look at:

What do they value?
What their company and community affiliations are.
Understands and is committed to our Code of Conduct to promote a healthy community.
What workgroups did they participate in?
What are they committed to working on and changing in their next term?

What I’m prioritizing

I prioritize diverse candidates and representation. This includes geographical representation.

I prioritize candidates who do not work for Big Tech / FAANG companies. These companies often give raises and promotions to employees who make it on open-source boards. It’s a KPI goal for some Dev Rels and I’m not here to help any Big Tech companies. Thankfully, I have served with many selfless directors, despite them working for Big Tech companies. Still, you have a right to know when you vote for someone if their position will check off one of their KPIs.

I prioritize the Python community’s needs over individual needs.

I prioritize candidates with a track record of getting things done over only showing up to be seen. (Yes, this is a thing.)

I prioritize practical communication skills. If you write over the heads of the community, then you could be more effective at communicating.

I’m looking for reform.

Overall, I am looking for reform. A few public mishaps by the board have damaged the community’s trust in the organization.

I want to see reform across those impacted workgroups, and I will prioritize candidates who show awareness of this. I considered listing them here, but “what I’m prioritizing” when I vote is not up for public debate. I might write more on this later.

If you are running

If you are running for the board, thank you for putting yourself out there.

💬 On the PSF Bylaw changes

Tue, 02 Jul 2024 07:00:00 -0600

The Python Software Foundation has three bylaw changes up for a vote in this year’s election. I support all there.

Here is their post, For your consideration: Proposed bylaws changes to improve our membership experience and a follow-up post FAQ for Proposed Changes to PSF Bylaws that addresses questions that came up.

Change 1: Merging Contributing and Managing member classes

The existing two classes need to be clarified and updated. We want contributors, and code is only one of many ways to contribute to a healthy community.

✅ I’m all for this. I support this change.

Change 2: Simplifying the voter affirmation process by treating past voting activity as intent to continue voting

Before the voting affirmation process was enforced, meeting the 30% quorum mark was an issue.

Last year, over 70% of our members voted, and I suspect we can land somewhere in the middle, letting people continue to vote if they did the previous year and still staying above 50% of our active membership voting.

✅ I support this change.

Change 3: Allow for removal of Fellows by a Board vote in response to Code of Conduct violations, removing the need for a vote of the membership

No one in the Python community is above Python’s Code of Conduct.

While I disagree that the board cannot already remove a Fellow, it’s better to bring it to the community for a vote.

✅ As a Python Fellow, past board member, and Code of Conduct WG member, I support this change.

📓 Weeknotes for Week 26: June 24 to 30

Mon, 01 Jul 2024 08:30:00 -0600

Family

I took two days off for a funeral and some other family stuff. I saw cousins and other relatives who I only see at funerals, which is starting to feel like a habit.

Work

I was mentally blocked on a project, or at least the part I was putting off returning to. Something finally clicked over the weekend, and I worked through it. I hate working the weekends, which our company frowns upon. I didn’t expect to be out two days this week, and I’m out for the holiday most of the next week, so I tried to get caught up and ahead.

Community Work

Djangonaut Space: Week 2 kicked off, and I had to push our weekly standup back a week due to the funeral. No idea, but our Djangonauts are doing good work and working through their first issues.

Side projects

Django News Newsletter: We shipped issue #239.

Django News Jobs: I had a weird issue that forced me to downgrade Python 3.12 to 3.11 to fix something internally that broke Pydantic. I’ll switch back in a few weeks, but I’m still annoyed by it.

Overall, I just treaded water this week.

Writing

I shifted my writing schedule around this weekend so that I am a day ahead. I want to get four or five days ahead so that I have a nice publishing schedule to work with. I still plan on writing daily, but I like having more wiggle room.

2024-06-30: 🐳 Using Just and Compose for interactive Django and Python debugging sessions When I wrote REST APIs, I spent weeks and months writing tests and debugging without looking at the …

2024-06-29: 🐘 Docker Postgres Autoupgrades Upgrading Postgres in Docker environments can be daunting, but keeping your database up-to-date is …

2024-06-28: 🐘 A Just recipe to backup and restore a Postgres database I have used this casey/just recipe to help backup and restore my Postgres databases from my Docker …

2024-06-27: 📅 Office Hours for June 28th It’s been a week, but I’m hosting Office Hours this Friday at 2:30 pm. I was on the …

2024-06-25: 🐳 Managing Docker Compose Profiles with Just: Switching Between Default and Celery Configurations For a recent client project, we wanted to toggle between various Docker Compose profiles to run the …

2024-06-24: 🚜 Mastodon Bookmark exporter to Markdown/Frontmatter I wrote a Mastodon Bookmark exporter tool over the weekend and decided to polish it up and release …

Entertainment

📺 Presumed Innocent

📺 Baby Reindeer

📺 Dark Matter

📺 The Acolyte - We got caught up, and OMG, it’s good.

📺 The Flash (2023) - I couldn’t care less for the Flash, but I really enjoyed seeing Michael Keaton’s Batman again.

Next week

We are spending our holiday in Illinois to see family. It’ll be the first time my youngest has met several family members flying in for the holiday.

🐳 Using Just and Compose for interactive Django and Python debugging sessions

Sun, 30 Jun 2024 08:00:00 -0600

When I wrote REST APIs, I spent weeks and months writing tests and debugging without looking at the front end. It’s all JSON, after all.

For most of my projects, I will open two or three tabs. I’m running Docker Compose in tab one to see the logs as I work. I’ll use the following casey/just recipe to save some keystrokes and to standardize what running my project looks like:

# tab 1
$ just up

In my second tab, I’ll open a shell that is inside my main web or app container so that I can interact with the environment, run migrations, and run tests.

We can nitpick the meaning of “console” here, but I tend to have another just recipe for “shell” which will open a Django shell using shell_plus or something more interactive:

# tab 2
$ just console

In my third tab, I’ll run a shell session for creating git branches, switching git branches, stashing git changes, and running my linter, which I prefer to run by hand.

# tab 3
$ echo "I'm boring"

Over the last year or two, the web has returned to doing more frontend work with Django and less with REST. Using ipdb, in my view, to figure out what’s going on has been really helpful. Trying to get ipdb to “just work” takes a few steps in my normal workflow.

# tab 1 (probably)

# start everything
$ just start

# stop our web container
$ just stop web

# start our web container with "--service-ports" 
# just start-web-with-debug

The only real magic here is using Docker’s --service-ports, which opens ports so we may connect to the open ipdb session when we open one in our view code.

My main justfile for all of these recipes/workflows looks very similar to this:

# justfile
set dotenv-load := false

@build *ARGS:
    docker compose build {{ ARGS }}

# opens a console
@console:
    docker compose run --rm --no-deps utility/bin/bash

@down:
    docker compose down

@start *ARGS:
    just up --detach {{ ARGS }}

@start-web-with-debug:
    docker compose run --service-ports --rm web python -m manage runserver 0.0.0.0:8000

@stop *ARGS:
    docker compose down {{ ARGS }}

@up *ARGS:
    docker compose up {{ ARGS }}

If you work on multiple projects, I encourage you to find patterns you can scale across them. Using Just, Make, shell scripts or even Python lightens the cognitive load when switching between them.

🚜 Mastodon Bookmark exporter to Markdown/Frontmatter

Mon, 24 Jun 2024 22:21:16 -0600

I wrote a Mastodon Bookmark exporter tool over the weekend and decided to polish it up and release it tonight.

I wrote the tool to help me sort out Mastodon posts that I might bookmark to follow up on or write about. I bookmark posts on the go or even from bed, and when I have time, I will pull them back up.

The Mastodon Bookmark exporter tool reads your Mastodon bookmarks and exports the latest posts to a markdown/frontmatter file.

I’m releasing the project as a gist under the PolyForm Noncommercial License for personal reasons. If you have licensing questions, contact me directly or through www.revsys.com for commercial inquiries, and we can work something out.

Weeknotes for Week 25: June 17 to 23

Sun, 23 Jun 2024 21:38:34 -0600

Last week, my side of the family drove out to Colorado Springs, Colorado, and we rented a house for the week. The trip was great; seeing family was good, and watching the kids spend time with family they don’t get to see very often was the highlight for me. The mountains were great, but I don’t want to ride a train from 8k to 14k feet of altitude soon.

Family

The last of my biological aunts and uncles passed away two nights before we left Colorado. We had some other unexpected news, and I hope we have better news after the funeral this week.

Work

I took the whole week off, so the only work this week was community work.

Community Work

Djangonaut Space: Week 1 kicked off, and we had our first team meeting. The meeting went well, and I have a follow-up meeting on Monday with one Djangonaut who couldn’t make it.

General Python: I had a ton of messages from Python friends this week asking for advice on upcoming bylaws changes. I wrote up a few drafts, and I might get around to publishing one or the other.

Side projects

Django News Newsletter: We shipped issue #238.

Django Packages: More small updates and more on this sooner.

I wrote a tool to export my Mastodon bookmarks. I tend to revisit these every day, and I wanted a way to create checklists for them and save them. I might find good content from the newsletter or posts I want to follow up on once I have more time.

Writing

I wrote every day but only published for three days, including today. I wrote several drafts, but I mostly spent time with my family.

2024-06-19: 💜 Follow Black Python Devs on Juneteenth If you haven’t heard about Black Python Devs, Juneteenth is an excellent day to learn about …

2024-06-17: Djangonaut Space has begun This is my first time helping out with Djangonaut Space as a Navigator for the Django Packages…

Entertainment

📺 Leave the World Behind - I rewatched half of this movie with my mother and sister.

📺 Hellboy - I didn’t realize I had missed this one. I liked it even if I don’t know much about the Hellboy story outside the other movies.

Next week

I am returning to work a day earlier than planned, but it will be a long, short week.

💜 Follow Black Python Devs on Juneteenth

Wed, 19 Jun 2024 12:47:59 -0600

If you haven’t heard about Black Python Devs, Juneteenth is an excellent day to learn about this organization and how to support it.

If you haven’t heard about Black Python Devs, here is a quick overview:

Black Python Devs was created by its founder Jay Miller after seeing a trend of the same handful of Black developers speaking at major conferences, taking leadership positions, and dealing with the same challenges towards burnout.

blackpythondevs.com/about/

I also want to share the goals of Black Python Devs:

Our goal is to become the largest community of Black Python Developers in the world and establish our community as a source for diverse leaders in the greater Python community.

We aim to:

Establish guidance, mentorship, and career support for Black Pythonistas around the world.

Create opportunities for the Python community to invest in local communities of Black Python Devs members

Increase participation of Black Python Devs members in existing Python community programs, events, and initiatives.

Follow them today

Juneteenth is also a wonderful day to follow them on Mastodon (248 followers) or on Xwitter (606 followers).

I would love to check these numbers tomorrow and see a non-zero increase in followers.

You can also financially support Black Python Devs through the GNOME Foundation by scrolling down to the Support Black Python Devs section of the website.

Disclosure

I am a Black Python Devs Leadership Advisor who has watched and supported this organization grow even before I was asked to join. This team is near and dear to my heart, and I invite you to follow them and help spread the message.

🦙 Exploring Ollama: An Open-Source LLM (my notes)

Tue, 11 Jun 2024 20:57:51 -0600

I have meant to do a proper write-up on the Ollama project for a while. Instead of putting it off any longer, I decided to publish my outline and notes, hoping that someone might find it helpful if you wanted to run your own local, offline LLM.

Why Ollama?

Open-source LLM server
Runs over two dozen models, with new ones released every week
My personal favorite: Ollama

My Journey with Ollama

Discovered Ollama a year ago
One of the first accessible projects for M-series Mac
User-friendly and performed well.

Ollama’s Growing Compatibility

Expanded to support Windows and Linux
Became AMD-friendly
More accessible to users with various hardware setups

The Llama3 model changes everything

Llama3 70b model introduction
70b model runs on Mac Studio with 64 GB of RAM
Enables running a powerful LLM locally
Benefits: privacy, customization, offline access

Making Life Easier with the LLM Python Project

Installed llm-ollama plugin into LLM Python project
Simplifies switching between different LLMs
Quicker and easier testing of new models compared to Ollama’s CLI

Ollama’s Library and APIs

ollama-python library for integrating Ollama models into Python projects
Ollama’s built-in OpenAI compatible API for seamless use with existing OpenAI-based applications

Final Thoughts

Ollama: a top choice for open-source, locally-run LLM
Expanding compatibility
Impressive Llama3 model
Easy integration with LLM Python project
Handy libraries and APIs
Recommended for researchers, developers, and curious individuals

🧱 Django ModelForm Template starting point

Sat, 08 Jun 2024 22:56:56 -0600

Recently, I have been doing a lot of Django formwork. I start with a basic template like form.as_div or form|crispy until it grows uncomfortable.

Today, I was bouncing between two projects, and I noticed I was working on the tasks that had grown uncomfortable to the point that I dreaded working on the templates.

While I enjoy working with Django’s template system, I was putting off these tasks, and all they had in common was finishing some of the form and template work.

I couldn’t quite understand why this was such a mental blocker, so I stopped working, disconnected, and mowed my yard. Thankfully, that did the trick.

As I finished mowing, I realized that I was struggling to complete these tasks because I was overwhelmed by needing to dump all the form fields into a template.

Once I realized why I was feeling this resistance, I realized I needed to focus on solving this issue to move on.

I remembered Daniel Roy Greenfeld’s Rapidly creating smoke tests for Django views from a few weeks ago, where he made a management command to print out a bunch of smoke tests.

I decided to try the same technique by passing a string path to a Django ModelForm and printing out my form template.

Edit: micro.blog did not like the templates in templates from my Python script, so I had to swap out the inline example with a gist. Sorry about that.

This template could be better, but it was good enough. I tested it on a few of the forms I’m using on Django News Jobs, and it’s an improvement over what I started with.

Something was in the water because when I checked our company Slack, Frank Wiles showed me his new make-management-command project, which takes a similar approach to creating the folders and files needed to create a new management command.

🍿 On the Google leak

Mon, 03 Jun 2024 18:37:29 -0600

Last week, Google leaked 2,500 pages of internal documentation by accidentally pushing these docs to their GoogleAPI GitHub repo.

A collection of 2,500 leaked internal documents from Google filled with details about data the company collects is authentic, the company confirmed today. Until now, Google had refused to comment on the materials.

These documents were also leaked Apache 2.0 “irrevocable copyright license” when released under normal circumstances.

The fun thing about accidentally publishing to the GoogleAPI GitHub is that, while these are sensitive internal documents, Google technically released them under an Apache 2.0 license. That means anyone who stumbled across the documents was granted a “perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license” to them, so these are freely available online now, like here.

www.theverge.com/2024/5/29…

Several SEO experts have proven that Google has lied to web developers for over a decade. While I don’t have any remorse for the SEO industry, knowing “best practices” is good for anyone who builds a website and wants a chance to reasonably rank.

So, while Google has been called out for years for misleading the general public, now a trove of documents prove it.

Both Fishkin and King accuse Google of “lying” to SEO experts in the past. One of the revelations in the documents is that the click-through rate of a search result listing affects its ranking, something Google has denied goes into the results “stew” on several occasions. The click tracking system is called “Navboost,” in other words, boosting websites users navigate to. Naturally, a lot of this click data comes from Chrome, even when you leave search. For instance, some results can show a small set of “sitemap” results below the main listing, and apparently a part of what powers this is the most-popular subpages as determined by Chrome’s click tracking. arstechnica.com/gadgets/2…

As a web developer, what’s frustrating is how obvious spammy some of the top websites are that mirror websites like python.org’s content and add spammy ads and content and somehow outrank the biggest programming language on the planet’s docs.

None of this is new. I have seen the same spammy websites for years, which led me to drop Google Search for Kagi so I could stop seeing them.

Behavior never lies.

💼 Packing and unpacking

Mon, 20 May 2024 20:06:16 -0600

I’m exhausted after an excellent PyCon US, but I had packing and unpacking on my mind. I usually pack too many clothes in a suitcase that is too small and regretting it on my way back.

I like to use packing cubes, and I still pack way too much into slightly too small of a suitcase, and then I regret it after bringing home too much swap home with me.

This year, I challenged myself to pack less and to bring my more giant suitcase. I could even see the bottom of my luggage around my packing cubes.

Overall, my take-home booth/vendor swag was very modest this year. I only brought home one T-shirt, a hat, and a bunch of stickers. I meant to buy a t-shirt to support Black Python Devs, but y’all bought them all before I could make it back with cash.

I also want to thank my friends who brought gifts for my kids and myself, which I didn’t see coming. I am humbled and amazed that my community work and friendship sparked that response. I am humbled, and everything fits perfectly in my suitcase.

My only misses during this trip were bringing too many USB-C cables, one book I never opened, and a smaller satchel to carry my laptop. I even left my computer in my hotel for a few days. Next year, I will bring fewer cables with me.

The only device I wish I had brought was a small power strip. I have a GHome Smart Power Strip, which has a few USB charger ports, and it would have been nicer than fighting some of the hotel’s loose, built-in chargers. With a power strip, I would have had more space to plug it into a wall jack.

Someday, I will write up a proper gear post since I picked up a few new items I enjoyed having this year, but tonight, I will start recovering from my sleep debt.

🍔 PyCon US, have it your way (you rule!)

Thu, 16 May 2024 08:40:06 -0600

Here is my PyCon US advice based on Trey Hunner’s Mastodon post:

What are some of your favorite @pycon tips? 💡

Reply and tag #PyConUS. 💖

mastodon.social/@treyhunn…

Talks are recorded

Keep in mind that talks are recorded.

Seeing every talk at PyCon US while you are attending is impossible. There are five tracks with three days of talks. There will be times when the two or three talks you most want to attend will all be given during the same block. Or you aren’t used to talking to people all day and need extra sleep.

It’s fine. The talks are recorded.

Ignore PyCon US advice anti-patterns

Python developers often have firm opinions. Every year, there are several blog posts about why you should “only go to talks” or “only hallway track” when the rest of us will do both.

Don’t sleep on the first two days of the Expo Hall

The Expo Hall has sponsor booths for one reception evening and the first two days of talks. On the third day of talks, the Expo hall gets converted into a Job Fair with other events running, and it takes a lot of work to bump into people.

There have been some years where I never made it out of the Expo Hall for either day, even though I wasn’t working a booth. I have even watched lunch get carted in and carted out while talking with people because it’s a great location to meet new people and run into everyone you know online or from other conferences over the years. Don’t do that. Stop and eat lunch.

Take advantage of the first two days of the Expo Hall, even if you can’t make it to the opening reception. Say hi to sponsors, too.

Embrace JOMO (Joy Of Missing Out)

At my first PyCon US attendance as a board director, I didn’t know I was supposed to attend a dinner for PyCon US organizers. My plane arrived late, and I got pinged if I knew about it about 30 minutes after the dinner started. I did a quick check. It was a 20-minute walk in a city I just landed in, so I thanked them and told them I couldn’t make it.

Instead, I went to the hotel bar to get dinner at the bar. I wasn’t even staying in the main conference hotel. I made small talk with one of the other patrons, and they were not only attendees, but they just happened to be maintainers of a few of my favorite Python packages.

You’d be amazed at how many people you’d otherwise never have met, even when you feel like not saying anything on the beaten path.

Remember Burger King’s golden rule, “have it your way”

My six-year-old loves the Burger King “You Rule” commercials because of their TV jingle. Even though he’s never had their food, that doesn’t stop him from singing the ending chorus, “At BK, have it your way,” and then shouting, “(You rule!).”

You rule You’re seizing the day At BK, have it your way (You rule!) genius.com/Burger-ki…

While I can’t speak for their food, it’s good advice.

You rule

First, everyone likes a compliment and “you rule.” You probably don’t deflect a compliment when a TV jingle gives you one, so carry this into life the next time someone compliments you. Don’t deflect it.

You’re seizing the day

While PyCon US feels like a lot of days, it goes quickly. Take time to enjoy it. I have been to enough PyCons to know that jobs, careers, families, budgets, and health change.

At BK, have it your way

The only proper way to PyCon US is to “have it your way.” Go to the talks you want to see and talk to the people you want to talk to, but I encourage you to put yourself out there and meet new people.

(You rule!)

You rule!

(take the damn compliment)

📓 Weeknotes for Week 19: May 6 to May 12

Sun, 12 May 2024 19:21:09 -0600

What a week.

Family

My youngest was home most of the week with a virus, which changed everything about my week.

We had a rare night out to see Jason Isbell and the 400 Unit at Liberty Hall

My family also had a birthday on Friday, my son’s friend’s birthday party on Saturday, and Mother’s Day today, so it was a busier than usual week for us and a sick kid all week.

Work

With my daughter sick, everything changed this week. I mostly worked on an e-commerce app on a rare Saturday to get my bearings.

Community

Conference Chats: We had our monthly Wednesday night meetup.

Office Hours: I hosted Friday Office Hours.

Side projects

Django News Newsletter: We shipped issue #232.

Django Packages: I mostly worked on some notes and task ideas for Djangonaut Space.

Upgrade Django: We had four updates this week, which is rare.

Writing

I wrote and published every day. I started writing and publishing on my https://micro.webology.dev every day since February 1st, and this week marked my 100th day in a row of writing.

2024-05-11: 📭 On vacation autoresponders and deleting email

2024-05-10: 🐍 TIL build-and-inspect-python-package GitHub Action workflow plus some bonus Nox + Tox

2024-05-09: 📚 African tech book drive at PyCon US

2024-05-08: 👥 Office Hours on May 10th

2024-05-07: 🎸 Jason Isbell and the 400 Unit at Liberty Hall

2024-05-06: 🌪️ Tornado Watch tonight

“Me” time

Early Saturday morning, I wrote a podcast downloader script to make downloading podcast episodes easier. I like to run some of my favorite podcast episodes through MacWhisper to generate a transcript to take better notes.

Saturday evening, I wired up Fabric to connect to my Macs to update them and run various Just recipes to keep everything updated. I ran into a few issues with my .bashrc file, but it was a good enough MVP.

Entertainment

📺 Space Jam: A New Legacy: My oldest has gotten into basketball and this was a fun family movie.

📺 Dead Boy Detectives: I didn’t have high expectations for this series, but watching two ghosts solve mysteries was fun.

🎮 Teenage Mutant Ninja Turtles: Shredder’s Revenge: We hadn’t played this one in a while, so we primarily focused on leveling up the two new characters from the expansion.

Next week

I leave for PyCon US on Wednesday and have a small mountain of things to finish between now and then. I finished my laundry tonight, so I can leisurely pack over the next few days.

🐍 TIL build-and-inspect-python-package GitHub Action workflow plus some bonus Nox + Tox

Fri, 10 May 2024 19:59:21 -0600

TIL: via [email protected]/112419234883043196">@joshthomas via @treyhunner via @hynek about the hynek/build-and-inspect-python-package GitHub Action.

This workflow makes it possible for GitHub Actions to read your Python version classifiers to build a matrix or, as Trey put it, “Remove so much junk” which is a pretty good example.

As a bonus, check out Hynek’s video on NOX vs TOX – WHAT are they for & HOW do you CHOOSE? 🐍

https://www.youtube.com/watch?v=ImBvrDvK-1U

Both Nox and Tox are great tools that automate testing in multiple Python environments.

I prefer Nox because it uses Python to write configs, which fits my brain better. I used Tox for over a decade, and there are some tox.ini files that I dread updating because I can only remember how I got here after a few hours of tinkering. That’s not Tox’s fault. I think that’s just a limitation of ini files and the frustration that comes from being unable to use Python when you have a complex matrix to try and sort out.

I recommend trying them out and using the best tool for your brain. There is no wrong path here.

PS: Thank you, Josh, for bringing this to my attention.

🎒 Everyone struggles with Django's static files

Tue, 30 Apr 2024 21:07:44 -0600

Josh Thomas did a great job documenting and walking us through how he prefers to set up static files in his Django projects last week in his How I organize staticfiles in my Django projects article.

Josh recommends the following config and naming convention:

# settings.py

# django.contrib.staticfiles
STATIC_ROOT = BASE_DIR / "staticfiles"

STATIC_URL = "/static/"

STATICFILES_DIRS = [
    BASE_DIR / "static" / "dist",
    BASE_DIR / "static" / "public",
]

Overall, this is very similar to what I do, but I settled into calling my STATICFILES_DIRS folder assets or frontend. After seeing Josh’s example, I changed this value to staticfiles to match the setting variable more closely.

Updated config

# settings.py

# django.contrib.staticfiles

# INPUT: Where to look for static files
STATICFILES_DIRS = [
    BASE_DIR / "staticfiles" / "dist",
    BASE_DIR / "staticfiles" / "public",
]

# OUTPUT: Where to put and look for static files to serve
STATIC_ROOT = BASE_DIR / "static"

# SERVE: Where to serve static files
STATIC_URL = "/static/"

This also changes our .gitignore to match our new settings. Since all of our files will be collected by Django and placed into the static folder, we can tell git to ignore this folder.

We can also ignore the staticfiles/dist/ folder if we have an asset building pipeline and need a place to store the intermediate files.

#.gitignore
static/
staticfiles/dist/
# Optionally, to universally ignore all 'dist' directories:
# dist

💸 Alphabet has never been more profitable

Mon, 29 Apr 2024 11:48:27 -0600

This Hacker News comment is quite the deep cut about Google’s recent layoff/firing of the entire Python Foundation team.

This happened the same day Alphabet reported $80.54 billion in revenue.

in addition to contributing to upstream python, we

maintained a stable version of python within google, and made sure that everything in the monorepo worked with it. in my time on the team we moved from 2.7 to 3.6, then incrementally to 3.11, each update taking months to over a year because the rule at google is if you check any code in, you are responsible for every single breakage it causes

maintained tools to keep thousands of third party packages constantly updated from their open source versions, with patch queues for the ones that needed google-specific changes

had highly customised versions of tools like pylint and black, targeted to google’s style guide and overall codebase

contributed to pybind11, and maintained tools for c++ integration

developed and maintained build system rules for python, including a large effort to move python rules to pure starlark code rather than having them entangled in the blaze/bazel core engine

developed and maintained a typechecker (pytype) that would do inference on code without type annotations, and work over very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama)

performed automated refactorings across hundreds of millions of lines of code

and that was just the dev portion of our jobs. we also acted as a help desk of sorts for python users at google, helping troubleshoot tricky issues, and point newcomers in the right direction. plus we worked with a lot of other teams, including the machine learning and AI teams, the colaboratory and IDE teams, teams like protobuf that integrated with and generated python bindings, teams like google cloud who wanted to offer python runtimes to their customers, teams like youtube who had an unusually large system built in python and needed to do extraordinary things to keep it performant and maintainable.

and we did all this for years with fewer than 10 people, most of whom loved the work and the team so much that we just stayed on it for years. also, despite the understaffing, we had managers who were extremely good about maintaining work/life balance and the “marathon, not sprint” approach to work. as i said in another comment, it’s the best job i’ve ever had, and i’ll miss it deeply.

They have never been more profitable, made more money, or cared less about their employees and giving back to the communities that got them there. If I had worked at Google, I would not have been comfortable continuing to work there.

I plan to keep de-Googling, which is going to hurt. Several friends shared why they can’t cut ties with their products. I get it, but that is a sign of an unhealthy monopoly.

I pay for YouTube TV and YouTube Premium as everyone else does, but I’m starting with the business tools. I’d rather pay Zoom than use Google Video. I stopped using Chrome.

Gmail is the hardest service to separate from because it’s also an identity service. I’m using it less and less until it’s only an identity service. I have >2k websites registered with my Gmail address, which could take years to clean up. I don’t need to pay for extra storage space, and I’m wondering if Google Docs is doing anything for me to pay for.

So, while it will be harder to stop using Google Products than it should be, the first step is admitting the problem and then working on it. Friends, we all have work to do.

💚 DjangoCon US Talks I'd Like to See 2024 Edition

Sun, 28 Apr 2024 18:45:13 -0600

Continuing in my tradition of 2023, 2022, 2018, 2017, and 2016 “DjangoCon US Talks I’d like to see” annual posts, here is my update for 2024.

This year’s updated talk and tutorial deadline is April 29, 2024, at 12 PM EDT, but check the official speaking page for updates and submit your proposal.

Before you read mine, I suggest you check out Kati Michel’s and Drew Winstel’s list of talk ideas too.

Modern Django

Three or four years ago, our Django stack was optimized to leverage the Django ORM to tie together a mix of authentication and REST API using the Django REST Framework. The front end was entirely buried by JavaScript, which printed HTML into your browser, and the HTML was entirely hidden away. What used to take a developer hours and days now took weeks because there were so many layers and abstractions to wade through.

Then it all started to change a few years ago, and we learned that we don’t need the additional weeks of development time to use JS/JSON layers for which we may never build that iOS/Android app. We don’t need React for contact us forms, and 90% of our web tasks could be built in a day.

The theme I’d like to see more of this year is doing more with HTML instead of hiding it behind layers of JavaScript.

I want to see lots of talks about technologies like HTMx and frameworks like Tailwind CSS, which are firmly centered inside HTML, instead of trying to avoid using HTML.

The two non-Python languages we are starting to talk about

Rust and WASM are starting to change everything. How do they fit in with Django and Python?

Theme: Rust

The Rust language is shaping the landscape of Python productivity in ways that none of us could have predicted. Tools written in Rust are significantly faster and are gaining steam.

What can a Django developer take from Rust?
How do I develop a package for Django and Python that uses Rust?
How libraries like PyDantic are speeding up development (Django Ninja uses it)

Always nice to have talks

What’s new in Django 5.x?
What’s new in Python 3.13?
What’s new in Wagtail CMS and Django CMS?
Where do we want the DSF to go?
How to contribute to Django or third party projects

Embracing HTML and lightweight front-end frameworks

Last year, I predicted it would be the year of “you probably don’t need JavaScript,” and I still feel good about that. We saw HTMx take off, but I think Django is still underserved and could benefit from more talks:

Django 5 added more granular template/form styling , and we could use a talk that deep dives
Using Django forms with htmx
Using search with htmx
Whatever cool things you are doing that I didn’t mention here

Panel ideas

Django Fellows panel
Django content creators' panel
Django apps and community - It would be nice to hear from the more significant apps and community projects.

Wagtail CMS

How do I migrate from WordPress to Wagtail CMS?
How to properly test with Wagtail CMS for me, because I struggle with it

Talks I don’t want to see

I find myself at a weird crossroads because I have never discouraged certain topics, but we need to talk more openly about what is bad for our industry and the world instead of avoiding it. Otherwise, we are left with dead air, and people assume we are all on the same page.

To kick us off, there are talks and topics that I do not want to see:

BAD AI talks and ones that spread more FUD into the community
crypto talks
why technology sucks
proposing a talk because you dislike or even hate something
bashing other communities
bashing our communities
lightning talks disguised as questions (not so much a talk idea, but this has been on my mind for a while)

So please, none of these:

The future of Django depends on AI
Why tech needs AI
Crypto talks
Your product unless you are sponsoring.

Good luck

You have less than a day, but between my list, Kati’s list, and Drew’s list you have no excuses for saying you didn’t know what to talk about at DjangoCon US this year.

Good luck!

🤖 Super Bot Fight 🥊

Thu, 18 Apr 2024 22:41:03 -0600

In March, I wrote about my robots.txt research and how I started proactively and defensively blocking AI Agents in my 🤖 On Robots.txt. Since March, I have updated my Django projects to add more robots.txt rules.

Earlier this week, I ran across this Blockin’ bots. blog post and this example, the mod_rewrite rule blocks AI Agents via their User-Agent strings.

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
# block “AI” bots
RewriteCond %{HTTP_USER_AGENT} (AdsBot-Google|Amazonbot|anthropic-ai|Applebot|AwarioRssBot|AwarioSmartBot|Bytespider|CCBot|ChatGPT|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|DataForSeoBot|Diffbot|FacebookBot|FacebookBot|Google-Extended|GPTBot|ImagesiftBot|magpie-crawler|omgili|Omgilibot|peer39_crawler|PerplexityBot|YouBot) [NC]
RewriteRule ^ – [F]
</IfModule>

Since none of my projects use Apache, and I was short on time, I decided to leave this war to the bots.

Django Middleware

I asked ChatGPT to convert this snippet to a piece of Django Middleware called Super Bot Fight. After all, if we don’t have time to keep up with bots, then we could leverage this technology to help fight against them.

In theory, this snippet passed my eyeball test and was good enough:

# middleware.py

from django.http import HttpResponseForbidden

# List of user agents to block

BLOCKED_USER_AGENTS = [
    "AdsBot-Google",
    "Amazonbot",
    "anthropic-ai",
    "Applebot",
    "AwarioRssBot",
    "AwarioSmartBot",
    "Bytespider",
    "CCBot",
    "ChatGPT",
    "ChatGPT-User",
    "Claude-Web",
    "ClaudeBot",
    "cohere-ai",
    "DataForSeoBot",
    "Diffbot",
    "FacebookBot",
    "Google-Extended",
    "GPTBot",
    "ImagesiftBot",
    "magpie-crawler",
    "omgili",
    "Omgilibot",
    "peer39_crawler",
    "PerplexityBot",
    "YouBot",
]

class BlockBotsMiddleware:

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        # Check the User-Agent against the blocked list
        user_agent = request.META.get("HTTP_USER_AGENT", "")
        if any(bot in user_agent for bot in BLOCKED_USER_AGENTS):
            return HttpResponseForbidden("Access denied")
        response = self.get_response(request)
        return response

To use this middleware, you would update your Django settings.py to add it to your MIDDLEWARE setting.

# settings.py

MIDDLEWARE = [
    ...
    "middleware.BlockBotsMiddleware",
    ...
]

Tests?

If this middleware works for you and you care about testing, then these tests should also work:


import pytest

from django.http import HttpRequest
from django.test import RequestFactory

from middleware import BlockBotsMiddleware

@pytest.mark.parametrize("user_agent, should_block", [
    ("AdsBot-Google", True),
    ("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", False),
    ("ChatGPT-User", True),
    ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3", False),
])
def test_user_agent_blocking(user_agent, should_block):
    # Create a request factory to generate request instances
    factory = RequestFactory()
    request = factory.get('/', HTTP_USER_AGENT=user_agent)

    # Middleware setup
    middleware = BlockBotsMiddleware(get_response=lambda request: HttpResponse())
    response = middleware(request)

    # Check if the response should be blocked or allowed
    if should_block:
        assert response.status_code == 403, f"Request with user agent '{user_agent}' should be blocked."
    else:
        assert response.status_code != 403, f"Request with user agent '{user_agent}' should not be blocked."

Enhancements

To use this code in production, I would normalize the user_agent and BLOCKED_USER_AGENTS variables to be case-insensitive.

I would also consider storing my list of user agents in a Django model or using a project like django-robots instead of a hard-coded Python list.

🚀 Starship is the shell prompt you have been looking for 🐚

Wed, 17 Apr 2024 20:37:05 -0600

I spent a lot of time in my shell (terminal, iTerm, bash, whatever you prefer), and by and large, my least favorite and most frustrating chore has always been trying to customize my shell prompt.

A few years ago, I tried to switch from BASH to Zsh and even the Fish Shell, and I quickly reverted because I found customizing my prompt to be such a chore.

Starship is one of those few life-changing development utilities that quickly solved how to customize my shell prompt. Starship also works across various shells, which means changing shells doesn’t mean starting over with a new prompt. Even though Fish and Zsh never stuck for me, Starship was a keeper.

Starship’s default prompt is really good and is a nicer starting point than Zsh or BASH gives you by default. Starship includes a bunch of default settings, which enables support for tools like aws, battery percentage, bun, git, k8s, node js, python, etc, so most of my dotfiles config is turning off the features that I don’t care about.

There is a Dracula Theme for Starship, which is my go-to theme for any application that I can find that supports the color scheme.

Setting up and configuring Starship is a breeze. The documentation is clear and concise, and Starship’s settings are neatly organized in one readable and updated file, ~/.config/starship.toml. Best of all, any changes you make are immediately applied, eliminating the need to restart your shell session. With Starship, you can confidently customize your prompt and see the results instantly.

If you use Homebrew, brew install starship will install it. You can activate Starship by adding eval "$(starship init bash)" to your ~/.bashrc file.

If you have been thinking about trying out Starship or need help figuring out where to start or how to customize your shell prompt, I highly recommend it.

🚜 Refactoring and fiddling with Django migrations for pending pull requests 🐘

Sat, 06 Apr 2024 13:59:54 -0600

One of Django’s most powerful features is the ORM, which includes a robust migration framework. One of Django’s most misunderstood features is Django migrations because it just works 99% of the time.

Even when working solo, Django migrations are highly reliable, working 99.9% of the time and offering better uptime than most web services you may have used last week.

The most common stumbling block for developers of all skill levels is rolling back a Django migration and prepping a pull request for review.

I’m not picky about pull requests or git commit history because I default to using the “Squash and merge” feature to turn all pull request commits into one merge commit. The merge commit tells me when, what, and why something changed if I need extra context.

I am pickier about seeing >2 database migrations for any app unless a data migration is involved. It’s common to see 4 to 20 migrations when someone works on a database feature for a week. Most of the changes tend to be fiddly, where someone adds a field, renames the field, renames it again, and then starts using it, which prompts another null=True change followed by a blank=True migration.

For small databases, none of this matters.

For a database with 10s or 100s of millions of records, these small changes can cause minutes of downtime per migration, which amounts to a throwaway change. While there are ways to mitigate most migration downtime situations, that’s different from my point today.

I’m also guilty of being fiddly with my Django model changes because I know I can delete and refactor them before requesting approval. The process I use is probably worth sharing because once every new client comes up.

Let’s assume I am working on Django News Jobs, and I am looking over my pull request one last time before I ask someone to review it. That’s when I noticed four migrations that could quickly be rebuilt into one, starting with my 0020* migration in my jobs app.

The rough steps that I would do are:

# step 1: see the state of our migrations
$ python -m manage showmigrations jobs
jobs
 [X] 0001_initial
 ...
 [X] 0019_alter_iowa_versus_unconn
 [X] 0020_alter_something_i_should_delete
 [X] 0021_alter_uconn_didnt_foul
 [X] 0022_alter_nevermind_uconn_cant_rebound
 [X] 0023_alter_iowa_beats_uconn
 [X] 0024_alter_south_carolina_sunday_by_four

# step 2: rollback migrations to our last "good" state
$ python -m manage migrate jobs 0019

# step 3: delete our new migrations
$ rm jobs/migrations/002*

# step 4: rebuild migrations 
python -m manage makemigrations jobs 

# step 5: profit 
python -m manage migrate jobs

95% of the time, this is all I ever need to do.

Occasionally, I check out another branch with conflicting migrations, and I’ll get my local database in a weird state.

In those cases, check out the --fake (“Mark migrations as run without actually running them.") and --prune (“Delete nonexistent migrations from the django_migrations table.") options. The fake and prune operations saved me several times when my django_migrations table was out of sync, and I knew that SQL tables were already altered.

What not `squashmigrations`?

Excellent question. Squashing migrations is wonderful if you care about keeping every or most of the operations each migration is doing. Most of the time, I do not, so I overlook it.

⛳ Syncing Django Waffle feature flags

Fri, 05 Apr 2024 20:58:28 -0600

The django-waffle feature flag library is helpful for projects where we want to release and test new features in production and have a controlled rollout. I also like using feature flags for resource-intensive features on a website that we want to toggle off during high-traffic periods. It’s a nice escape hatch to fall back on if we need to turn off a feature and roll out a fix without taking down your website.

While Waffle is a powerful tool, I understand the challenge of keeping track of feature flags in both code and the database. It’s a pain point that many of us have experienced.

Waffle has a WAFFLE_CREATE_MISSING_FLAGS=True setting that we can use to tell Waffle to create any missing flags in the database should it find one. While this helps discover which flags our application is using, we need to figure out how to clean up old flags in the long term.

The pattern I landed on combines storing all our known feature flags and a note about what they do in our main settings file.

# settings.py
... 

WAFFLE_CREATE_MISSING_FLAGS=True

WAFFLE_FEATURE_FLAGS = {
   "flag_one": "This is a note about flag_one",
   "flag_two": "This is a note about flag_two",
}

We will use a management command to sync every feature flag we have listed in our settings file, and then we will clean up any missing feature flags.

# management/commands/sync_feature_flags.py
import djclick as click

from django.conf import settings
from waffle.models import Flag


@click()
def command():
    # Create flags that don't exist
    for name, note in settings.WAFFLE_FEATURE_FLAGS.items():
        flag, created = Flag.objects.update_or_create(
            name=name, defaults={"note": note}
        )
        if created:
            print(f"Created flag {name} ({flag.pk})")

    # Delete flags that are no longer registered in settings
    for flag in Flag.objects.exclude(name__in=settings.FEATURE_FLAGS.keys()):
        flag.delete()
        print(f"Deleted flag {flag.name} ({flag.pk})")

We can use the WAFFLE_CREATE_MISSING_FLAGS settings as a failsafe to create any flags we might have accidently missed. They will stick out because they will not have a note associated with them.

This pattern is also helpful in solving similar problems for scheduled tasks, which might also store their schedules in the database.

Check out this example in the Django Styleguide for how to sync Celery’s scheduled tasks.

⬆️ The Upgrade Django project

Thu, 04 Apr 2024 18:37:35 -0600

Upgrade Django is a REVSYS project we created six years ago and launched three years ago.

The goal of Upgrade Django was to create a resource that made it easy to see at a glance which versions of the Django web framework are maintained and supported. We also wanted to catalog every release and common gotchas and link to helpful information like release notes, blog posts, and the tagged git branch on GitHub.

We also wanted to make it easier to tell how long a given version of Django would be supported and what phase of its release cycle it is in.

Future features

We have over a dozen features planned, but it’s a project that primarily serves its original purpose.

One feature on my list is that I’d love to see every backward incompatible change between two Django versions. This way, if someone knows their website is running on Django 3.2, they could pick Django 4.2 or Django 5.0 version and get a comprehensive list with links to everything they need to upgrade between versions.

Projects like Upgrade Django are fun to work on because once you collect a bunch of data and start working with it, new ways of comparing and presenting the information become more apparent.

If you have ideas for improving Upgrade Django that would be useful to your needs, we’d love to hear about them.

🚀 Mise is really fast

Wed, 03 Apr 2024 18:27:03 -0600

A month or two ago, I stumbled on justpath, which is worth installing to help clean up any PATH environment issues you may or may not realize you have.

I was frustrated because every time I created a new tab in iTerm2, it took three or more seconds for whatever slowed everything in my bash/dotfiles.

With justpath, I found it straightforward to identify and address several paths related to ruby, rbenv, cargo, golang, and others that were causing issues. Rather than continue to fight these apps, I deleted them and removed them from my .bash* files. To see if I broke anything, I opened a new iTerm tab, and I had a new shell prompt in less than a second.

Since I needed access to Ruby, NPM, and other tools, I switched over to mise, which promises to be one tool for managing these languages and tools. mise is also written in Rust, so I know it’s fast by default.

To get started, check out Mise’s Quickstart docs in their README. I modified their bash example and added it to my .bash* files.

if command -v mise > /dev/null 2>&1; then
    eval "$(mise activate bash)";
fi

Once Mise was set up, and I opened a new tab, I installed Go, Node.js, Ruby, and Rust:

# to install Go
mise install golang

# to install Node.js
mise install node

# to install Ruby
mise install ruby

# to install Rust
mise install rust

# to reshim/add these tool to our path...
mise reshim

Eventually, I used Mise to install kubectl (boo) and yarn (hiss). Here is a list of everything on my older MacBook Pro that Mise is managing for me.

$ mise list
Plugin   Version  Config Source    Requested
go       1.22.2
kubectl  1.22.2
node     20.10.0  ~/.tool-versions 20.10.0
ruby     3.3.0
rust     1.77.1
yarn     1.22.19

Another nice feature of Mise is that you can list the versions of each tool that you want Mise to install and manage for you in a .tool-versions file.

$ cat ~/.tool-versions
# direnv latest
# golang latest
# kubectl 1.22.2
# nodejs 18.9.1
# nodejs lts
# ruby 3.1.2
# starship latest
# yarn 1.22.19
nodejs 20.10.0

As a nice bonus, I discovered I could use Mise to manage the direnv and starship tools. In theory, Mise can also replace direnv with their built-in environment support), but I haven’t tried that feature yet.

Results

Mise is a nice win. I cut my new iTerm2 tab/session time down to close to 1/4 of a second, and I have more granular control over which exact versions of each tool I’m using.

Overall, I still prefer to use pyenv to manage my Python versions, so I’m keeping that for now mainly because I use pyenv-virtualenvwrapper and a few other nice-to-haves. See my Python Development on macOS Notes: pyenv and pyenv-virtualenvwrapper article on why and how I manage those.

🔥 An update on OpenSSF's XZ fumble

Mon, 01 Apr 2024 21:07:39 -0600

OpenSSF’s Scorecards is Going Just Great 🔥

While I’m happy that OpenSSF updated their blog post to remove all mentions of their scorecards, they forgot to mention why, apologize, or publicly acknowledge that they messed up.

Since they rushed their first blog post and spent so little time on it, I wanted to point out the changes and equally spend very little time on why their scorecard is actively harmful to projects that handle security well.

Today was a rare day off for me, and documenting the numerous red flags of OpenSSF’s Scorecards isn’t how I spent my Monday night, nor will I spend my Tuesday or any night on this.

Instead, check out the Scorecards for these three popular projects.

Flask scores a 6.9
cpython scores a 7.0
Django scores a 7.2

If you aren’t sure where to start, compare the “Pinned-Dependencies” and “Best-Practices” over the three projects. Then, look at the reference files and see if you agree.

🤷

Updates

This [email protected]/112198968259149635">post by David Lord is worth a star.

Things I can never remember how to do: Django Signals edition

Mon, 25 Mar 2024 15:13:15 -0600

I am several weeks into working on a project with my colleague, Lacey Henschel. Today, while reviewing one of her pull requests, I was reminded how to test a Django Signal via mocking.

Testing Django signals is valuable to me because I need help remembering how to test a signal, and even with lots of effort, it never works. So bookmark this one, friends. It works.

Thankfully, she wrote it up in one of her TIL: How I set up django-activity-stream, including a simple test

[email protected]">https://mastodon.social/@[email protected]

2024-W12 - Weeknotes for Week 12: March 18 to March 24

Sun, 24 Mar 2024 20:02:23 -0600

Family

Spring Break is over for my kids, and they both returned to school. Both kids seemed healthy again, but then we had a doctor’s appointment on Thursday and a follow-up appointment on Friday. Everything was fine, and everyone was healthy.

Work

It was a short week for us. I spent most of my week working on the green field project again and then on some internal projects.

Community

I only had a little time for community projects this week. What little time I did have was spent at the bank getting paperwork for the next DEFNA board.

Side projects

Django News Newsletter: We shipped issue #224.

Django News Jobs: I fixed/simplified a few layout issues. I also spoke to a few companies that use the service, changed how jobs are submitted, and cleaned up some of the messaging around it.

Django News Portal: Django News has a portal I have never publicly launched. I use it for some of my contributions and to double-check that we get all the big releases. I spend some time working on it over my break.

Django News Browser Extension: We also have an unreleased Browser Extension I have used for three or more years. I used it with Chrome, but since switching to Vivaldi, I have yet to figure out how to make it work as my new page. I might release this in April.

Django Packages: We only had a few minor updates this week. I fixed UV, and we improved Sentry profiling.

Fun projects: I dabbled with a Bootstrap to Tailwind CSS converter project. I also dusted off an old Git Bot project I wrote about in my On Scratching itches with Python post.

Writing

I wrote and published every day.

2024-03-23: On scratching itches with Python

2024-03-22: Automated Python and Django upgrades

2024-03-21: Fixing a CSS layout issue

2024-03-20: 🤖 On Robots.txt

2024-03-19: Justfile Alfred Plugin

2024-03-18: Weeknotes for Week 11: March 11 to March 17

“Me” time

I took my annual March Madness break and watched basketball from Thursday through Sunday.

My Jayhawks Men’s basketball team won the first round but fell in the second round, while the Jayhawks Women’s team was still in their respective tournament. Overall, it was a rare down year for the men, and it was great to see the women make the tournament and knock off a good team to advance to the Round of 32.

Next week

I was asked to appear on two different podcasts, and I’ll be recording those next week.

As of tonight, I plan to host Office Hours next week, but someone mentioned it being a holiday weekend. I will figure that out sooner than later this week.

On scratching itches with Python

Sat, 23 Mar 2024 22:26:41 -0600

Python is such a fantastic glue language. Last night, while watching March Madness basketball games, I had a programming itch I wanted to scratch.

I dusted off a demo I wrote several years ago. It used Python’s subprocess module, which strings together a bunch of shell commands to perform a git checkout, run a few commands, and then commit the results. The script worked, but I struggled to get it fully working in a production environment.

To clean things up and as an excuse to try out a new third-party package, I converted the script to use:

GitPython - GitPython is a Python library used to interact with Git repositories.
Shelmet - A shell power-up for working with the file system and running subprocess commands.
Django Q2 - A multiprocessing distributed task queue for Django based on Django-Q.

Using Django might have been overkill, but having a Repository model to work with felt nice. Django Q2 was also overkill, but if I put this app into production, I’ll want a task queue, and Django Q2 has a manageable amount of overhead.

GitPython was a nice improvement over calling git commands directly because their API makes it easier to see which files were modified and to check against existing branch names. I was happy with the results after porting my subprocess commands to the GitPython API.

The final package I used is a new package called Shelmet, which was both a nice wrapper around subprocess plus they have a nice API for file system operations in the same vein as Python’s Pathlib module.

Future goals

I was tempted to cobble together a GitHub bot, but I didn’t need one. I might dabble with the GitHub API more to fork a repo, but for now, this landed in a better place, so when I pick it back up again in a year, I’m starting in a good place.

If you want to write a GitHub bot, check out Mariatta’s black_out project.

Automated Python and Django upgrades

Fri, 22 Mar 2024 10:02:57 -0600

Recently, I have been maintaining forks for several projects that are no longer maintained. Usually, these are a pain to update, but I have found a workflow that takes the edge off by leveraging pre-commit.

My process:

Fork the project on GitHub to whichever organization I work with or my personal account.
Check out a local copy of my forked copy with git.
Install pre-commit
Create a .pre-commit-config.yaml with ZERO formatting or lint changes. This file will only include django-upgrade and pyupgrade hooks.

We skip the formatters and linters to avoid unnecessary changes if we want to open a pull request in the upstream project. If the project isn’t abandoned, we will want to do that.

For django-upgrade, change the—-target-version option to target the latest version of Django I’m upgrading to, which is currently 5.0.
For pyupgrade, update the python settings under default_language_version to the latest version of Python that I’m targetting. Currently, that’s 3.12.

The django-upgrade and pyupgrade projects attempt to run several code formatters and can handle most of the more tedious upgrade steps.

Run pre-commit autoupdate to ensure we have the latest version of our hooks.
Run pre-commit run --all-files to run pyupgrade and django-upgrade on our project.
Run any tests contained in the project and review all changes.
Once I’m comfortable with the changes, I commit them all via git and push them upstream to my branch.

Example `.pre-commit-config.yaml` config

From my experience, less is more with this bane bones .pre-commit-config.yaml config file.

# .pre-commit-config.yaml

default_language_version:
  python: python3.12

repos:
  - repo: https://github.com/asottile/pyupgrade
    rev: v3.15.1
    hooks:
      - id: pyupgrade

  - repo: https://github.com/adamchainz/django-upgrade
    rev: 1.16.0
    hooks:
      - id: django-upgrade
        args: [--target-version, "5.0"]

If I’m comfortable that the project is abandoned, I’ll add ruff support with a more opinionated config to ease my maintenance burden going forward.

Justfile Alfred Plugin

Tue, 19 Mar 2024 11:49:25 -0600

A few years back, I had a productivity conversation with Jay Miller about Alfred plugins, which led to him sharing his Bunch_Alfred plugin. At the time, I played around with the Bunch.app, a macOS automation tool, and Alfred’s support was interesting.

I created my Alfred plugin to run Just command runner commands through my Alfred setup. However, I never got around to packing or writing the plugin’s documentation.

My Alfred plugin runs Script Filter Input, which reads from a centrally located justfile and generates JSON output of all of the possible options. This will be displayed, and Alfred will run that command, whichever option you select.

I was always unhappy with how the JSON document was generated from my commands, so I dusted off the project over lunch and re-engineered it by adding Pydantic support.

Alfred just announced support for a new User Interface called Text View, which could make text and markdown output from Python an exciting way to handle snippets and other productive use cases. I couldn’t quite figure it out over lunch, but now I know it’s possible, and I might figure out how to convert my Justfile Alfred plugin to generate better output.

Weeknotes for Week 11: March 11 to March 17

Mon, 18 Mar 2024 18:33:40 -0600

Family

This week was Spring Break for both of my kids. My son had all week off, while my daughter had Thursday and Friday off. My son was also sick and hanging out with me the first two days of the week instead of getting to go to day camp.

I worked on several house projects, including cleaning our smoker and outdoor griddle. I didn’t winterize our smoker nearly as well as I thought I had, so it took me much longer to clean than it should have.

Sunday was cold and windy, so I couldn’t do as much outside as I planned.

Work

I spent most of my week working on the green field project again.

I also hosted a rather minute(ish) Office Hours session again.

Community

I had two DjangoCon US meetings this week. One is to discuss our online experience, and the other is to kick off our new website and eventual new website framework.

Side projects

I spent most of my side project time on Django News, Django News Jobs, and Django Packages. I also upgraded several project configs to use some new uv options.

Django Packages: We updated our profiles to drop support for Google Code and add Bitbucket and GitLab profiles.

Trailhawks: My local running club’s website has been running on Django since 2009. I upgraded the project from Django 3.2 to 5.0 and to Python 3.1x, including upgrading two outdated third-party packages to Python 3.1x and Django 5. I also changed one of our race websites to Tailwind CSS, which has been in flight for a while.

Django Packages: We released an update to our profiles to drop support for Google Code and to add Bitbucket and GitLab profiles. I also broke the website with a wsgi change, which I quickly found a fixed.

**django-startproject: ** I updated django-startproject project to have better support for uv.

Django News Newsletter: We shipped issue #223.

Writing

I wrote and published every day.

2024-03-17: 🍕 On tortilla pizza

2024-03-16: 🏡 On Gardening

2024-03-15: Five for Friyay

2024-03-14: Python’s UV tool is even better

2024-03-13: On environment variables and dotenv files

2024-03-12: Completed, finished, or done

2024-03-11: My Python Roots

“Me” time

🏀 I watched Kansas get swept in the first round of the Big 12 tournament because our best two players were out with injuries. I set aside time on Wednesday and Thursday afternoons to watch some conference tournaments, but they were all blowouts.

Next week

🚸 My kids are back in school, and our schedules return to regular routines.

🪚 I have more rot repair, and I now have a compound miter saw to cut wood for framing and fixing things. I had to order a portable workbench to attach the miter saw in mid-week.

🏀 Next week also starts the NCAA tournament, a sign that the third basketball season has begun. I am taking two days off to enjoy the madness, and I suspect this will be most of my weekend, too.

👟 I should have to get the other half of the Trailhawks website out the door.

Five for Friyay

Fri, 15 Mar 2024 19:05:35 -0600

It’s been too long since I posted my five favorite links of the week. Continuing that theme for today, here are five links from my Friyay (Friday).

1. A new code editor

Tonight, I installed the Zed code editor on a whim, and to my surprise, my LSP was automatically picked up and just worked. Zed was a surprisingly productive experience. I even spotted a bug in one of my projects that I needed help to figure out earlier today.

2. My favorite theme

I wasn’t impressed with Zed’s default theme, so I searched the Dracula website and found a Dracula Theme for Zed. Dracula is one of my favorite themes, and they have custom-built themes for over 370 different apps.

3. A new to me Python library

shelmet is a nice wrapper around Python’s subprocess. The API is clean, and it has a nice wrapper around using temporary environment variables, which has been helpful for me.

4. The best presentation tool

Deckset is the best tool for quickly writing nice-looking presentations using markdown. After years of being in private beta, they released their iOS/iPad version publicly.

5. A long read for the weekend

This exhaustive review of the Vision Pro is everything I have been looking for and more to get a feel for what the Vision Pro can do.

Have thoughts?

Python's UV tool is even better

Thu, 14 Mar 2024 10:51:21 -0600

Last month, I wrote Python’s UV tool is actually pretty good about Astral’s new Python package installer and resolver uv, and this is a follow-up post.

Since last month, I have added uv to over a dozen projects, and I recently learned that you could skip the venv step for projects that use containers or CI where the environment is already isolated.

I mistakenly thought uv required a virtual environment (aka venv), but Josh Thomas recently pointed out that it’s unnecessary.

The trick is to pass the --system option, and uv will perform a system-wide install. Here’s an example:

uv pip install --system --requirement requirement.txt

Now that I have seen this, I wish pip also used this approach to avoid developers accidentally installing third-party packages globally.

local development

Nothing has changed with my justfile example from last month.

When I’m working with containers, I create a virtual environment (venv) because I will need most of my project requirements installed outside of the container so that my text editor and LSP can resolve dependencies. uv’s default behavior of respecting a venv is all we need here.

Every one of my projects has a justfile (it’s like Make but works the same everywhere) with “bootstrap” and “lock” recipes. My “bootstrap” recipe installs everything I need to work with the project locally. I use my “lock” recipe to lock my requirements.txt file to use the exact requirements locally and in production.

`justfile` before

My justfile might look like this:

@bootstrap
    python -m pip install --upgrade pip
    python -m pip install --upgrade --requirement requirements.in
    
@lock *ARGS:
    python -m piptools compile {{ ARGS }} ./requirements.in \
        --resolver=backtracking \
        --output-file requirements.txt

`justfile` after

For the most part, uv shares most of the same syntax as pip so you can start by changing your pip references to uv pip:

@bootstrap
    python -m pip install --upgrade pip uv
    python -m uv pip install --upgrade --requirement requirements.in
    
@lock *ARGS:
    python -m uv pip compile {{ ARGS }} requirements.in \
        --resolver=backtracking \
        --output-file requirements.txt

Dockerfiles

Everyone’s container setup is going to be different, but I use Docker and Orbstack, which use a Dockerfile.

`Dockerfile` before

FROM python:3.12-slim-bookworm

ENV PIP_DISABLE_PIP_VERSION_CHECK 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONPATH /srv
ENV PYTHONUNBUFFERED 1

RUN apt-get update

RUN pip install --upgrade pip

COPY requirements.txt /src/requirements.txt

RUN pip install --requirement /src/requirements.txt

WORKDIR /src/

`Dockerfile` after

FROM python:3.12-slim-bookworm

ENV PIP_DISABLE_PIP_VERSION_CHECK 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONPATH /srv
ENV PYTHONUNBUFFERED 1

RUN apt-get update

RUN pip install --upgrade pip uv  # this is updated

COPY requirements.txt /src/requirements.txt

RUN uv pip install --system --requirement /src/requirements.txt  # this is updated

WORKDIR /src/

GitHub Actions

GitHub Actions are a little more complicated to explain, but my workflows started similar to this before I made the switch to uv:

`main.yml` before

  - name: Set up Python 3.12
    uses: actions/setup-python@v5
    with:
      python-version: '3.12'

  - name: Install dependencies
    run: |
            python -m pip install --requirement requirements.in

`main.yml` after

The most significant pain point I ran into was related to GitHub Issue #1386, which has a useable workaround.

  - name: Set up Python 3.12
    uses: actions/setup-python@v5
    with:
      python-version: '3.12'

  - name: Install dependencies
    run: |
      python -m pip install --upgrade uv  # this is new
      python -m uv pip install --system --requirement requirements.in  # this is updated

Gotchas

The only gotchas I have encountered with uv is when I’m trying to install a Python package from a remote zip file.

Previously, I could copy and paste the GitHub repo URL, but uv required we use the format package-name @ url-to-zip-file

`requirements.in` before

# requirements.in
https://github.com/jefftriplett/django-feedreader/archive/main.zip

`requirements.in` after

# requirements.in
django-feedreader @ https://github.com/jefftriplett/django-feedreader/archive/main.zip

Conclusion

This update helps remove a few steps from updating your projects, and it should shave a few minutes off of updating projects to use it.

I hope this was helpful to anyone who is considering making the switch to uv. I love to hear about how much time it saves you.

On environment variables and dotenv files

Wed, 13 Mar 2024 20:23:26 -0600

Brett Cannon recently vented some frustrations about .env files.

I still hate .env files and their lack of a standard

[email protected]/112056455108582204">https://mastodon.social/@[email protected]/112056455108582204

Brett’s thread and our conversation reminded me that my rule for working with dotenv files is to have my environment load them instead of my Python app trying to read from the .env file directly.

What is a `.env` (dotenv) file?

A .env (aka dotenv) is a file that contains a list of key-value pairs in the format of {key}=value.

At a basic level, this is what a bare minimum .env file might look for in a Django project.

# .env
DEBUG=true
SECRET_KEY=you need to change this

My go-to library for reading ENV variables is environs. While the environs library can read directly from a dotenv file, don’t do that. I never want my program to read from a file in production because I don’t want a physical file with all of my API keys and secrets.

Most hosting providers, like Fly.io, have a command line interface for setting these key-value pairs in production to avoid needing a physical dotenv file.

Instead, we should default to assuming that the ENV variables will bet in our environment, and we should fall back to either a reasonable default value or fail loudly.

Using the environs library, my Django settings.py file tends to look like this:

# settings.py
import environs

env = environs.Env()

# this will default to False if not set.
DEBUG = env.bool("DJANGO_DEBUG", default=False)

# this will error loudly if not set
SECRET_KEY = env.str("SECRET_KEY")

# everything else...

I lean on Docker Compose for local development when I’m building web apps because I might have three to five services running. Compose can read a dotenv file and register them into environment variables.

`.envrc` files aren’t `.env` files

On my macOS, when I’m not developing in a container, I use the direnv application to read an .envrc file which is very similar to a dotenv file.

A .envrc is very similar to a .env file, but to register the values into memory, you have to use Bash’s export convention. If you don’t specify export, the environment variables won’t be available in your existing Bash environment.

# .envrc
export DEBUG=true
export SECRET_KEY=you need to change this

I’m a fan of direnv because the utility ensures that my environment variables are only set while I am in the same folder or sub-folders that contain the .envrc file. If I move to a different folder location or project, direnv will automatically unload every environment variable that was previously set.

This has saved me numerous times over the years when I have run a command that might upload a file to s3 and ensure that I’m not uploading to the wrong account because an environment variable is still set from another project.

Clients are generally understanding, but overriding static media for one client with another client’s files is not a conversation I want to have with any client.

direnv is excellent insurance against forgetting to unset an environment variable.

Seeding a `.env` file

I prefer to ship an example .env.example file in my projects with reasonable defaults and instructions for copying them over.

# .env.example
DEBUG=true
SECRET_KEY=you need to change this

If you are a casey/just justfile user, I like to ship a just bootstrap recipe that checks if a .env file already exists. If the .env file does not exist, it will copy the example in place.

My bootstrap recipe typically looks like this:

# justfile
bootstrap *ARGS:
    #!/usr/bin/env bash
    set -euo pipefail

    if [ ! -f ".env" ]; then
        echo ".env created"
        cp .env.example .env
    fi

How do we keep dotenv files in sync?

One pain point when working with dotenv files is keeping new environment variables updated when a new variable has been added.

Thankfully, modenv is an excellent utility that can do precisely this. I run modenv check and will compare the .env* files in the existing folder. It will tell us which files are missing an environment variable when it exists in one but not one of the other files.

I use modenv check -f to sync up any missing keys with a blank value. This works well to sync up any new environment variables added to our .env.example file with our local .env file.

Alternatives

I recently wrote about Using Chamber with Django and managing environment variables, which dives into using Chamber, another tool for managing environment variables.

If you are working with a team, the 1Password CLI’s op run command is an excellent way to share environment variables securely. The tool is straightforward and can be integrated securely with local workflows and CI with just a few steps.

My Python Roots

Mon, 11 Mar 2024 18:49:57 -0600

Last week, during office hours, I shared the two libraries that were my gateways to learning Python.

Cog

I stumbled on Ned Batchelder’s Cog while running an ISP in SWMO in the mid-00s. At the time, I was writing lots of PHP code and had a few layers of ORM code that I could generate with Cog’s help. This code was mainly boilerplate, and Cog was great at templating code. Thankfully, I didn’t need to know Python with Cog to make it work.

In recent years, I have still used Cog to update docs and to document Justfiles, Click, Typer, and console apps by grabbing the output and embedding it into docs.

Beautiful Soup

Beautiful Soup is the library that pushed me to learn Python. Beautiful Soup motivated me to learn Python and even more advanced feats like installing LXML and processing unparseable HTML or XML. I have always liked writing web scrapers and processing HTML documents, which is a weird hobby of mine.

My first Python app

My friends and I worked in our first post-college dot com job, and Dell was running an incredible deal on their 20" widescreen monitors over the Christmas holiday.

Dell ran a daily Dell Elf (Delf) contest where you gave them your email address, and they would give you a discount code for their various products.

The best code was 50% off of their 20" widescreen displays, which was an incredible deal then. The display retailed for $499, so getting one for $249.50 was great. These codes were random, and the odds were 1 in 25 to get one.

Using Python and having an email catchall, I wrote my first script to submit a series of email addresses until we found the daily 50% off code. At least four or five of my friends and I stocked up on these monitors that fall, and I have been a fan of Dell displays ever since.

Today

I still use Cog and Beautiful Soup 4 in several projects, including a few daily drivers. Last year, during their end-of-year sale, I picked three Dell 27-inch displays, and I still have fond memories of Dell’s displays.

Weeknotes 2024 W10

Sun, 10 Mar 2024 19:06:03 -0600

We were reminded that Winter is still here this week, starting with warmer weather and ending with colder.

Family

My oldest was out two days this week at parent-teacher conferences, and next week, he has off for Spring Break. He missed his last basketball practice and game because he’s had a bug for a few days with a runny nose and a low fever that keeps coming back.

My youngest child is firmly in the four to six-word sentence phase. We are up to our eyeballs in potty training, and my daughter’s new favorite phrase is, “No, not yet,” when you suggest we try again.

The kid’s grandfather drove down from Chicago to spend the weekend with us.

I picked up Oliver’s bug on Saturday and spent it more down than up, so my new compound miter saw sat in the box all weekend.

Work

We are a few weeks into a new client project, and that’s led to several refactors and working with newer tools, including Pydantic and Django Ninja for Model Schemas. I also expect to spend much time in Tailwind CSS land soon.

Community

We had our monthly DEFNA meeting, elected officers, and discussed upcoming announcements.

I had a PSF Code of Conduct WG meeting that conflicted with morning kid drop-offs, but I could join for the last 15 to 20 minutes to help them make a quorum so we could vote to pass a few things.

I only had time to join the first half of one Wiggle Work event this week over lunch.

This week, I hosted Office Hours.

Side projects

I spent most of my side project time on Django News, Django News Jobs, and Django Packages and upgraded several projects to the latest Django 5.0.x security release and Python 3.12 releases. I also moved several projects to use uv Python package manager, including Django Packages and Django News Jobs, which shaves minutes off of build times.

Django News Newsletter: We shipped issue #222.

I started on a Bootstrap to Tailwind CSS conversion tool after noticing that several popular tools last updated 3 to 6 years ago. I can brute force about 90% of it, but I have also meant to play around with cssutils for a while. With cssutils, I think it might be possible to read from the various Bootstrap CSS versions and match up to some Tailwind CSS classes by comparing them via their CSS output.

Writing

I wrote and published every day.

2024-03-09: Bootstrap to Tailwind CSS

2024-03-08: 🎮 Dave the Diver

2024-03-07: How to test with Django, parametrize, and lazy fixtures

2024-03-06: Importing data with Django Ninja’s ModelSchema

2024-03-05: Upgrading Python from 3.11 to 3.12 notes

2024-03-04: On pip isolation

2024-03-03: The Weekend Woodworker 🪚

“Me” time

🏀 I watched two Kansas Basketball games, a bunch of random men’s and primarily women’s games.

Next week

I have more rot repair, and I now have a compound miter saw to make cutting wood for framing much more manageable.

My oldest is off with spring break, so our morning routines start 10 minutes later and last about 40 minutes longer.

Next week also starts the Big 12 Men’s tournament, a sign that the second and third basketball seasons are beginning. I usually take a day off for the start of the Big 12 tournament, and I take a few days off for March Madness.

Bootstrap to Tailwind CSS

Sat, 09 Mar 2024 22:14:27 -0600

I spent a few hours tonight weighing my options to port a few websites from Bootstrap to Tailwind CSS.

I started with what seems to be the original awssat/tailwindo project is a PHP console app whose goal was to convert any Bootstrap to Tailwind CSS and was last updated three years ago. I couldn’t get it to work from the console or via Docker, so I punted and looked at other options.

This led me to the node-tailwindo project, which did install successfully for me. node-tailwindo project hadn’t been updated in six years, so much has changed in both projects.

Since node-tailwindo was installed successfully and seemed to run OK, I ran it on a few projects, including Django Packages, and the results were not terrible. They were not amazing, but things worked.

I looked at commercial options, and they fall into either Browser Extensions that let you view an existing website with a copy/convert to Tailwind CSS option or tools that rewrite your existing CSS. Neither felt like a good option to me.

I finally did what any Python developer would and installed BeautifulSoup4. Next, I wrote a script to read all the files in a template folder, and it extracted all the class attributes from the existing HTML. One hundred seventy-six unique classes later, I had my answer.

Writing my upgrade tool felt like a bigger project that I wanted to take on, but it helped me spot a few issues that node-tailwindo would struggle with.

This is where BeautifulSoup4 shines, and I could quickly swap out a few classes before I fed them into node-tailwindo, and it fixes several bugs where the project was confused by {% block %} and {{ variable }} tags/blocks.

This might be a project; I slowly update as I get bored since I can probably add and test 10 to 20 tests over lunch. For a brief minute, I debated if this would be my first Rust app. Spoiler: It is not.

How to test with Django, parametrize, and lazy fixtures

Thu, 07 Mar 2024 19:53:36 -0600

This article is a follow-up to my post on How to test with Django and pytest fixtures.

Here are some notes on how I prefer to test views for a Django application with authentication using pytest-lazy-fixture.

Fixtures

pytest-django has a django_user_model fixture/shortcut, which I recommend using to create valid Django user accounts for your project.

This example assumes that there are four levels of users. We have anonymous (not authenticated), “user,” staff, and superuser levels of permission to work with. Both staff and superusers follow the Django default pattern and have the is_staff and is_superuser boolean fields set appropriately.

# users/tests/fixtures.py
import pytest


@pytest.fixture
def password(db) -> str:
    return "password"


@pytest.fixture
def staff(db, django_user_model, faker, password):
    return django_user_model.objects.create_user(
        email="[email protected]",
        first_name=faker.first_name(),
        is_staff=True,
        is_superuser=False,
        last_name=faker.last_name(),
        password=password,
    )


@pytest.fixture()
def superuser(db, django_user_model, faker, password):
    return django_user_model.objects.create_user(
        email="[email protected]",
        first_name=faker.first_name(),
        is_staff=True,
        is_superuser=True,
        last_name=faker.last_name(),
        password=password,
    )


@pytest.fixture()
def user(db, django_user_model, faker, password):
    return django_user_model.objects.create_user(
        email="[email protected]",
        first_name=faker.first_name(),
        last_name=faker.last_name(),
        password=password,
    )

Testing our views with different User roles

We will assume that our website has some working Category pages that can only viewed by staff or superusers. The lazy_fixture library allows us to pass the name of a fixture using parametrize along with the expected status_code that our view should return.

If you have never seen parametrize, it is a nice pytest convention that will re-run the same test multiple times while passing a list of parameters into the test to be evaluated.

The tp function variable is a django-test-plus fixture.

user, staff, and superuser are fixtures we created above.

# categories/tests/test_views.py
import pytest

from pytest import param
from pytest_lazyfixture import lazy_fixture


def test_category_noauth(db, tp):
    """
    GET 'admin/categories/'
    """
    url = tp.reverse("admin:category-list")

    # Does this view work with auth?
    response = tp.get(url)
    tp.response_401(response)


@pytest.mark.parametrize(
    "testing_user,status_code",
    [
        param(lazy_fixture("user"), 403),
        param(lazy_fixture("staff"), 200),
        param(lazy_fixture("superuser"), 200),
    ],
)
def test_category_with_auth(db, tp, testing_user, password, status_code):
    """
    GET 'admin/categories/'
    """
    url = tp.reverse("admin:category-list")

    # Does this view work with auth?
    tp.client.login(username=testing_user.email, password=password)
    response = tp.get(url)
    assert response.status_code == status_code

Notes

Please note: These status codes are more typical for a REST API. So I would adjust any 40x status codes accordingly.

My goal in sharing these examples is to show that you can get some helpful testing in with a little bit of code, even if the goal isn’t to dive deep and cover everything.

Updates

To make my example more consistent, I updated @pytest.mark.django_db() to use a db fixture. Thank you, [email protected]/112060829098386312">Ben Lopatin, for the feedback.

Importing data with Django Ninja's ModelSchema

Wed, 06 Mar 2024 21:54:11 -0600

I have recently been playing with Django Ninja for small APIs and for leveraging Schema. Specifically, ModelSchema is worth checking out because it’s a hidden gem for working with Django models, even if you aren’t interested in building a Rest API.

Schemas are very useful to define your validation rules and responses, but sometimes you need to reflect your database models into schemas and keep changes in sync. https://django-ninja.dev/guides/response/django-pydantic/

One challenge we face is importing data from one legacy database into a new database with a different structure. While we can map old fields to new fields using a Python dictionary, we also need more control over what the data looks like coming back out.

Thankfully, ModelSchema is built on top of Pydantic’s BaseModel and supports Pydantic’s Field alias feature.

This allows us to create a ModelSchema based on a LegacyCategory model, and we can build out Field(alias="...") types to change the shape of how the data is returned.

We can then store the result as a Python dictionary and insert it into our new model. We can also log a JSON representation of the instance to make debugging easier. See Serializing Outside of Views for an overview of how the from_orm API works.

To test this, I built a proof of concept Django management command using django-click, which loops through all our legacy category models and prints them.

# management/commands/demo_model_schema.py
import djclick as click

from ninja import ModelSchema
from pydantic import Field

from legacy.models import LegacyCategory
from future.models import Category


class LegacyCategorySchema(ModelSchema):
    name: str = Field(alias="cat_name")
    description: str = Field(alias="cat_description")
    active: bool = Field(alias="cat_is_active")

    class Meta:
        fields = ["id"]
        model = Category


@click.command()
def main():
    categories = LegacyCategory.objects.all()
    for category in categories:
        data = LegacyCategorySchema.from_orm(category).dict()
        print(data)
        # save to a database or do something useful here

More resources

If you are curious about what Django Ninja is about, I recommend starting with their CRUD example: Final Code, and working backward. This will give you a good idea of what a finished CRUD Rest API looks like with Django Ninja.

Upgrading Python from 3.11 to 3.12 notes

Tue, 05 Mar 2024 09:15:40 -0600

Recently, I have been slowly moving several of my side projects and client projects from various Python versions to Python 3.12.

I never see people write about this, so it might be nice to write and share some notes.

Where to start

The first thing we do with a relatively simple upgrade is figure out what Python version we use. Thankfully, the project we picked mentioned in the README.md that it was using Python 3.11.

Once we know which version of Python we are using, we can open up iTerm and get a git checkout of the project.

Next, we will run git grep 11, where “11” is the shortened form of the Python version that we are running. There are so many variations of 3.11 and 311 that using the minor version tends to be about right.

$ git grep 11
... really long list...
.github/workflows/actions.yml:      - name: Set up Python 3.11
.github/workflows/actions.yml:          python-version: '3.11'
.github/workflows/actions.yml:      - name: Set up Python 3.11
.github/workflows/actions.yml:          python-version: '3.11'
.pre-commit-config.yaml:  python: python311
.pre-commit-config.yaml:        args: [--py311-plus]
README.md:This project will use Python 3.11, Docker, and Docker Compose.
README.md:Make a Python 3.11.x virtualenv.
docker/Dockerfile:FROM python:3.11-slim as builder-py
docker/Dockerfile:FROM python:3.11-slim AS release
pyproject.toml:requires-python = ">= 3.11"
pyproject.toml:# Assume Python >=3.11.
pyproject.toml:target-version = "py311"
requirements.txt:# This file is autogenerated by pip-compile with Python 3.11
... lots and lots of files...

This output will give us a long list of files. Usually, this is 100s or 1000s of files we will pipe or copy into our code editor. We will make a few passes to remove all of the CSS, SVG, and HTML files in the list, and that pairs down the results to half a dozen or a dozen files.

Create a new git branch

Next, we will create a new git branch called upgrade-to-python-3.12, and we will open each file one by one, and replace every “3.11” and “311” reference with “3.12” and “312” respectively.

Lint/format our code base

Once we have all of our files updated, we will commit everything. Then we will note special files like .pre-commit-config.yaml and pyproject.toml, impacting how my Python files are linted and formatted. Then, we will run pre-commit immediately after and commit any formatting changes.

Rebuild our Docker image

Since this project contains docker/Dockerfile that tells us the project uses Docker, we will need to rebuild our container image and note anything that breaks.

Re-pin/freeze our Python dependencies

Next, we will run pip-tools compile from within our newly rebuilt Docker container to build a new requirements.txt using Python 3.12.

Re-rebuild our Docker image (again)

“Insert Xzibit Yo Dawg meme." Next, we rebuild our Docker image using the newly pinned requirements.txt file, and this should be our final image.

Did our tests pass?

Assuming Docker builds cleanly, we will run my test suite using pytest. Once our tests pass, we’ll commit any uncommitted changes, git push our branch to GitHub, and open a Pull Request for review.

Did our tests pass in CI?

If our tests pass on GitHub Actions in CI, then we know our upgrade was successful, and we are reasonably confident that

When things don’t “just work.”

If you keep up with your upgrades, most of the time, everything works. Half a dozen projects did work for me, but I had one that did not work on Monday. There was a sub-dependency issue, so I closed my branch and opened a new issue to revisit this upgrade once the next version of Python 3.12.3 is released.

Even though this wasn’t a Python 3.12.2 bug, it takes the Python ecosystem time to catch up with newer versions. Since Python 3.11 is still supported for another 3 years and 7 months (as of this writing), it won’t hurt to wait a few weeks or months and revisit these changes.

If you are curious about how I decide when to adopt a new version, I wrote about that last month: Choosing the Right Python and Django Versions for Your Projects

On pip isolation

Mon, 04 Mar 2024 08:50:39 -0600

I saw this post by Trey Hunner about pip isolation, and I wanted to share a third method.

I’ve just updated my ~/.config/pip/pip.conf & my dotfiles repo to disallow pip installing outside virtual environments! 🎉

TIL 2 things about #Python’s pip:

pip has a config file. If I ever knew this, I’d forgotten.

pip has an option that stops it from working outside of a virtual environment!

https://mastodon.social/@treyhunner/112032637878747686

To Trey’s point, I never pip to install to easily install anything globally. If I want something installed globally, I can jump through a few hoops to avoid polluting my global pip cache.

My preferred way of disallowing pip installation outside virtual environments is to use the PIP_REQUIRE_VIRTUALENV environment variable.

I have export PIP_REQUIRE_VIRTUALENV=true set in my .bash_profile, which is part of my dotfiles. I prefer the ENV approach because I share my files over many computers, and it’s one less file to keep up with.

When I want to pip install something globally, I use pipx, which installs each Python application into its isolated environment.

For the few times that I do need to install a Python application globally, I use:

PIP_REQUIRE_VIRTUALENV=false python -m pip install \
    --upgrade \
    pip \
    pipx

I have this recipe baked into my global justfile so I can quickly apply upgrades.

New `django-startproject` update

Sat, 02 Mar 2024 22:27:42 -0600

I updated my django-startproject project today to support the latest versions of Django, Python, Compose, and other tools I’m a fan of. I use django-startproject to spin up projects that need some batteries quickly, but not every battery.

Features:

Django 5.0
Python 3.12
Docker Compose 3
Adds casey/just recipes/workflows (Just is a command runner, not a build tool)
Adds uv support

uv is the newest addition, which is a Python package installer and pip-tools replacement. It’s not a 100% drop-in replacement for pip and pip-tools, but it cuts my build times in half, and I have yet to hit any significant show-stoppers.

We did it, and what's next

Fri, 01 Mar 2024 20:07:26 -0600

My February goal was to publish an article every day for the entire month, and I’m happy to have completed it.

Shout out to the wonderful Wiggle Work community who started this Winter Writing Workshop. As someone who prides themself on never succumbing to peer pressure, the joke is on me! I finished and only used my phone a friend lifeline once.

Shoutout to my Wiggle Work friends, and thank you to everyone who joined this month and everyone who cheered us on from the Discord. Your emojis of support were the true friends we made along the way.

The other four who finished:

Writing tools

Every year, I put “write more” on my list of goals, and every year, I look back and realize some invisible barrier gets in my way. I get easily frustrated with every writing tool I have ever used, and my brain obsesses over the platform’s limitations.

I had a little bit of new writing tool productivity with both Obsidian and Micro.blog which were both invaluable to me.

Writing every day in February instilled in me that Obsidian is the best writing platform for me. I even paid for a Catalyst license to support Obsidian’s development.

Micro.blog also made it easy to post every day, and I suspect it will shape and heavily influence my blog platform going forward. I’m still annoyed that I can’t upload images or edit my social media posts with it, but once I understood the service’s limitations, it never got in my way.

Going forward

My existing blog uses GitHub Pages and Jekyll, but I’m ready to return to Django with a basic REST API for posting new blog posts and links.

My existing blog is built with Frontmatter, Markdown, and Yaml, which makes importing and editing everything doable with Python and a text editor.

I plan to use a mix of Obsidian with a custom Python script that can publish and sync to my website until I find a nice JS writing interface that I like.

Now that Obsidian supports Properties via Frontmatter, I can store my Post metadata between my text files and my blog to sync between the two.

I also want a slick Typefully-like experience for quick posts. Just one big markdown-friendly writing area that can handle dragging and dropping images to it. I want to pick where I am publishing when I choose to publish.

That’s where I plan to go next.

Python's UV tool is actually pretty good

Thu, 29 Feb 2024 13:31:04 -0600

I carved out some time recently to start playing with the new Python package installer and resolver, uv.

uv makes big promises and claims to be 10-100x faster than pip and pip-tools. From my experiments over the last few weeks, it lives up to this promise.

I’m using it locally for my virtual environments, in my Dockerfiles to rebuild my containers, and for CI using GitHub Actions. Across the board, anything I do with pip or pip-tools is remarkably faster.

My average GitHub Actions CI workflows dropped from ~2 minutes to 50 seconds. This cuts the minutes I use in half and, in theory, my monthly bill in half.

My goal in sharing my configs is more “show” than “tell' because I will copy and paste these for weeks and months to come.

local development

Every one of my projects has a justfile (it’s like Make but works the same everywhere) with “bootstrap” and “lock” recipes. My “bootstrap” recipe installs everything I need to work with the project locally. I use my “lock” recipe to lock my requirements.txt file so that I’m using the exact requirements locally and in production.

`justfile` before

My justfile might look like this:

@bootstrap
    python -m pip install --upgrade pip
    python -m pip install --upgrade --requirement requirements.in
    
@lock *ARGS:
    python -m piptools compile {{ ARGS }} ./requirements.in \
        --resolver=backtracking \
        --output-file ./requirements.txt

`justfile` after

For the most part, uv shares most of the same syntax as pip so you can start by changing your pip references to uv pip:

@bootstrap
    python -m pip install --upgrade pip uv
    python -m uv pip install --upgrade --requirement requirements.in
    
@lock *ARGS:
    python -m uv pip compile {{ ARGS }} ./requirements.in \
        --resolver=backtracking \
        --output-file ./requirements.txt

Dockerfiles

Everyone’s container setup is going to be different, but I use Docker and Orbstack, which use a Dockerfile.

`Dockerfile` before

FROM python:3.12-slim-bookworm

ENV PIP_DISABLE_PIP_VERSION_CHECK 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONPATH /srv
ENV PYTHONUNBUFFERED 1

RUN apt-get update

RUN pip install --upgrade pip

COPY requirements.txt /src/requirements.txt

RUN pip install --requirement /src/requirements.txt

WORKDIR /src/

`Dockerfile` after

FROM python:3.12-slim-bookworm

ENV PATH /venv/bin:$PATH. # this is new
ENV PIP_DISABLE_PIP_VERSION_CHECK 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONPATH /srv
ENV PYTHONUNBUFFERED 1

RUN apt-get update

RUN pip install --upgrade pip uv  # this is updated

RUN python -m uv venv /venv  # this is new

COPY requirements.txt /src/requirements.txt

RUN uv pip install --requirement /src/requirements.txt  # this is updated

WORKDIR /src/

GitHub Actions

GitHub Actions are a little harder to explain, but my workflows started off similar to this before I made the switch to uv:

`main.yml` before


  - name: Set up Python 3.12
    uses: actions/setup-python@v5
    with:
      python-version: '3.12'

  - name: Install dependencies
    run: |
            python -m pip install --requirement requirements.in

  - name: Collect Static Assets
    run: |
            python -m manage collectstatic --noinput

`main.yml` after

The biggest pain point that I ran into along the way was related to GitHub Issue #1386, which has a useable workaround.


  - name: Set up Python 3.12
    uses: actions/setup-python@v5
    with:
      python-version: '3.12'

  - name: Install dependencies
    run: |
      python -m pip install uv
      python -m uv venv .venv
      echo "VIRTUAL_ENV=.venv" >> $GITHUB_ENV
      echo "$PWD/.venv/bin" >> $GITHUB_PATH
      python -m uv pip install --requirement requirements.in      

  - name: Collect Static Assets
    run: |
      . .venv/bin/activate
      python -m manage collectstatic --noinput

Conclusion

I hope this was helpful to anyone who is considering making the switch to uv. I love to hear about how much time it saves you.

Updates

2024-03-08 - I modified the ENV PATH statement to prepend instead of replacing the value.

Using Django Q2

Tue, 27 Feb 2024 22:23:33 -0600

I’m long overdue to write about how Django Q2 has become part of my development toolkit. As the maintained successor to Django Q, Django Q2 extends Django to handle background tasks and scheduled jobs.

Django Q2 is flexible in managing tasks, whether sending out daily emails or performing hourly tasks like checking RSS feeds. The project works seamlessly with Django, making it one of the more straightforward background task solutions to integrate into your projects.

Using Django Q2 involves passing a method or a string reference to a method to an async_task() function, which will run in the background.

One feature of Django Q2 that particularly impresses me is its adaptability to various databases. Whether your project uses the default Django database or something more scalable like Redis, Django Q2 fits perfectly. This flexibility means that a database queue suffices without any hiccups for most of my projects, even those that are small to medium.

Unlike other task queues that require managing multiple processes or services, Django Q2 keeps it simple. The only necessity is to have the qcluster management command running, which is a breeze compared to other task queues because you only need to run one service to handle everything.

Django Q2’s flexibility, ease of use, and seamless integration with Django make it an excellent tool to reach for when you need background tasks.

Things I do every time I start a Django project

Mon, 26 Feb 2024 19:15:29 -0600

In the spirit of Brenton Cleeland’s Six things I do every time I start a Django project, my goal is to document and share my process since this often comes up.

Like Brenton, I have a lot of projects, and their scope may range from quick one-offs that might last a few hours to projects I have worked on for a decade or more.

Overview and upfront notes

I use pyenv to manage my Python versions, but I don’t go into any details here.

I use pyenv-virtualenvwrapper to manage what Python calls a “virtual environment,” which copies and symlinks files and folders where they need to be so that your Python projects are isolated from one another.

pyenv-virtualenvwrapper is also a pyenv-friendly fork of virtualenvwrapper.

Please note: To avoid even more confusion, please assume that at any time in this article, when I mention virtualenvwrapper or its various commands, I mean the pyenv-virtualenvwrapper version of these tools.

For more detail, see my article Python Development on macOS Notes: pyenv and pyenv-virtualenvwrapper.

Creating a new project

First, I need a folder to store all my project files. One nice feature of virtualenvwrapper is it can manage both my virtualenvs and where my projects live.

So my first step for a new project is to run mkvirtualenv {project_name}, where project_name is the name of the domain name or client that I’m working with. mkvirtualenv will then create a new virtual environment (virtualenv or venv is fine too) for me, make a new project folder for me to put my files into, and then transport me to this new project folder.

virtualenvwrapper also supports custom hooks, which is a fancy way of saying it runs a series of commands I tell it I want to run after it creates a virtualenv. These are a few of the custom steps that I run for my projects:

Create a Sublime Text project file so Alfred can detect it.
Create a Sublime Text pyrightconfig.json file, which enables LSP support for the project so that my code autocompletion works.
Create a Syncthing .stignore file.
Create a direnv .envrc file that loads my project’s environment variables.
Transports me into my projects folder.
Activate the new virtualenv, so Python is ready.
Install the latest version of pip, Python’s package manager.
Install the latest version of pip-tools, which I like to use for managing my project Python requirements.
Soon: Install the latest version of uv, which will soon replace pip-tools for me.

Even though most of my projects use Docker Compose, I still install my Python dependencies outside of Docker in the virtual environment so that my text editors and pyright can read them.

Create my Django app

Once I have a project folder and virtualenv set up, I must create a Django project. I use either my django-startproject or REVSYS’s Django starter template, which we call AlphaKit. Both projects aim to start jumpstart development and spend zero time configuring projects. Since my stater project is public, I will run this snippet to jumpstart it.

django-admin startproject \
    --extension=ini,py,yml \
    --template=https://github.com/jefftriplett/django-startproject/archive/main.zip \
    config

I landed on config for my projects because I like consistency when switching between projects, and over a decade and a half, I have never had a naming collision with config. The name config tends to sort to the top of my folder listings, making copying and pasting configs from project to project easier.

Setup Git

Now that I have my project started, I will run git init inside the project folder.
bunx gitignore python to set standard files to ignore
I will create a new repo on Git(Hub|Lab) and set it as a remote source for my project
I will run gittower ., select which files I’m ready to commit, and write my initial commit message.
I will git push from my terminal or GitTower and confirm that my new project is set up.

Switching projects

When I want to work on an existing project, I use virtualenv’s workon command to switch projects. workon {project_name} will transport me to the correct project folder and will activate my Python virtual environment so that I’m ready to work.

Do I need templates?

If my project needs templates, I’ll use a starter templates/base.html and the Tailwind CSS Play CDN.

I am a few years into using Tailwind CSS, and it’s my first choice for quickly bootstrapping a project. Despite the masto-hate, Tailwind CSS works for me and is vanilla CSS-friendly.

I like their Play CDN because I can start fiddling with Tailwind CSS with zero runtime and only one script tag to include in my HTML docs head.

Weeknotes 2024 W08

Sun, 25 Feb 2024 19:42:20 -0600

It warmed up this week, and we started our mornings out in the 30s and ended them in the 60s and even 70s all week. Our neighbors started putting their furniture back out on their porches, which is a local sign that winter is almost over.

Family

We found a small gas leak in our >100-year-old house and hired someone to fix it. If you ever smell gas, friends, please call and have it fixed as soon as possible.

I have been working with my son every day on his basketball skills. He missed his first game Saturday because of a stomach bug but bounced back enough Sunday to join me at the gym to practice. He went from zero dribbling skills to leading the team in dribbling drills, which he is pretty proud of.

My youngest child is firmly in the three to four-word sentence phase. My other half returned to work this week after being off for a procedure.

Work

I started a new client project this week, which involved spinning up a new green field project, taking a bunch of CSV files, and importing millions of records. The goal is to use Django, HTMX, and a lot of neat tech to build a vast parts catalog.

Community

We met to discuss our DjangoCon US website options as we try to figure out what we want in a new platform. We had a follow-up meeting about our Health and Safety policies, and I also had a meeting to help onboard someone to assist with our automation projects.

I joined the Conference Chats meetup this week and spent a few hours discussing conferences with other community organizers.

I joined two Wiggle Work events this week over lunch with the same group that had been writing this month.

I hosted Office Hours again this week.

Side Projects

I spent most of my side project time on Django News, Django News Jobs, and Django Packages.

Django News Jobs: I fixed several bugs, styled some message notifications, and refactored how various job status states work. We have different workflows for when a human submits a job versus jobs we collect from RSS feeds, APIs, or scraped. Most of these jobs are collected but have yet to be displayed. They should start seeing the light of day once I get a better handle on a moderation queue.

Django Packages: The website now runs on Django 5.0 thanks to Maksudul Haque. I merged in several rounds of dependency updates and pre-commit updates.

Django News Newsletter: We shipped issue #220. I worked on our secret backend project, which helps us collect news and makes updating the newsletter easier. I migrated our Tweet model to a Drafts model and dusted off some code that would post via the Typefully API.

All three projects received security updates, and I created a new blog app called the Changelog, which we’ll slowly roll out over the next few weeks or months. That should help us blog about project updates and release and share thoughts.

I started using the uv project to shave a minute or more off local updates and some Docker rebuilds. I’m trying it out for a week and then rolling it out.

“Me” time

I finished True Detective: Season 4: Night Country

I watched the first two seasons of Resident Alien

I watched two Kansas Basketball games and a bunch of random games.

I wrote seven blog posts, part of my goal to blog daily in February with the Wiggle Work Discord community I’m part of.

2024-02-24: Fetch the contents of a URL with Django service

2024-02-23: Three years later, email is less of a pain

2024-02-22: Scratching Itches with Python and ChatGPT

2024-02-21: Using Chamber with Django and managing environment variables

2024-02-20: Transferred - django-rq-email-backend found a new home

2024-02-19: 🥷 Django Ninja

2024-02-18: My 2024 Development Setup

Next week

I’ll finally take down those Christmas lights. I must repair rot damage on the front porch and our back steps. It’s almost time to start reseeding our yard and removing the extra sticks that have accumulated over the winter.

Fetch the contents of a URL with Django service

Sat, 24 Feb 2024 14:21:43 -0600

For the last few months, I have been using the cooked.wiki recipe-saving website, which initially impressed me because of how easy the website’s API is to use.

To use the service, all one has to do is prepend any website that contains a food recipe with https://cooked.wiki/, and you get the recipe without a coming-of-age discovery story.

This is a fun pattern, so I wrote my own in Django to illustrate how to build a Django view, which accepts a URL like http://localhost:8000/https://httpbin.org/get/ where https://httpbin.org/get/ will be fetched and the contents stored for processing.

# views.py 
import httpx

from django.http import HttpResponse
from urllib.parse import urlparse

def fetch_content_view(request, url: str) -> HttpResponse:
    # Ensure the URL starts with http:// or https://
    parsed_url = urlparse(url)
    if parsed_url.scheme in ("http", "https"):
        try:
            response = httpx.get(url)

            # Check for HTTP request errors
            httpx.raise_for_status()
            content = httpx.content

            # TODO: do something with content here...
            assert content
            return HttpResponse(f"{url=}")

        except httpx.RequestException as e:
            return HttpResponse(f"Error fetching the requested URL: {e}", status=500)

    else:
        return HttpResponse("Invalid URL format.", status=400)

# urls.py

from django.urls import path
from . import views

urlpatterns = [
    # other URL patterns here...
    ...
    
    path("<path:url>/", views.fetch_content_view, name="fetch_content"),
]

If you create your fetch the contents of a URL-like service, please consider putting it behind authentication to avoid someone discovering it and using it to DDOS someone’s website. I recommend throttling the view to prevent overloading a website by spamming requests to it.

Updated: I updated the example to switch from the python-requests references to the HTTPX library.

Scratching Itches with Python and ChatGPT

Thu, 22 Feb 2024 19:56:38 -0600

A few times a week over the last several months, I have paired with ChatGPT to work on Python scripts that solve problems that I would otherwise have spent less time on. Some might feel too niche or even too tedious that I would otherwise not take the time to work on. Most of the time, these are scratching an itch and solving a problem on my mind.

Because of the time constraint, I have been impressed with the results. I usually spend 10 to 15 minutes prompting ChatGPT, then I spend 10 to 15 minutes refactoring the script, adding Typer, and refining the code. Sometimes, this involved copying all or parts of my script and pasting it back into ChatGPT to have it refine or refactor some section of code.

YouTube Playlist to Markdown file

My YouTube playlist to markdown script is helpful for quickly getting a list of video URLs and titles back from a YouTube playlist. I have used this for DjangoCon US and a few other conferences to help collect links for social media and a few times for the Django News Newsletter.

ChatGPT even documented the process, including links for how to set permissions for the YouTube API.

Use Playwright to pull data out of the Django Admin

I have database access for most projects, but I recently needed to export a list of RSS feeds from the admin of a Django website. ChatGPT could quickly write a Playwright script to log in to the website, access a list page, and pull the feed field from the detail page. The script generated a JSON feed and could understand pagination and how to page through the links.

GitHub Issues and Pull Request templates

For the Awesome Django project, I asked ChatGPT to generate GitHub Issues and Pull Request templates based on my criteria. Once the templates were complete, I prompted ChatGPT to help me write a script that uses the GitHub API to read a pull request and validate the answers filled out while adding some other contextual data that makes it easier to verify the request.

Modeling HTML with PyDantic

I am still trying to figure out what to do with the project, but I asked ChatGPT to use Pydantic to create a class that could represent HTML tags. Once I was happy with the API, I asked ChatGPT to represent all HTML tags. After a few more prompts, I could read, write, and represent an HTML document using this script using Pydantic.

Outro

I’m still determining how useful these scripts are, but I have enjoyed these quick sessions to write a one-off script or to solve problems that come up a few times a year that never seemed worth the time spent trying to write it from scratch.

Using Chamber with Django and managing environment variables

Wed, 21 Feb 2024 22:01:19 -0600

One of my favorite hosting setups is getting a cheap slice from Digital Ocean or your favorite provider, installing Docker Compose and Tailscale, and then fire-walling everything off except for port 443.

Whenever I want to host a new project, I copy a docker-compose.yml file to the server, and then I start it with `docker compose up -d'.

I run Watchtower in Docker on the server, which looks for new Docker images from GitHub Packages, pulls them, and restarts any updated containers.

I can update my projects, git push changes, and GitHub Actions will build and store a new container image for me.

My main pain point was juggling environment variables until someone pointed out Chamber, which manages environment variables well.

Since creating this setup, I have shared several GitHub Gists with curious friends, and my goal of this post is to serve as more of an overview of options than it is to be a comprehensive guide to using Chamber.

Prerequisites

You’ll need an AWS account, some essential Docker and Compose knowledge, and to follow Chamber’s Installing instructions.

Setting up my environment

Ironically, my goal of eliminating individual environment variables led me to need four environment variables to bootstrap Chamber itself. The environment variables I’m using:

AWS_ACCESS_KEY_ID
AWS_REGION
AWS_SECRET_ACCESS_KEY
CHAMBER_KMS_KEY_ALIAS=aws/ssm

Dockerfile Setup

To make running Chamber running more straightforward, I used the segment/chamber Docker image, copied the /bin/chamber binary into my image, and configured it to run it as a ENTRYPOINT.

FROM segment/chamber:2.14 AS chamber

FROM python:3.11-slim-buster AS dev

FROM dev AS production
...
COPY --from=chamber /chamber /bin/chamber
ENTRYPOINT ["/bin/chamber", "exec", "django-news.com/production", "--"]

I prefer to namespace these variables based on the project and the environment I’m referencing, like django-news.com/production.

I am using a Docker entrypoint so that my secrets/environment variables work by default, whether running the image or overriding the default command, so I may shell into my container.

Docker Compose Setup

services:
  web:
    entrypoint: /bin/chamber exec django-news.com/production --
    command: gunicorn config.wsgi --bind 0.0.0.0:8000
    ...

Please note that the entrypoint line is optional if you set it in your ENTRYPOINT setting in your DOCKERFILE.

Using Chamber

Now that you have seen how we use Chamber in Docker and Docker Compose, this is how we get things into Chamber.

Listing our projects

$ chamber list-services

$ chamber list django-news.com/production

Write new settings

$ chamber write django-news.com/production DJANGO_DEBUG true

Delete an existing setting

$ chamber delete django-news.com/production DJANGO_DEBUG

Export our settings into a dotenv (.env) file

$ chamber export --format=dotenv django-news.com/production

Consuming an env variable from Django

The environs project is my go-to for parsing environment variables. Here is an example of how to toggle Django’s debug mode.

# settings.py
import environs

env = environs.Env()

DEBUG = env.bool("DJANGO_DEBUG", default=False)

Conclusion

I’m happy to manage my environment variables from the command line without syncing files. Using Chamber with KMS increased my monthly AWS bill by $0.01, which is money well spent for the flexibility of using Chamber.

Alternatives

I had a good experience using the 1Password CLI for a recent client project to share and load secrets into the environment. If you are working with a team, consider checking it out for your team in case it’s a good fit. Check out their Load secrets into the environment docs.

🥷 Django Ninja

Mon, 19 Feb 2024 16:58:37 -0600

I have heard good things about Django Ninja, and I was looking for an excuse to try it out.

One pain point is syncing production data to a local Django development website. Most of the time, I can dump a database and restore it locally. Sometimes, creating a REST endpoint and creating a local Django management command to consume that input is easier.

That’s what my example is doing here with Django Ninja.

`schemas.py`

If you have used Django REST framework’s ModelSerializer before, you will feel at home with Django Ninja’s ModelSchema.

A ModelSchema class can generate a schema from a Django model and only needs to know which Django model and which fields you want to be exposed to your endpoint.

from ninja import ModelSchema

from places.models import Place


class PlaceSchema(ModelSchema):
    class Meta:
        model = Place
        fields = [
            "name",
            "slug",
            "active",
            "place_id",
            "address",
            ...
        ]

`views.py`

Our Django Ninja view is going to going to query our database and return all of the active records in our Place model with respect to their creation date. Django Ninja will do the heavy lifting for us and apply our PlaceSchema which will return JSON data.

from ninja import Router
from ninja.pagination import LimitOffsetPagination
from ninja.pagination import paginate

from places.models import Place
from places.schemas import PlaceSchema


router = Router()


@router.get("/places/", response={200: list[PlaceSchema]})
@paginate(LimitOffsetPagination)
def list_places(request):
    qs = Place.objects.active().order_by("-created")
    return qs

`routers.py`

This routers.py file is optional, but I prefer to keep my routers separate from my urls.py, and this felt cleaner. I would keep this code for a small app in my urls.py instead.

from ninja import NinjaAPI

from places.views import router as places_router

# API setup
api = NinjaAPI(csrf=True, version="1.0.0")
api.add_router("", places_router)

`urls.py`

Our urls.py file is where we expose our router to the Internet so that our /apis/places/ URI will be accessible to anyone who knows it’s there.

...
from places.views import api
...


urlpatterns = [
    ...
    path("apis/", api.urls),
    ... 
]

`import_from_production.py`

This is a reasonably common Django management command pattern that I write. I prefer to use django-click because the click simplifies how to write Python command line interfaces.

We connect to our new endpoint, and if we get a good status code back, we will loop over our results and sync up our database with them.

Please note: I didn’t dive into how to do this security, but you will want to add some protection to your websites unless you are comfortable with it being public.

import djclick as click
import requests
from rich import print

from places.models import Place

# Constants
API_URL = "https://your-website.com/apis/places/"


[@click](https://micro.blog/click)()
@click.option("--limit", type=int, default=100)
def command(limit):
    # Start a requests session
    with requests.Session() as session:
        try:
            params = {"limit": limit}
            response = session.get(API_URL, params=params)
            response.raise_for_status()

            items = response.json()["items"]
            for item in items:
                if "name" in item:
                    name = item["name"]
                    address = item["address"]

                    defaults = dict(item)
					# delete foreign keys	
                    del defaults["cuisines"]
                    del defaults["neighborhood"]

                    try:
                        place, created = Place.objects.update_or_create(
                            name=name, address=address, defaults=defaults
                        )
                        print(f"{name} :: {created=}")

                    except Exception as e:
                        print(f"[red]{e=}[/red]")

        except requests.HTTPError as http_err:
            print(f"HTTP error occurred: {http_err}")

        except Exception as err:
            print(f"An error occurred: {err}")

Fin

Overall, I’m pretty happy with this pattern and Django Ninja. My only nitpick is that I wish that Django Ninja shipped with some class-based view batteries. I find them to be much easier to extend and work from than function-based views, but I’m told some third-party apps are starting to help fill in this space.

My 2024 Development Setup

Sun, 18 Feb 2024 21:39:06 -0600

I have wanted to document my development setup for a while to revisit it from time to time to see how my setup changes.

Hardware

I’m typing this from my couch on a MacBook Pro Intel/2019 model in Lawrence, KS.

I’m using a Mac Studio (M2/2023 model) at my office.

I’m using a Mac Mini Pro (M2/2023 model) in my home office.

I’m using an Intel Skull NUC (2017) and several Raspberry Pis (3s and 4s) wired to my router.

After a few decades of working on a laptop, I recently switched to a desktop machine because I wanted to sit down at either machine and work without carrying anything around. With two kids, I’m also juggling backpacks, water bottles, and whatever they need at school or preschool that day. Not having to worry about carrying anything is a huge mental weight off my shoulders.

At home, I run three Dell S2722QC 27-inch 4K USB-C monitors and a 16" portable monitor that I use as a dedicated iTerm display. At work, I use a pair of Monoprice 30" monitors and another 16" portable monitor for iTerm. At some point, I want to pick up the same Dell monitors for work.

Software

Since 2007, I have preferred developing on macOS. I’m running on macOS Ventura on everything, but I plan to migrate to macOS Sonoma sooner rather than later.

I use Tailscale, so I can connect to any of my machines from anywhere. I can even connect from my iPhone or an iPad. If you have more than one machine, even if that machine is a Raspberry Pi, Tailscale is worth the ~5-minute install.

I use Alfred, with some custom plugins, as my go-to “do everything on macOS” tool of choice.

Themes

Dracula is my favorite theme because it looks nice, and the community has built a theme for everything.

Home

I run Home Assistant at home and have a love/meh relationship with it. I have a liberal usage of Wyze cameras and gear, doubling as kid/baby monitors and external perimeter coverage around our gates and cars.

Backups

Backups should be handled in layers:

Quick local backups: Every Mac runs TimeMachine and backups to an encrypted, external drive. I built my NVMe backup drive solution that I will write about sometime.
External, off-sight backups: I store external copies of my data on BackBlaze.
I keep a few big, noisy 8TB external backup drives for occasional snapshots.

File Syncing

My documents are backed up to iCloud.
I use Syncthing to sync my Macs and NUC. I use this to sync projects across machines.

Development

SublimeText is my go-to text/code editor. Search is blazingly fast, runs amazingly well on both old and new hardware, and is a stable product.

Almost every Django project I work on runs in a container using Docker Compose or Orbstack. Orbstack is nice and fast, but I have many legacy projects in Docker, and the migration only worked so well.

I use the “scripts to rule them all” pattern because I work on tons of projects, and I never want to think twice about how to start, stop, update, open a console/shell, etc, and it does the job nicely. I will write about this later.

I prefer to script project workflows using Just because it’s a one-file install that doesn’t require a large development stack. It runs well both inside and outside of a container.

I wrote another post about Python Development on macOS Notes: pyenv and pyenv-virtualenvwrapper setup.

I use direnv to manage my environment variables.

Conclusion

Overall, I’m pretty happy with everything. I would like the same Dell monitor setup for the office and a TS4 dock, but everything works.

Office Hours

Sat, 17 Feb 2024 19:51:55 -0600

Please note: Just so you know, Grammarly told me my writing is fine. So please blame Grammarly and not me for the typos.

Last summer, I started hosting Office Hours on Friday afternoons. The basic idea around hosting office hours was an excuse to work on various projects and invite people to join to work on whatever projects they wanted to end their week.

Friday afternoons work well for me because I will have finished every task I wanted for the week, or anything left over will take me longer than an afternoon to complete, so I might as well start those on Monday instead.

My first session had 3 to 6 people, and then each week started slowly growing, and our last session had ~15 people. While half of the people who join each week I have known and met outside of a computer, more than half of the people who have joined office hours I have never met in person before. We have had people from all over the world, including Africa, the UK, the Netherlands, India, and Alaska, join in to chat while working on projects.

My humble goals for this time have always been to keep things light and serve as a welcoming space where people can join to discuss Python and Django, work on whatever project they want, and ask questions.

My goal wasn’t to build a community, even if it feels like one from seeing familiar faces week after week. So there is no Discord channel, mailing list, or iCal invite.

Instead, I have a private GitHub gist with the invite details, the starting date + time, and a Zoom link. I DM people who ask or people find the link from having accessed it before. I have not optimized my process for growth, but it works.

On weeks when my son has basketball practice, I have the freedom of moving the time around earlier or later, and to my surprise, a friendly group of people always join and trickle in over the time that I have to keep the Zoom meeting up.

On weeks when I’m out of town or need that Friday afternoon to hit a deadline, I update the gist to let people know that we’ll meet again in a few weeks. The success of office hours is not forcing them and letting people know that we have them most weeks but not every week when something is up.

I intentionally created a space where I don’t have the pressure to feel “on,” which is why it works for me, and I hope people will attend, too. There is no agenda, and I do my best to ensure everyone can talk and feel included while hanging out.

My stretch goal for 2024 might be to add a page on my website with the date of our next open hours that I can share with anyone curious. Maybe I’ll even use a nice font for the page, but for now, I think the gist works as well as anything.

The DACO Stack

Thu, 15 Feb 2024 20:36:33 -0600

For the last few months, I have used the Django + Adam Johnson + Carlton Gibson + Oliver Andrich stack, or what I’m calling the DACO stack. Maybe the DjACO stack rolls off the tongue more easily.

The DACO stack combines new Django tech with wrappers around libraries like Heroicons, TailwindCSS, and htmx, which integrates a nice modern front-end development experience with Django.

`carltongibson/neapolitan`

The Neapolitan project brings CRUD views to Django in what feels like a marriage between the Django Admin meets Django Rest Framework’s model viewsets but focused on the front-end. Neapolitan gives you CRUD views for your application in a few lines of Python code.

I have used it on several projects where I wanted to quickly build a front end around some data and give a limited number of people access to help maintain it.

`adamchainz/django-htmx`

Adam’s django-htmx adds htmx support to Django.

https://github.com/adamchainz/django-htmx

`carltongibson/django-template-partials`

Carlton’s django-template-partials project helps create reusable inline template blocks. When paired with django-htmx, we can render a form, search results, and individual table rows without rewriting the web page.

https://github.com/carltongibson/django-template-partials

`adamchainz/heroicons`

Adam brings the Heroicons library to Django, a series of SVG images that are nice for navigation menus and anywhere you might want to embed an icon image quickly. Heroicons is one of my goto libraries for quickly adding extra polish for apps that I might normally fall back to boring text links.

https://github.com/adamchainz/heroicons

`oliverandrich/django-tailwind-cli`

Oliver’s django-tailwind-cli project integrates Tailwind CSS into Django, includes a live reload server, and enables python manage.py tailwind to work.

https://github.com/oliverandrich/django-tailwind-cli

Conclusion

Try the DACO stack and let me know if it saves you time and if you find some new tools to add to it.

Year in review 2023

Mon, 12 Feb 2024 19:39:38 -0600

I write one of these posts every year, and I manage never to publish them. Since I’m trying to write one post a day for all of February, tonight I was revisiting some of my drafts, and this felt like a very Monday post to dust off and publish.

If 2023 had a theme, it was stepping down and stepping back.

Family

🚙 We took our first family trip with most of my immediate family for the first time in too many years in Surfside Beach, Texas
🚙 I bought a 2023 Subaru Ascent to replace my 2001 Nissan Pathfinder
6️⃣ My oldest turned six and started Kindergarten
2️⃣ My youngest turned two and started preschool

Django/Python

✅ PSF Board: I served two terms over five years and did not run for a third term. During my tenure, I served as Treasurer and then Vice Chair. I’m still on the Code of Conduct WG but left the Fiscal and PyCon US Committees.
✅ DEFNA: I stepped down as president after eight years on the board. I technically stepped down two or three years ago, but I filled in for a year to help out while we transitioned back to running an in-person conference. I’m still on the board and helping with DjangoCon US, but I’m doing so with fewer balls in the air.
🚀 DjangoCon US 2023 We held a successful conference in Durham, North Carolina.
🚀 Launched Django News Jobs

Community

🎙️ Attended/co-hosted dozens of Conference Chats meetups
🚀 I started weekly(ish) office hours on Fridays as an excuse to work on projects and hang out with community members.
🎟️ Attended PyCon US virtually
🎟️ Attended PyGotham virtually
🎟️ Attended DjangoCon US
✏️ I joined a weekly writing group
🎫 I joined a bunch of interesting Discord communities

Python Development on macOS Notes: pyenv and pyenv-virtualenvwrapper

Sat, 10 Feb 2024 20:04:43 -0600

Here are my notes for using pyenv and pyenv-virtualenvwrapper on macOS.

I use pyenv to install and manage multiple Python versions on the same machine. pyenv makes it possible to upgrade my default version of Python without breaking every project that was created with the older version until I’m ready to upgrade them.

I use pyenv-virtualenvwrapper to manage my projects. pyenv-virtualenvwrapper is a set of tools that make it easier to create, delete, copy, and manage Python virtual environments.

Install pyenv through Homebrew

brew update
brew install pyenv
brew install pyenv-virtualenvwrapper

Configuring pyenv to work with my shell

## set a variable for finding pyenv
export PYENV_ROOT="${HOME}/.pyenv"

# 2024-02 Python  pyenv settings
if command -v pyenv > /dev/null; then
    eval "$(pyenv init --path)";
    eval "$(pyenv init -)";
    pyenv virtualenvwrapper_lazy
fi

Install the “latest” Python versions using pyenv

# install latest Python 3.11
pyenv install 3.11:latest

# to see which version of 3.11 was installed
pyenv latest 3.11
> 3.11.7

# install latest Python 3.12
pyenv install 3.12:latest

# to see which version of 3.12 was installed
pyenv latest 3.12
> 3.12.1

# set each version globally to our latest python 3.x version
pyenv global 3.11.7 3.12.1

Upgrading Python versions

pyenv install --skip-existing 3.11:latest
pyenv install --skip-existing 3.12:latest

List every version of Python that is installable

pyenv install --list

List every installed version of Python

pyenv versions

The Power of Semi-Automation: Why Less Can Be More

Wed, 07 Feb 2024 23:21:58 -0600

According to the ninety-ninety rule, the final 10 percent of code needed to automate a task completely can take up 90 percent of the development time. This disproportionate effort is why I recommend that everyone consider semi-automating tasks and skipping the last 10%.

Quick wins

My goal with semi-automating tasks is to reduce the scope of a project to make automating the task easier and quicker. The more mundane and repetitive the task, the more likely I will prioritize automating it.

My favorite tasks to automate fall are quick wins or redundant processes. We use Toggl at work to log our billable hours and time spent on various projects. We have daily stand-ups in Slack, so everyone can keep up with everyone and be on the same page about vacations or might need another set of eyes on a project.

Using the Toggl API, I wrote a Python script that exports my notes so I can copy and paste them into Slack. I wrote a Python stand-up script that I run daily to see my notes from the previous day or week.

I could have over-engineered my script to attempt to post to Slack for me, but it turns out that all I needed was a better way to copy and paste from Toggl into Slack.

Why do I semi-automate tasks?

Semi-automation is easier. The benefit of semi-automating tasks is to reduce the scope to be a quick, productive win.

Semi-automation saves time. I often spend 30 minutes writing a Python script, saving me 15 minutes or more daily.

Semi-automation saves wasted effort. For my daily stand-up project, I can use an existing Python library to access the Toggl API and display all my entries for a day or range of dates. Most of the time, I will copy these notes as they are and paste them into Slack, or I will summarize them.

Semi-automation takes less maintenance. Since my Python script is wrapping an existing API, there is little to no maintenance. If the author decides they don’t want to maintain a Toggl library anymore, switching to another library is less effort and much less complex than if I were trying to maintain my library.

Semi-automation is often good enough. I thought about writing a Slackbot to post updates for me, but I quickly learned that it would be a bigger hassle and ultimately not worth the effort. My most significant pain point was having to pull up Toggl to get information out of it, and with my stand-up script, I could focus on writing better summaries while logging my hours instead.

Types of projects I like to semi-automate

The projects I like to semi-automate are repetitive tasks I have to do daily, weekly, and frequent enough to warrant The more repetitive a task is, the more likely I will write a Python script or casey/just justfile recipe to automate it.

“copy to clipboard” page
my daily stand-ups and monthly reports, which I pull from Toggl
use RSS feeds to help automate the collection of articles to review for a weekly newsletter
use justfile recipes for every project and even system-level macOS updates, so I don’t have to remember how to update or run my projects

Migrating unfinished projects

Once I realized that some of my unfinished projects were too ambitious, I started revisiting them to reduce the scope to something I could semi-automate to complete them.

Django Queryset to Markdown Table

Tue, 06 Feb 2024 21:24:25 -0600

Today, I asked on Mastodon if anyone knew of a good solution for generating a Markdown table from a Django queryset. Typically, people want to convert markdown into HTML, but this was a case where I needed to export data from Django to a static website.

Not missing a beat, [email protected]/111886900012564229">Katie McLaughlin recommended the python-tabulate library, which is what I was looking for.

Since I was short on time tonight, I asked ChatGPT to:

Write a queryset to markdown function using the Python tabulate library.
Add Python types support (because it’s 2024)
Rewrite the docstring

Once the code looked right, I loaded up my local copy of Django News Jobs, and I created a quick Django management command.

django-click is my go-to library for quickly writing CLI apps or what Django calls a management command. django-click reduces the boilerplate needed to write a management command down to one import, one decorator, and one function, and you get the best of the Python click library with everything Django has to offer.

# management/commands/table.py

import djclick as click

from django.db.models.query import QuerySet
from tabulate import tabulate

from jobs.models import JobListing


def queryset_to_markdown(queryset: QuerySet, fields: list[str] | None = None) -> str:
    """
    Convert a Django queryset to a markdown table using the tabulate library.

    Args:
        queryset (QuerySet): The Django queryset to convert into a markdown table.
        fields (list[str] | None, optional): A list of strings specifying the model fields to include in the table.
            If None, all fields from the model will be included. Defaults to None.

    Returns:
        str: A string representing the markdown table formatted according to GitHub-flavored Markdown.
    """

    # If fields are not specified, use all fields from the model
    if fields is None:
        fields = [field.name for field in queryset.model._meta.fields]

    # Prepare data for tabulation
    data = []
    for obj in queryset:
        row = [getattr(obj, field) for field in fields]
        data.append(row)

    # Generate markdown table
    markdown_table = tabulate(data, headers=fields, tablefmt="github")

    return markdown_table


@click.command()
def command():
    job_listings = JobListing.objects.all().active().order_by("-published")
    print(queryset_to_markdown(job_listings, fields=["title", "employer_name"]))

Running python manage.py table gave me the following table. Please note: I removed the employer_name column to fit the output in my article.

| title                                               |
|-----------------------------------------------------|
| Software Engineer (Terraso)                         |
| Django Fellow                                       |
| Senior Django Developer for Boutique Digital Agency |
| Software Engineer                                   |
| Business Systems Analyst                            |
| Python Django E-Commerce Full Stack                 |
| Principal Software Engineer                         |
| Michigan Online, Software Engineer                  |
| Full Stack Python Developer                         |
| Senior Backend Developer - Supportsite Team         |
| Python Backend Developer                            |
| Python / Django Developer (f/m/d)                   |
| Full Stack Python Developer                         |
| Software Engineer - Ubuntu Systems Management       |
| Senior Python/Django Engineer                       |

Overall, I’m happy with this approach. Django is an excellent framework for quickly prototyping and solving these problems.

I decided to write a management command instead of a view because markdown tables are very readable from the command line. I plan to convert it to a Django view to automate embedding tables like this into other projects.

How to test with Django and pytest fixtures

Mon, 05 Feb 2024 22:57:33 -0600

Here are some notes on how I like to write tests for a Django app that tests a model down to a view.

Django Model + pytest Fixture

This part is a mouthful, but I prefer to use the Model Bakery library to automatically create instances of Django models (aka a fixture) with good test data instead of manually creating them. This approach saves time and adds some variety to my test data.

We will also use the pytest fixture function to create and return an instance of a Django model using our Model Bakery fixture. This pytest fixture can be used in multiple test cases, ensuring consistency and reducing boilerplate code.

Every app in my project contains a fixtures.py file. My news app has a Category and a Post model, and my fixture looks like this.

# news/tests/fixtures.py

import pytest
from model_bakery import baker

@pytest.fixture()
def category(db):
    return baker.make("news.Category", name="Category Name")

@pytest.fixture()
def post(db, category):
    return baker.make("news.Post", title="Post Title", category=category)

Please note that a post fixture can accept category fixture as an argument.

Configuration

At the top of my project, we will have a conftest.py file. This is where we tell pytest to look for our fixtures so they may be automatically loaded.

# conftest.py
pytest_plugins = ["news.tests.fixtures"]

Testing our models

Next, we write a very basic test to ensure our fixture can create an instance of a model.

# news/tests/test_models.py

def test_get_category(category):
    assert category.name == "Category Name"


def test_get_post(post):
    assert post.title == "Post Title"

Testing our views

I prefer working with django-test-plus because it helps make writing tests cleaner and more straightforward. Then, for every view we have, we will write a test to ensure that our URL patterns match our expected routes and that we return a predicted status code.

# news/tests/test_views.py
import pytest

# test that our view resolves to the right uri path
def test_post_uri(tp):
    expected_url = "/"
    reversed_url = tp.reverse("post-list")
    assert expected_url == reversed_url

# test that our view works
def test_post_get(db, tp):
	response = tp.get("post-list")
    tp.assert_http_200_ok(response)

Running tests

To run our tests, we can run pytest.

pytest

The Django apps I actually use (rough cut)

Sun, 04 Feb 2024 23:20:30 -0600

This is an updated version of my response to the Top 5 3rd party packages from the Django Forum.

I plan on writing something more in-depth, but life, family, and getting the kids ready for the week got away from me today. Here is my list in case anyone finds them to be helpful.

django-allauth
django-click
django-htmx
django-q2
django-test-plus
django-tailwind-cli
django-template-partials
environs[django]
heroicons[django]
neapolitan
python-slugify
ruff via pre-commit but previously was black
whitenoise

Projects I no longer use

django-dbbackup I dropped this app for DSLR and then I more recently dropped I dropped them both because I didn’t want to install the extra Postgres files in my containers. I use pg_backup, pg_restore, and psql via the official

Choosing the Right Python and Django Versions for Your Projects

Thu, 01 Feb 2024 23:32:08 -0600

When deciding when to adopt a new major or minor Python or Django version, I prefer to wait until the 3rd or 4th patch release because reliability significantly improves.

Python and Django {major}.{minor}.{patch>2} releases are always more stable than {patch<3} releases.

Understanding Version Numbers

Version numbers typically follow a {major}.{minor}.{patch} format. For instance, in Django 5.0.2, “5” is the major version, “0” the minor, and “2” the patch. I’ve found that a version reaching its .2 or .3 patch release (like Django 5.0.2 or 5.0.3) is generally a reliable indicator of stability.

Why Wait for the 3rd or 4th Patch Release?

The early releases of any major software version can be unpredictable. Despite Python and Django’s commendable track record for quality, the broader ecosystem of packages and tools that integrate with them often requires time to catch up. This adjustment period is critical for ensuring compatibility and stability, reducing the risk of unexpected issues in production environments.

My Experience

Through trial and error, I’ve observed that holding off until at least the 2nd patch release mitigates around 90% of potential issues. However, waiting for the 3rd or 4th patch guarantees that most significant bugs have been fixed.

Recommendations for Implementation:

Stay Informed: Regularly review the release notes for Python and Django. This will help you understand the scope of changes and the introduction of any new features or critical fixes.
Use a Testing Matrix: If your project employs Continuous Integration (CI), include the latest versions of Python and Django in your testing matrix sooner rather than later. This proactive measure can help identify compatibility issues early, saving you time and reducing the risk of deploying unstable code to production.
Community Feedback: Pay attention to feedback from the Python and Django communities. I keep a good feel of the pulse by checking social media to see what developers are having issues with. I also like to check open and recently closed GitHub Issues and Pull Requests for packages I use to see what was broken by a new release and how to fix it.

Python on Jeff Triplett's Micro.blog

How I Use django-simple-nav for Dashboards, Command Palettes, and More

Defining nav items once and reusing them everywhere

Flexible URLs and named routes

Handling permissions without template logic

Rendering a dashboard nav with Django templates

Building a command palette from the same nav items

Final thoughts

📅 Office Hours for November

Office Hours Schedule

Join Us

How to Join

ℹ️ Meeting Guidelines

Default Apps 2025

Commentary

Changes from 2024

🤖 On GitHub Copilot CLI and prompts as code

GitHub Copilot CLI uses Claude Sonnet 4.5 by default

The “You are” trick

Use Xml tags in your prompts

Natural language prompts are becoming the universal programming interface

🗓️ Office Hours for Late August

Office Hours Schedule

Join Us

How to Join

ℹ️ Meeting Guidelines

Ditching Cloud APIs: How I Set Up Local Text-to-Speech with Kokoro TTS and Python

Installation

Basic Usage

Conclusion

🤷‍♂️ I miss Visual Basic

💍 Announcing The Great Django Webring

🤖 A better Django version/update command and Prompt-driven-development

What about funding?

Thinking outside the box

🤖 Ideas for "The Levels of Claude" Notes

Accessing Claude

Organization by experience level

Claude’s developer APIs

Conclusion: Claude as a Spectrum

🤖 "Agents" are Cheat Codes

Our Cheat Codes

More areas to explore

🤖 My big list of AI/LLM tools, notes, and how I'm using them

Mac Apps I use:

Baseline rules for running a model

Keeping up

Python

Python Click, django-click, and Typer notes

The Python Click library

The Python django-click library

The Python Typer library

Conclusion

📩 Email, Calendars, and the Chaos of Modern Workflows

Calendars and shared Documents

Calendar Agent

The better email solution

Conclusion

django-templated-email-md notes aka if you want to format emails with Markdown, use it

jobs/models.py

templates/emails/job_new.md

Bonus: Django-Q2

Forward Email

🤔 Rethinking Django's Command-Line Tool: Why We Should Rename `django-admin`

Why Rename django-admin?

Keep It Simple and Pythonic

Easy Transition for Everyone

Real-Life Benefits

Improving Compatibility with Tools Like UV

Balancing Old and New

Is this change zero work?

Conclusion

🐚 My most used commands in my terminal history

🎊 Year in review (short version) - It was a good year

Family

Writing

Conferences

Volunteer/Community Work

Office Hours

Side Quests

`jobs/models.py`

`templates/emails/job_new.md`

Why Rename `django-admin`?