I'm filing this one under "blog posts I wish existed when I started building Claude Code plugins."

Here's the thing: plugin development gets weird fast. Your settings.json gets borked and your hooks vanish. You have three versions of a plugin installed and Claude is loading the wrong one. You're developing two plugins at once and they're stepping on each other because they're both mutating the same shared state. You spend 20 minutes debugging a behavior that turns out to be the other plugin. And then, while trying to fix everything, you nuke your ~/.claude/ and lose every previous Claude session you've ever had.

Ask me how I know.

The fundamental problem: when you build tools for your own development environment, your dev environment is production. There's no staging environment for ~/.claude/. Every plugin you install, uninstall, or reinstall mutates the same config directory you rely on for actual work. And when something goes sideways (which it will), you're doing forensics on your own daily-driver config, which is now harder to debug because you are trying to use your daily-driver.

What Lives in ~/.claude/

Two files you actually edit:

• settings.json: permissions, env vars, hooks, Bedrock config. The big one.
• CLAUDE.md: your user memory. Instructions Claude loads on every session. You've probably spent real time curating this.

Then there's the plugin plumbing that every install/uninstall cycle churns through:

• plugins/installed_plugins.json: the source of truth for what's installed. Each entry has an installPath pointing to the cache, a projectPath for scoping, version info. If any of those paths go stale... well, good luck with that
• plugins/cache/{marketplace}/{plugin}/{version}/: where installed plugins actually live. Directory-source marketplaces copy files here at install time. Not symlink. Copy. We'll come back to why that matters.
• plugins/known_marketplaces.json: marketplace registry.

And then everything else: fifteen-odd internal directories for session history, conversation logs, debug output, shell snapshots, task state. You never touch these. But they all live in ~/.claude/ too, and they're all in the blast radius when something goes wrong.

And there's no sandbox mode for plugins (there is a general sandbox though). No "install this plugin but like, in a pretend way." You're always live.

So I went looking for a way to get out of production.

The Undocumented Env Var

Here's where it gets fun: this isn't in the docs.

If you dig through Claude Code's environment variable table (the big one in the settings docs), you'll find ANTHROPIC_API_KEY, CLAUDE_CODE_USE_BEDROCK, BASH_MAX_TIMEOUT_MS. But CLAUDE_CONFIG_DIR? Not listed. Not mentioned. Nowhere.

I'm pretty sure I found it in someone else's code on GitHub and validated that it worked. I wanted to go back and check the Claude session where I first used it, but, well. That session was in ~/.claude/. Which I nuked. While debugging a plugin. You see the problem.

CLAUDE_CONFIG_DIR=/tmp/test-claude claude

That's it. Claude boots up, but instead of reading from ~/.claude/, it reads from /tmp/test-claude/. Fresh slate. Your real config is untouched.

Except... this minimal version has problems:

• No auth. Your API keys and Bedrock credentials live in your real settings.json (and subscription login state is stored outside ~/.claude/ entirely, in the system keychain on macOS). A bare config dir can't talk to any model.
• No plugins. You get a naked Claude with no commands, no hooks. Which is the thing you're trying to test.
• No permissions. Every tool call prompts for approval. You'll be hitting "yes" a lot.

So you can't just point at an empty directory and go. You need an isolated config with just enough of your real settings to function.

What does "just enough" look like? It depends on how you connect to Claude:

Subscription users (Max, Pro, Teams, Enterprise): login state is stored outside ~/.claude/ (in the system keychain on macOS), so CLAUDE_CONFIG_DIR doesn't affect it. Your auth just works. You don't need to copy anything.API key users: set ANTHROPIC_API_KEY in your test config's settings.json env block, or just export it in the shell.Bedrock/AWS users: you need the env block (endpoint URLs, region) and awsAuthRefresh command copied from your real settings.json. This is the most involved case, so the examples below show this path.

Beyond auth, everyone needs:

• A minimal permissions allowlist so you're not approving echo and ls fifty times

That's the config side. For plugins, the CLI itself respects CLAUDE_CONFIG_DIR: claude plugin marketplace add and claude plugin install both write to whatever config dir you've pointed at, not ~/.claude/. The CLI handles marketplace state for you.

Still, that's a lot of steps to do by hand every time which is why I built a script.

Building a Proper Test Harness

Once I knew CLAUDE_CONFIG_DIR worked, I wanted something I could run without thinking: A script that sets up an isolated config, installs my plugins, and drops me into Claude. Plus a --clean flag that nukes everything and rebuilds in seconds.

Here's the shape of it. The full script lives in my plugins monorepo as bin/test-claude.

1. Create the test config directory

TEST_CONFIG="$REPO_ROOT/tmp/test-claude-config"
mkdir -p "$TEST_CONFIG"

I put it in tmp/ inside the repo so it's gitignored by default and lives next to the code I'm testing.

2. Copy auth from your real config (Bedrock/API key users)

GLOBAL_SETTINGS="$HOME/.claude/settings.json"
ENV_SETTINGS=$(jq '.env // {}' "$GLOBAL_SETTINGS")
AWS_AUTH=$(jq -r '.awsAuthRefresh // empty' "$GLOBAL_SETTINGS")

This pulls just the env block (where Bedrock endpoint config lives) and the awsAuthRefresh command from your real settings. Nothing else comes along: no permissions, no hooks, no plugin state.

3. Write a minimal settings.json

{
  "env": { "...your bedrock stuff..." },
  "awsAuthRefresh": "aws sso login --profile whatever",
  "permissions": {
    "allow": [
      "Bash(echo:*)",
      "Bash(git status:*)",
      "Bash(ls:*)",
      "Bash(pwd:*)",
      "Bash(uv run:*)"
    ]
  },
  "hooks": {}
}

The permissions list is deliberately minimal, just enough that you're not hitting "approve" on every trivial command. The empty hooks object means no hooks from your real config bleed in. You're testing your plugin's hooks, not whatever else you've got configured.

4. Register your repo as a marketplace

CLAUDE_CONFIG_DIR="$TEST_CONFIG" claude plugin marketplace add "$REPO_ROOT"

The CLI writes known_marketplaces.json, creates the plugins/marketplaces/ directory, all of it inside the test config, not ~/.claude/.

5. Install plugins into the isolated config

MARKETPLACE="$REPO_ROOT/.claude-plugin/marketplace.json"
jq -r '.plugins[] | select(.source | type == "string") | "\(.name)\t\(.source)"' \
    "$MARKETPLACE" | while IFS=$'\t' read -r name source; do
    full_path="$REPO_ROOT/${source#./}"
    if [ -d "$full_path" ]; then
        CLAUDE_CONFIG_DIR="$TEST_CONFIG" claude plugin install "$name@local-test"
    else
        echo "Error: plugin '$name' source not found at $full_path" >&2
        exit 1
    fi
done

Instead of scanning for plugins/*/ directories and assuming a layout, this reads .claude-plugin/marketplace.json, the file that actually defines what plugins exist. The jq filter grabs every plugin with a local path as its source (skipping any that point at GitHub repos or other remote sources), then verifies the directory exists before installing. CLAUDE_CONFIG_DIR means the install writes to the test config's cache and installed_plugins.json, not your real ones.

6. Run Claude

exec env CLAUDE_CONFIG_DIR="$TEST_CONFIG" claude "$@"

That's it. You get a Claude session with all your local plugins installed, talking to your real API through Bedrock, but with completely isolated config state. Fix a bug, --clean, reinstall, test again. Your real ~/.claude/ never knows it happened.

The Gotchas That Bit Me

Even with isolation, plugin development has some sharp edges. These all cost me at least an hour each.

You fix a bug, but don't see the fix.

This one's subtle. You find a problem in your hook script, fix it, test again. Same broken behavior. You add a print statement. Nothing. You start questioning reality.

The issue: directory-source marketplaces copy your plugin files into the cache at install time. They don't symlink. When you plugin install, Claude copies plugins/my-plugin/ into plugins/cache/marketplace-name/my-plugin/1.0.0/. Your fix is in the source. The cache still has the old code.

The fix is boring but non-negotiable: reinstall after every code change. Uninstall, install, restart Claude. Or if you're using an isolation script, --clean and start fresh. Adds a few seconds to each iteration. You will forget this.

Plugins silently stop loading.

Your plugin was working, now it's not. No error. Claude just doesn't have your commands anymore.

The cause: installed_plugins.json is pointing at a cache path that doesn't match what's actually on disk. Maybe you did a partial cleanup, or an install wrote a new version directory but the old entry is still around. The file says your plugin is at cache/marketplace/plugin/1.0.0/ but the cache only has 1.2.0/.

Claude doesn't crash when this happens. It just... doesn't load the plugin. Silent failure.

The fix: uninstall and reinstall. Or nuke the cache entry for that plugin and start clean. With an isolated test config, --clean solves this instantly. In your real config, you'll need to be more surgical: check the installPath entries in installed_plugins.json and make sure they actually exist.

Only some of your plugins load.

I had a monorepo with multiple plugins. Some loaded fine. Others were invisible, no error, just missing.

The issue: local-scoped plugins are registered with a projectPath matching where you installed them. If your cwd doesn't match that path exactly, Claude skips them. I was running from plugins/tool-routing/ instead of the repo root, so only plugins installed from that subdirectory were in scope.

This gets worse when plugins read from each other. tool-routing discovers route config from all enabled plugins (via claude plugin list --json), regardless of which marketplace they came from. If only half your plugins are in scope because of a projectPath mismatch, it only sees half the routes, and you get mysterious "route not found" behavior with no obvious cause.

The fix: always run from the repo root, or set CLAUDE_PROJECT_ROOT explicitly. In the test harness this is less of an issue because you control the working directory, but it'll bite you in manual testing.

The config dir you forgot about. (Bedrock users)

You set up CLAUDE_CONFIG_DIR, build a clean test config, everything works. Then you try something that needs authentication and it fails. You copy over more settings. Still fails. You spend 20 minutes wondering if the env var is even working.

Turns out login state doesn't live in ~/.claude/ at all. On macOS it's in the system keychain, and it works regardless of CLAUDE_CONFIG_DIR. You don't need to copy it. But you might spend a while looking for it in the wrong place first.

What does need to be in your test config is the Bedrock/AWS configuration: the env block and awsAuthRefresh in settings.json. That's what bin/test-claude copies over. The login session itself just works.

If you're on a subscription plan, this gotcha doesn't apply to you. Your auth is global and just works with an empty test config. Lucky you.

Where Isolation Is Just the Start

CLAUDE_CONFIG_DIR and a good test script get you pretty far. You can iterate on plugins without worrying about trashing your real config, and --clean gives you a predictable baseline on every run.

But isolation is one layer. Once your plugin installs cleanly, there's more:

• Unit tests for internal logic: route matching, config parsing, the stuff that doesn't need Claude running at all. pytest with temp directories and env var overrides.
• Integration tests for hooks: subprocess calls with realistic JSON payloads, checking exit codes and stderr.
• Behavioral tests for skills and prompts. This one's wild. You can dispatch subagents with and without your plugin's prompts and grade the difference against a rubric. Your plugin's effectiveness becomes measurable.

That's a whole other post. For now: isolate your config, script the setup, and stop testing in "production".

I have done my share of shell scripting and debugging, thanks in part to those early years doing system administration and Gentoo Linux development. I have managed to keep those skills sharp, so when my teammate @mclark ran into a shell problem, I was the first person he reached out too. This post is a dramatic retelling of our debugging session, with additional explanation.

He's been leading an effort to partition the GitHub monolith using packwerk. Matt has been working on getting some branches (1, 2) upstream, and it has been slow going for a variety of reasons. No judgement there, I know have been guilty of leaving pull requests unreviewed/unmerged and there are other reasons they are not merged, but we did want to start taking advantage of the improvements they offered.

We have this script that has been around a long time called vendor-gem. Basically, it takes a Git URL and tries to build a .gem file for it in a few steps:

• clones it locally
• tries to gem build it
• saves it to vendor/cache
• updates Gemfile with the resulting hash

It is a good fit for using mclark's fork of packwerk, and start to use it, and exactly what we do for other dependencies.

Now, I won't try to justify it's existence, or why it does what it does, except to say that it has been used for awhile, and it is occasionally useful. It could very likely be replaced any number of ways, but let us not forget the Parable of Chesteron's fence:

“In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”

But, I digress. The point is that this vendor-gem could be considered technical debt that needs to be dealt with, but in the mean time we have other work to handle.

So let us begin!

The Problem

@technicalpickles you know anything about the vendor-gem  script? I'm getting an error when trying to vendor my packwerk fork

$ script/vendor-gem https://github.com/mclark/packwerk -r 0f92c385c92d378fdbc2d912e4b20c9f524dbd42  
Cloning https://github.com/mclark/packwerk for gem build  
Building packwerk  
HEAD is now at 0f92c38 parallel packwerk  
commit 0f92c385c92d378fdbc2d912e4b20c9f524dbd42 (HEAD -> master, parallel-packwerk-check)  
Author: Matt Clark <[email protected]>  
Date:   Fri Apr 9 11:14:09 2021 -0400    parallel packwerk  
~/github/github/tmp/gems/packwerk ~/github/github/tmp/gems/packwerk  
Building packwerk.gemspec  
Traceback (most recent call last):  
	3: from -e:1:in \`<main>'  
	2: from -e:1:in \`eval'  
	1: from (eval):3:in \`<main>'  
(eval):3:in \`require\_relative': cannot infer basepath (LoadError)

Before digging in any further, here's what I already know:

• it is a shell script from prior experience
• it looks like a ruby stack trace at the bottom
• the error is cannot infer basepath (LoadError)

Narrow it down

Opening up the file, you can see a wall of bash code. We could spend some time reading the source to understand it, but I tend to favor finding ways to identify where it is failing.

For bash, a good way to do that is to use set -x. From the [Bash Guide for Beginners][debugging bash scripts]:

When things don't go according to plan, you need to determine what exactly causes the script to fail. Bash provides extensive debugging features. The most common is to start up the subshell with the -x option, which will run the entire script in debug mode. Traces of each command plus its arguments are printed to standard output after the commands have been expanded but before they are executed.

set -x			# activate debugging from here
w
set +x			# stop debugging from here

The top of the file already has a set -eu, so that's a good spot to add x for now. Alternatively, we can call it with bash -x as well to avoid modifying the file.

This gives us more info about what is happening when it fails:

++ ruby -e 'require '\''rubygems'\''; spec=eval(File.read('\''packwerk.gemspec'\'')); print spec.name.to_s'
Traceback (most recent call last):
        3: from -e:1:in `<main>'
        2: from -e:1:in `eval'
        1: from (eval):3:in `<main>'
-e:1:in `eval': cannot infer basepath (LoadError)
+ gemname=
+ rm -rf tmp/gems/packwerk

Making sense of it

The ruby -e defines a script on the command line without it having make a file.

It looks like we are trying to read the file, and then eval it in order to pull the gem out of it. .gemspec basically are creating an instance of [Gem::Specification], and looks like:

Gem::Specification.new do |s|
  s.name        = 'example'
  s.version     = '0.1.0'
  s.licenses    = \['MIT'\]
  s.summary     = "This is an example!"
  s.description = "Much longer explanation of the example!"
  s.authors     = \["Ruby Coder"\]
  s.email       = '[email protected]'
  s.files       = \["lib/example.rb"\]
  s.homepage    = 'https://rubygems.org/gems/example'
  s.metadata    = { "source\_code\_uri" \=> "https://github.com/example/example" }
end

It is literally Ruby, and so it can be read and then eval'd. Because we are evaling it, there's an implict return value of the Gem::Specification that was just built. That is assigned to spec, and then returning spec.name.

That is all to try to get the gem's actual name in the case it doesn't match the repository name. It usually does match, but not always, and repositories can contain multiple gems.

Narrow it down pt. 2

We know which command failed, but can we reproduce it? If you look at the bottom, the directory it was running is in was deleted:

+ rm -rf tmp/gems/packwerk

What could be doing that? Searching the file for rm -rf is a good way to find it:

# clone the repo under tmp, clean up on exit
echo "Cloning $url for gem build"
mkdir -p "tmp/gems/$gem"
trap "rm -rf tmp/gems/$gem" EXIT

Well, it's cleaning up at exit which explains why it's not there. It's using trap to catch signals like QUIT, INT (control-c), etc. EXIT is a special signal that bash sends when it's exiting.

This is good code hygiene, but it can make debugging harder in cases like right now. Commenting out is the easiest thing while we debug:

# clone the repo under tmp, clean up on exit
echo "Cloning $url for gem build"
mkdir -p "tmp/gems/$gem"
# trap "rm -rf tmp/gems/$gem" EXIT

If we run the script again, we get the failure again, but now tmp/gems/packwerk is there.

We need to know where to run the command. tmp/gems/packwerk is a good guess since it was what was cleaned up. We can also look for cd or pushd in the code:

# go in and build the gem using the HEAD version, clean up this tmp dir on exit
echo "Building $gem"
(
    cd "tmp/gems/$gem"

Now we can cd tmp/gems/packwerk and run the command to reproduce it:

$ ruby -e 'require '\''rubygems'\''; spec=eval(File.read('\''packwerk.gemspec'\'')); print spec.name.to_s'
Traceback (most recent call last):
        3: from -e:1:in `<main>'
        2: from -e:1:in `eval'
        1: from (eval):3:in `<main>'
-e:1:in `eval': cannot infer basepath (LoadError)

Making sense of it pt. 2

With just the command and the error, we have a lot less noise to get distracted by. The output is a Ruby error. Normally, it shows one line per file with line information, and the method it is in. We don't actually have much of that. What's more, there's this ruby -e. If you aren't familiar with it, from man ruby:

     -e command     Specifies script from command-line
                    while telling Ruby not to search the
                    rest of the arguments for a script
                    file name.

That means you can put the ruby you want to run as an argument, rather than taking a filename. This is convenient for one-line commands like we are doing right now.

Knowing that, we see -e is showing up in the stacktrace. I'm taking that to mean, that is what is being interpreted as the file:

$ ruby 💥-e💥 'require '\''rubygems'\''; spec=eval(File.read('\''packwerk.gemspec'\'')); print spec.name.to_s'
Traceback (most recent call last):
        3: from 💥-e💥:1:in `<main>'
        2: from 💥-e💥:1:in `eval'
        1: from (eval):3:in `<main>'
💥-e💥:1:in `eval': cannot infer basepath (LoadError)

Since we are reading a file, and evaling it, it'd be a good time to look at what is in the file:

# frozen_string_literal: true

require_relative "lib/packwerk/version"

Gem::Specification.new do |spec|
  spec.name          = "packwerk"
  spec.version       = Packwerk::VERSION
  spec.authors       = ["Shopify Inc."]
  spec.email         = ["[email protected]"]

  spec.summary       = "Packages for applications based on the zeitwerk autoloader"

  spec.description   = <<~DESCRIPTION
    Sets package level boundaries between a specified set of ruby
    constants to minimize cross-boundary referencing and dependency.
  DESCRIPTION

It's using require_relative to load a file:

module Packwerk
  VERSION = "1.1.3"
end

This is a pretty ubiquitous pattern. Things like bundler new (and jeweler too if you remember that far back even generate it for you.

Googling

It is a good point to Google the error. Let's just focus on the last part of line of it, strip out -e and other punctuation, and add ruby: ruby eval cannot infer basepath (LoadError).

This is the first result that comes up: require - ruby require_relative gives LoadError: cannot infer basepath inside IRB - Stack Overflow:

require_relative requires a file relative to the file the call to require_relative is in. Your call to require_relative isn't in any file, it's in the interactive interpreter, therefore it doesn't work.

This question is about using irb, but a lot of it applies to ruby -e too.

It's a good time to look for require_relative [Ruby's documentation on require_relative]:

Ruby tries to load the library named string relative to the requiring file’s path. If the file’s path cannot be determined a LoadError is raised. If a file is loaded true is returned and false otherwise.

Based on that post, it seems that changing require_relative 'lib/packwerk/version' to require './lib/packwerk/version' would be enough to work. The problem is it's part of gem's source, so we'd have to replace it as part of our script. This is a little bit of outside the box thinking, but it does feel a little hacky. Let's keep it in our back pocket, but keep working through this.

A red herring, and back on track

While it's useful information about require_relative, we forgot an important part of our error and original search: eval. eval can take a string, and execute it as Ruby. What happens if there are errors while evaling? Exactly like what we saw:

        1: from (eval):3:in `<main>'
💥-e💥:1:in `eval': cannot infer basepath (LoadError)

It doesn't matter if running with ruby -e means there isn't information the current file there. That isn't the context that the error is happening, the context is eval.

If you have done much meta-programming with Ruby and eval, you may recall it has some extra arguments. From the [ruby eval][docs]:

eval(string [, binding [, filename [,lineno]]]) → objclick to togg
Evaluates the Ruby expression(s) in string. If binding is given, which must be a Binding object, the evaluation is performed in its context. If the optional filename and lineno parameters are present, they will be used when reporting syntax errors.

The solution

I haven't seen it spelt out anywhere, but I have this hunch that adding a filename as an argument to eval will fix it. The file being eval'd is the gemspec, so that should be the filename.

These are positional arguments, so we need to specify the second argument (binding) too. gemspecs are pretty standalone and won't have any other context they need to bind to, so nil seems valid.

Let's try this:

$ ruby -e ''require '\''rubygems'\''; spec=eval(File.read('\''packwerk.gemspec'\''), nil, '\''packwerk.gemspec'\''); print spec.name.to_s'

packwerk⏎

Success! Take what we learn back to the script, and leave the set -x just in case...

    gemname=$(ruby -e "require 'rubygems'; spec=eval(File.read('$gemspec'), nil, '$gemspec'); print spec.name.to_s")

And it works! As we review the code, we see a few other spots on other conditional branches using the same logic, so we fix those too. In fact, there's another eval that is correctly giving a filename:

  spec = eval(File.read("$gemspec"), nil, "$gemspec")
  spec.version = "$GEM_VERSION"
  spec

Cleanup

While we are making the change, I thought it'd be useful to add a debugging option. That way, it's a little easier for the next person debugging. Here's what we ended up with:

: "${debug:=}"
while [ $# -gt 0 ]; do
        case "$1" in
                -d|--debug)
                        debug=1
                        shift
                        ;;
		            # other options here
        esac
done

# snip

if [ -n "$debug" ]; then
    set -x
fi

if [ -z "$debug" ]; then
    trap "rm -rf tmp/gems/$gem" EXIT
fi

There is some more bashisms and shellisms here that are probably worth further explanation, but this post is long enough already :blush:

In summary, if there's a -d or --debug option:

• enable tracing with set -x
• don't cleanup the gem at the end

Conclusion

With this problem solved, Matt was able to go on to start using his packwerk changes in production. In the end, this ended up being less than an hour of pairing. I spent more time writing this in fact!

The tools, errors, and people will change, but these kinds of problems come up constantly. I've found it valuable to use them as opportunities to learn a bit more in the areas they touch upon. This process, in turn, is it's own skill that you get really good at with enough time and bugs!

Here are some debugging techniques and philsophy you can take away from this:

• try to reproduce the problem
• narrow down where it is happen
• be curious; really read what the messages are telling you
• sometimes you end up chasing down a false lead, but it's okay as long as you learn something from it
• get creative in how you google for things
• don't forget to use known documentation like man pages and project documentation
• leave things a little better than you left it; make it easier for the next person

And some tidbits we learned about bash and ruby:

• set -x generates a lot of output, but is incredibly informative for debugging
• trap can be used for cleanup when a bash script exits, in addition to normal signals
• ruby -e lets you write one-liners without having to have a file
• require_relative won't work in irb, ruby -e or eval (without a filename)
• giving eval file and line information will almost always make debugging easier

You know when you are running your code, and you see a bunch of messages, and you have no idea what they are? And then one day, you just get used to them...

I'm reminded of this article:

It was my first day at work.

As I was talking to [the receptionist], I noticed a bright orange extension cord plugged in to the wall right behind the desk

I asked her, "What's with the orange extension cord?"

She replied, "What orange extension cord?"

I pointed it out and said, "That one, right behind you."

"Oh," she said as she turned, "That's been there a while."

I thought to myself, "Really? It's bright orange!"

This past week, the orange extension cord I decided to figure out was this one:

W, [2021-02-16T20:03:39.269073 #38]  WARN -- : Creating scope :open. Overwriting existing method Flake.open.
W, [2021-02-16T20:03:39.319617 #38]  WARN -- : Creating scope :open. Overwriting existing method FlakeCandidate.open.

I had primarily noticed in the Rails production console, but not as frequently in development. Rails does try to eagerly load all your classes in production, so that makes sense it wouldn't come up unless you are specifically using the class in development.

The classes in question:

class Flake < ApplicationRecor
  enum github_issue_status: { open: 0, closed: 1 }
end


class FlakeCandidate < ApplicationController
  enum github_issue_status: { open: 0, closed: 1 }
end

Step 1: Reproduce

This is easy enough, just access the class:

> Flake
Creating scope :open. Overwriting existing method Flake.open.
=> Flake (call 'Flake.connection' to establish a connection)

Step 2: Investigate

Unfortunately, this particular message doesn't give us a ton more insight to where it's happening. If you have big classes, have several enums, and one of which has open, it'd be easy to not even know where to begin.

It's easy to google things. It's hard to google the correct thing. If you search the exact message as is, you may end up with unrelated results based on what's in your models.

You can guess that the model and method name are getting substituted into an error message. One thing you can do is remove your local references from it (file paths, class names), and then quote the fragments of the original message. Add some keywords, and I end up with:

rails enum "Creating scope" "Overwriting existing method"

The first result is super promising: Rails warning: Overwriting existing method <model_name>.open · Issue #31234 · rails/rails

There is this comment from @matthewd:

Something’s defining an open method on the Issue class

The something is Ruby, and it's on Kernel.

I'm not sure how we can make the warning clearer; you seem to have interpreted it as a claim that something is overwriting your scope ("does not actually overwrite the method"), while that's the opposite of what it means to convey: your scope is itself overwriting the existing open method.

Thinking back to when I first saw this, I think I interpreted it the same way: something else was overriding the scope I defined. I agree the error is kind of confusing, and it's also hard to make it more accurate.

What makes this even trickier is that the method is coming from something you don't even think of as being part of the class! I have been doing Ruby for awhile, and I don't often stop to think how methods from Kernel are available everywhere.

I was curious if there was a way to show this, without having to know that Kernel is included, and that it has an open method. Turns out pry's show-source can show you!

> show-source User#open

From: /Users/technicalpickles/.rbenv/versions/2.6.6/lib/ruby/2.6.0/open-uri.rb @ line 29:
Owner: Kernel
Visibility: private
Number of lines: 11

def open(name, *rest, &block) # :doc:
  if name.respond_to?(:open)
    name.open(*rest, &block)
  elsif name.respond_to?(:to_str) &&
        %r{\A[A-Za-z][A-Za-z0-9+\-\.]*://} =~ name &&
        (uri = URI.parse(name)).respond_to?(:open)
    uri.open(*rest, &block)
  else
    open_uri_original_open(name, *rest, &block)
  end
end

Step 3: Fix

@bf4 was kind enough to leave a comment for future googlers:

For future googlers:

since the output comes from the logger, we can either wrap the enum definition in logger.silence (which may not be available) or something like that, or what is probably more correct, what I've done, it to undef the offending method before

  class << self; undef_method :open; end # the enum overrides the Kernel#open method which we don't care about
  enum status: { open: 0, closed: 1 }

I confirmed this works. I already have two models that would need it though, and it's not really useful for telling you why it's needed without having a code comment.

I ended up with a class method on my ApplicationRecord:

class ApplicationRecord < ActiveRecord::Base
	
  # rest of class goes here

  # This is needed to avoid a warning like "Creating scope :open. Overwriting existing method Flake.open"
  # when you have class with an enum with a value of 'open'
  #
  # See https://github.com/rails/rails/issues/31234 for an explanation
  def self.undefine_kernel_open!
    class << self
      undef_method :open
    end
  end

Now I can call it whenever I need it like:

class Flake < ApplicationRecor
  undefine_kernel_open!
  enum github_issue_status: { open: 0, closed: 1 }
end


class FlakeCandidate < ApplicationController
  undefine_kernel_open!
  enum github_issue_status: { open: 0, closed: 1 }
end

I like this a lot better because it gives future readers (including myself!) better context for understanding what it's doing.

Takeaways

Here's what I learned (or relearned):

• it can be satisfying to take 15 minutes to understand what some random message you see
• use pry's show-source to find where a method is defined
• Kernel is like everywhere!
• Issues on the rails/rails are a treasure trove of knowledge

Obligatory first post! It only took 7 years and a pandemic to get an update up on my blog.

I purchased pickles.dev back when it was released, but never got around to doing anything. So here it is, a new website! With a new design! And new content! Plus some old content too!

All my prior content is still online at technicalpickles.com for archival purposes, but I'll start migrating content soon.

I’m filing this one under “blog posts I wish existed when I was googling.” If you are dealing with data on the web, you are probably most familiar with JSON and XML. Less common nowadays is CSV, but if it’s all you have, and the alternative is screen scraping, then you are thankful.

Imagine we have some data in CSV:

Year,Make,Model,Description,Price
1997,Ford,E350,"ac, abs, moon",3000.00
1999,Chevy,"Venture ""Extended Edition""","",4900.00
1999,Chevy,"Venture ""Extended Edition, Very Large""",,5000.00
1996,Jeep,Grand Cherokee,"MUST SELL!
air, moon roof, loaded",4799.00

Now as a Ruby developer, particularly that has been infected by Rails, you’d be able to imagine this as an array of hashes, with keys/values using the column header, as the keys symbolized, and the values converted to numerics and blank ones converted to nil:

[
  {:year => 1997, :make => 'Ford', :model => 'E350', :description => 'ac, abs, moon', :price => 3000.00},
  {:year => 1999, :make => 'Chevy', :model => 'Venture "Extended Edition"', :description => nil, :price => 4900.00},
  {:year => 1999, :make => 'Chevy', :model => 'Venture "Extended Edition, Very Large"', :description => nil, :price => 5000.00},
  {:year => 1996, :make => 'Jeep', :model => 'Grand Cherokee', :description => "MUST SELL!\nair, moon roof, loaded", :price => 4799.00}
]

It might be tempting to just use regular expressions or read each line and split(','), but there are many nuances to the CSV format. Ruby’s stdlib includes CSV support to help us realize this dream with minimal hassle.

The documentation is unclear on the differences between methods of opening a CSV, nor does it help realize this Ruby data structure easily. We’ll be walking through the discovery of this from the documentation, but if you want the final solution jump down to the last code snippet.

A closer look at #new shows that it can take either a String or an IO-like object. The latter is interesting if you’ve gotten uploaded data, like with ActionPack’s UploadedData.

csv = CSV.new(body)

With this csv loaded, you can use to_a to get the array of data on it. This is a convenience method for #read and #readlines. This reads the remaining data in the string or IO you passed in, so any subsequent calls ends up returning an empty array.

csv.to_a
# => [["Year", "Make", "Model", "Description", "Price"], ["1997", "Ford", "E350", "ac, abs, moon", "3000.00"], ["1999", "Chevy", "Venture \"Extended Edition\"", "", "4900.00"], ["1999", "Chevy", "Venture \"Extended Edition, Very Large\"", nil, "5000.00"], ["1996", "Jeep", "Grand Cherokee", "MUST SELL!\nair, moon roof, loaded", "4799.00"]]
csv.to_a
# => []

This gives us an array of arrays, and the first element is an array with the headers. We are further than we started, but we still don’t have an array of hashes.

One of new #new’s options is :headers, which basically does just that:

csv.to_a
#=> [#<CSV::Row "Year":"1997" "Make":"Ford" "Model":"E350" "Description":"ac, abs, moon" "Price":"3000.00">, #<CSV::Row "Year":"1999" "Make":"Chevy" "Model":"Venture \"Extended Edition\"" "Description":"" "Price":"4900.00">, #<CSV::Row "Year":"1999" "Make":"Chevy" "Model":"Venture \"Extended Edition, Very Large\"" "Description":nil "Price":"5000.00">, #<CSV::Row "Year":"1996" "Make":"Jeep" "Model":"Grand Cherokee" "Description":"MUST SELL!\nair, moon roof, loaded" "Price":"4799.00">]

Actually, this looks like an array of CSV::Rows. It has a #to_hash. We can use map to apply that to each element in the array:

csv.to_a.map {|row| row.to_hash }
# => [{"Year"=>"1997", "Make"=>"Ford", "Model"=>"E350", "Description"=>"ac, abs, moon", "Price"=>"3000.00"}, {"Year"=>"1999", "Make"=>"Chevy", "Model"=>"Venture \"Extended Edition\"", "Description"=>"", "Price"=>"4900.00"}, {"Year"=>"1999", "Make"=>"Chevy", "Model"=>"Venture \"Extended Edition, Very Large\"", "Description"=>nil, "Price"=>"5000.00"}, {"Year"=>"1996", "Make"=>"Jeep", "Model"=>"Grand Cherokee", "Description"=>"MUST SELL!\nair, moon roof, loaded", "Price"=>"4799.00"}]

We are getting warmer! It’s an array of hashes, but the keys are literally the values from the header row as Strings. Another peak back at #new shows a :header_converters for converting the headers from their raw values. It’s not easy to find on the page, but here is a list of all header converters, with :symbol being the one we care about.

csv = CSV.new(body, :headers => true, :header_converters => :symbol)
csv.to_a.map {|row| row.to_hash }
# => [{:year=>"1997", :make=>"Ford", :model=>"E350", :description=>"ac, abs, moon", :price=>"3000.00"}, {:year=>"1999", :make=>"Chevy", :model=>"Venture \"Extended Edition\"", :description=>"", :price=>"4900.00"}, {:year=>"1999", :make=>"Chevy", :model=>"Venture \"Extended Edition, Very Large\"", :description=>nil, :price=>"5000.00"}, {:year=>"1996", :make=>"Jeep", :model=>"Grand Cherokee", :description=>"MUST SELL!\nair, moon roof, loaded", :price=>"4799.00"}]

Even warmer! The hash keys are now symoblized, but we still have some wonky data in there. There’s blank strings (“”), and numerics as strings (“1999”, “4900.00”). Yet another option to #new is :converts, to convert each row’s values. Here is a list of available converts, but we want :all which converts numerics (float and intergers) and date & datetimes:

csv = CSV.new(body, :headers => true, :header_converters => :symbol, :converters => :all)
csv.to_a.map {|row| row.to_hash }
#=> [{:year=>1997, :make=>"Ford", :model=>"E350", :description=>"ac, abs, moon", :price=>3000.0}, {:year=>1999, :make=>"Chevy", :model=>"Venture \"Extended Edition\"", :description=>"", :price=>4900.0}, {:year=>1999, :make=>"Chevy", :model=>"Venture \"Extended Edition, Very Large\"", :description=>nil, :price=>5000.0}, {:year=>1996, :make=>"Jeep", :model=>"Grand Cherokee", :description=>"MUST SELL!\nair, moon roof, loaded", :price=>4799.0}]

SO WARM! The only thing I can complain about is that there’s blank strings (“”). While there’s nothing else built in to help us, there is support for adding your own custom converters. You can add your own converts to CSV::Converters and CSV::HeaderConverters as you want. These are just hashes, with the key being the name, and the value being a lambda that takes a string and should return the converted value. You can then pass an array of converters to use to #new.

CSV::Converters[:blank_to_nil] = lambda do |field|
  field && field.empty? ? nil : field
end
csv = CSV.new(body, :headers => true, :header_converters => :symbol, :converters => [:all, :blank_to_nil])
csv.to_a.map {|row| row.to_hash }
# => [{:year=>1997, :make=>"Ford", :model=>"E350", :description=>"ac, abs, moon", :price=>3000.0}, {:year=>1999, :make=>"Chevy", :model=>"Venture \"Extended Edition\"", :description=>nil, :price=>4900.0}, {:year=>1999, :make=>"Chevy", :model=>"Venture \"Extended Edition, Very Large\"", :description=>nil, :price=>5000.0}, {:year=>1996, :make=>"Jeep", :model=>"Grand Cherokee", :description=>"MUST SELL!\nair, moon roof, loaded", :price=>4799.0}]

HOT HOT HOT. That is basically exactly what I set out to do, so we are all done here.