Purpose: Guide for AI agents working ON the ecto_libsql codebase itself

⚠️ IMPORTANT: This guide is for developing and maintaining the ecto_libsql library. ⚠️ IMPORTANT: For USING ecto_libsql in applications, see USAGE.md instead.

• British/Australian English for all code, comments, and documentation (except SQL keywords and compatibility requirements)
• ⚠️ CRITICAL: ALWAYS check formatting BEFORE committing:
  1. Run formatters: mix format && cd native/ecto_libsql && cargo fmt
  2. Verify checks pass: mix format --check-formatted && cargo fmt --check
  3. Only then commit: git commit -m "..."
  • Formatting issues caught at check time, not after commit
• NEVER use .unwrap() in production Rust code - use safe_lock helpers (see Error Handling)
• Tests MAY use .unwrap() for simplicity

When ending a work session, you MUST complete ALL steps below. Work is NOT complete until git commit succeeds.

MANDATORY WORKFLOW:

1. File issues for remaining work - Create Beads issues for anything that needs follow-up (see Issue Tracking with Beads)
2. Run quality gates (if code changed) - Tests, linters, builds
3. Update issue status - Close finished work, update in-progress items
4. COMMIT - This is MANDATORY:

   git commit -m "Your commit message"
   bd sync

5. Clean up - Clear stashes, prune remote branches
6. Verify - All changes committed
7. Hand off - Provide context for next session

CRITICAL RULES:

• Work is NOT complete until git commit succeeds
• If commit fails, resolve and retry until it succeeds

• Project Overview
• Architecture
• Code Structure
• Development Workflow
• Issue Tracking with Beads
• Error Handling Patterns
• Testing
• Common Tasks
• Troubleshooting
• Quick Reference
• Resources

EctoLibSql is a production-ready Ecto adapter for LibSQL, implemented as a Rust NIF for high performance.

• Full Ecto support (schemas, migrations, queries, associations)
• Three connection modes: Local SQLite, Remote Turso, Embedded replica
• Vector search, encryption, prepared statements, batch operations
• High-performance async/await with connection pooling

• Local: database: "local.db"
• Remote: uri: "libsql://..." + auth_token: "..."
• Replica: database + uri + auth_token + sync: true

Phoenix / Application
  ↓
Ecto.Adapters.LibSql (storage, type loaders/dumpers)
  ↓
Ecto.Adapters.LibSql.Connection (SQL generation, DDL)
  ↓
EctoLibSql (DBConnection protocol)
  ↓
EctoLibSql.Native (Rust NIF wrappers)
  ↓
Rust NIF (libsql-rs, connection registry, async runtime)

Elixir:

• lib/ecto_libsql.ex - DBConnection protocol
• lib/ecto_libsql/native.ex - NIF wrappers
• lib/ecto_libsql/state.ex - Connection state
• lib/ecto/adapters/libsql.ex - Main adapter
• lib/ecto/adapters/libsql/connection.ex - SQL generation

Rust (native/ecto_libsql/src/):

• lib.rs - Root module, NIF registration
• models.rs - Core data structures (LibSQLConn, CursorData, TransactionEntry)
• constants.rs - Global registries (connections, transactions, statements, cursors)
• utils.rs - Safe locking, error handling, type conversions
• connection.rs - Connection lifecycle
• query.rs - Query execution
• transaction.rs - Transaction management with ownership tracking
• savepoint.rs - Nested transactions
• statement.rs - Prepared statement caching
• batch.rs - Batch operations
• cursor.rs - Cursor streaming
• replication.rs - Replica sync
• metadata.rs - Metadata access
• decode.rs - Value type conversions
• tests/ - Test modules

Tests:

• test/*.exs - Elixir tests (adapter, integration, migrations, error handling, Turso)
• native/ecto_libsql/src/tests/ - Rust tests (constants, utils, integration)

Documentation:

• USAGE.md - API reference for library users
• CLAUDE.md - This file (development guide)
• README.md - User documentation
• CHANGELOG.md - Version history
• ECTO_MIGRATION_GUIDE.md - Migrating from PostgreSQL/MySQL
• RUST_ERROR_HANDLING.md - Error pattern reference
• TESTING.md - Testing strategy

14 focused modules, each with single responsibility:

Key Data Structures:

// Connection resource
pub struct LibSQLConn {
    pub db: libsql::Database,
    pub client: Arc<Mutex<libsql::Connection>>,
}

// Transaction with ownership tracking
pub struct TransactionEntry {
    pub conn_id: String,        // Which connection owns this transaction
    pub transaction: Transaction,
}

// Cursor pagination state
pub struct CursorData {
    pub columns: Vec<String>,
    pub rows: Vec<Vec<Value>>,
    pub position: usize,
}

git clone <repo-url>
cd ecto_libsql
mix deps.get
mix compile                        # Includes Rust NIF compilation
mix test
cd native/ecto_libsql && cargo test

ALWAYS branch from main for new work:

git checkout main
git pull origin main
git checkout -b feature-descriptive-name   # or bugfix-descriptive-name

Branch naming:

• Features: feature-<descriptive-name>
• Bug fixes: bugfix-<descriptive-name>

The repository often has untracked/uncommitted files (docs, notes, etc.) that must NOT be lost when switching branches. Git preserves untracked files across branch switches automatically, but:

• NEVER run git clean without explicit user approval
• NEVER run git checkout . or git restore . on the whole repo
• NEVER run git reset --hard without explicit user approval
• When switching branches, untracked files stay in place - this is expected

1. Create feature/bugfix branch from main (see above)
2. Make changes to Elixir or Rust code
3. ALWAYS format: mix format --check-formatted && cargo fmt
4. Run tests: mix test && cargo test
5. Fix any issues from formatting or tests
6. Commit ONLY files you touched - other untracked files stay as-is

All changes go through PRs to main (for review bot checks):

git push -u origin feature-descriptive-name
gh pr create --base main --title "feat: description" --body "..."

After PR is merged, clean up:

git checkout main
git pull origin main
git branch -d feature-descriptive-name     # Delete local branch

This project uses Beads (bd command) for issue tracking across sessions. Beads persists work context in .beads/issues.jsonl.

When to use Beads vs TodoWrite:

• Beads: Multi-session work, dependencies between tasks, discovered work that needs tracking
• TodoWrite: Simple single-session task execution

When in doubt, prefer Beads — persistence you don't need beats lost context.

Essential commands:

# Finding work
bd ready                           # Show issues ready to work (no blockers)
bd list --status=open              # All open issues
bd show <id>                       # Detailed issue view with dependencies

# Creating & updating (priority: 0-4, NOT "high"/"low")
bd create --title="..." --type=task|bug|feature --priority=2
bd update <id> --status=in_progress
bd close <id>                      # Or: bd close <id1> <id2> ...

# Dependencies
bd dep add <issue> <depends-on>    # Add dependency
bd blocked                         # Show all blocked issues

# Sync & health
bd sync --from-main                # Pull beads updates from main
bd stats                           # Project statistics
bd doctor                          # Check for issues
bd prime                           # Session recovery after compaction

Typical workflow:

# Starting work
bd ready && bd show <id> && bd update <id> --status=in_progress

# Completing work
bd close <id1> <id2> ...           # Close completed issues
bd sync --from-main                # Pull latest beads
git add . && git commit -m "..."   # Commit changes

• Check bd ready at session start to find available work
• Update status as you work (in_progress → closed)
• Create new issues with bd create when you discover tasks
• Use descriptive titles and set appropriate priority/type
• Always bd sync before ending session

IMPORTANT: Modern Rustler auto-detects all #[rustler::nif] functions. No manual registration needed.

Steps:

1. Choose the right module:

   • Connection lifecycle → connection.rs
   • Query execution → query.rs
   • Transactions → transaction.rs
   • Batch operations → batch.rs
   • Statements → statement.rs
   • Cursors → cursor.rs
   • Replication → replication.rs
   • Metadata → metadata.rs
   • Savepoints → savepoint.rs
   • Utilities → utils.rs

2. Define Rust NIF (e.g., in native/ecto_libsql/src/query.rs):

/// Execute a custom operation.
///
/// # Returns
/// - `{:ok, result}` - Success
/// - `{:error, reason}` - Failure
#[rustler::nif(schedule = "DirtyIo")]
pub fn my_new_function(conn_id: &str, param: &str) -> NifResult<String> {
    let conn_map = safe_lock(&CONNECTION_REGISTRY, "my_new_function")?;
    let _conn = conn_map
        .get(conn_id)
        .ok_or_else(|| rustler::Error::Term(Box::new("Connection not found")))?;
    
    // Implementation
    Ok("result".to_string())
}

3. Add Elixir wrapper in lib/ecto_libsql/native.ex:

# NIF stub
def my_new_function(_conn, _param), do: :erlang.nif_error(:nif_not_loaded)

# Safe wrapper using State
def my_new_function_safe(%EctoLibSql.State{conn_id: conn_id} = _state, param) do
  case my_new_function(conn_id, param) do
    {:ok, result} -> {:ok, result}
    {:error, reason} -> {:error, reason}
  end
end

4. Add tests in both Rust (native/ecto_libsql/src/tests/) and Elixir (test/)

5. Update documentation in USAGE.md and CHANGELOG.md

1. Update lib/ecto/adapters/libsql/connection.ex for SQL generation
2. Update lib/ecto/adapters/libsql.ex for storage/type handling
3. Add tests in test/ecto_*_test.exs
4. Update README and USAGE.md

NEVER use .unwrap() in production code - all 146 unwrap calls eliminated in v0.5.0 to prevent VM crashes.

See RUST_ERROR_HANDLING.md for comprehensive patterns.

✅ CORRECT
let conn_map = safe_lock(&CONNECTION_REGISTRY, "function_name context")?;

❌ WRONG - will panic!
let conn_map = CONNECTION_REGISTRY.lock().unwrap();

✅ CORRECT
let client_guard = safe_lock_arc(&client, "function_name client")?;

❌ WRONG
let result = client.lock().unwrap();

✅ CORRECT
let conn = conn_map
    .get(conn_id)
    .ok_or_else(|| rustler::Error::Term(Box::new("Connection not found")))?;

❌ WRONG
let conn = conn_map.get(conn_id).unwrap();

TOKIO_RUNTIME.block_on(async {
    let guard = safe_lock_arc(&client, "context")
        .map_err(|e| format!("{:?}", e))?;
    guard.query(sql, params).await.map_err(|e| format!("{:?}", e))
})

let conn_map = safe_lock(&CONNECTION_REGISTRY, "function")?;
let client = conn_map.get(conn_id).cloned()
    .ok_or_else(|| rustler::Error::Term(Box::new("Connection not found")))?;
drop(conn_map); // Release lock!

TOKIO_RUNTIME.block_on(async { /* async work */ })

# Case match
case EctoLibSql.Native.query(state, sql, params) do
  {:ok, _, result, new_state} -> # Handle success
  {:error, reason} -> # Handle error
end

# With clause
with {:ok, state} <- EctoLibSql.connect(opts),
     {:ok, _, result, state} <- EctoLibSql.handle_execute(sql, [], [], state) do
  :ok
else
  {:error, reason} -> handle_error(reason)
end

mix test                                    # All Elixir tests
cd native/ecto_libsql && cargo test         # All Rust tests
mix test test/ecto_integration_test.exs     # Single file
mix test test/file.exs:42 --trace           # Single test with trace
mix test --exclude turso_remote             # Skip Turso tests

• Happy path (successful operations)
• Error cases (invalid IDs, missing resources, constraint violations)
• Edge cases (NULL values, empty strings, large datasets)
• Transaction rollback scenarios
• Type conversions (Elixir ↔ SQLite)
• Concurrent operations

For state threading in tests, use consistent variable names and patterns:

Variable Naming by Scope:

state      # Connection scope
trx_state  # Transaction scope  
cursor     # Cursor scope
stmt_id    # Prepared statement ID scope

Error Handling Pattern:

When an error operation returns updated state, you must decide if that state is needed next:

# ✅ If state IS needed for subsequent operations → Rebind
result = EctoLibSql.handle_execute(sql, params, [], trx_state)
assert {:error, _reason, trx_state} = result  # Rebind - reuse updated state
:ok = EctoLibSql.Native.rollback_to_savepoint_by_name(trx_state, "sp1")

# ✅ If state is NOT needed → Discard with underscore
result = EctoLibSql.handle_execute(sql, params, [], trx_state)
assert {:error, _reason, _state} = result  # Discard - not reused
:ok = EctoLibSql.Native.rollback_to_savepoint_by_name(trx_state, "sp1")

# ✅ For terminal operations → Use underscore variable name
assert {:error, %EctoLibSql.Error{}, _conn} = EctoLibSql.handle_execute(...)

Add clarifying comments when rebinding state:

# Rebind trx_state - error tuple contains updated transaction state needed for recovery
assert {:error, _reason, trx_state} = result

See TEST_STATE_VARIABLE_CONVENTIONS.md for detailed guidance.

⚠️ Cost Warning: Creates real cloud databases. Only run when developing remote/replica functionality.

Setup: Create .env.local:

TURSO_DB_URI="libsql://your-database.turso.io"
TURSO_AUTH_TOKEN="your-token-here"

Run:

export $(grep -v '^#' .env.local | xargs) && mix test test/turso_remote_test.exs

Tests are skipped by default if credentials are missing.

If function is native to SQLite, update lib/ecto/adapters/libsql/connection.ex:

defp expr({:random, _, []}, _sources, _query) do
  "RANDOM()"
end

Add test:

test "generates RANDOM() function" do
  query = from u in User, select: fragment("RANDOM()")
  assert SQL.all(query) =~ "RANDOM()"
end

Update loaders/dumpers in lib/ecto/adapters/libsql.ex:

def loaders(:boolean, type), do: [&bool_decode/1, type]
defp bool_decode(0), do: {:ok, false}
defp bool_decode(1), do: {:ok, true}

def dumpers(:boolean, type), do: [type, &bool_encode/1]
defp bool_encode(false), do: {:ok, 0}
defp bool_encode(true), do: {:ok, 1}

Transactions track their owning connection via TransactionEntry:

pub struct TransactionEntry {
    pub conn_id: String,        // Connection that owns this transaction
    pub transaction: Transaction,
}

Always validate ownership:

if entry.conn_id != conn_id {
    return Err(rustler::Error::Term(Box::new(
        "Transaction does not belong to this connection",
    )));
}

When a function cannot be implemented due to architectural constraints:

1. Return :unsupported atom error in Rust
2. Document clearly in Elixir wrapper with alternatives
3. Add comprehensive tests asserting unsupported behaviour

Example: freeze_database (see full pattern in original CLAUDE.md if needed)

mix test test/file.exs:123 --trace                  # Trace
cd native/ecto_libsql && cargo test -- --nocapture  # Rust output
mix clean && mix compile                            # Rebuild
for i in {1..10}; do mix test test/file.exs:42; done # Race conditions

STRICT ORDER (do NOT skip steps or reorder):

# 1. Format code (must come FIRST)
mix format && cd native/ecto_libsql && cargo fmt

# 2. Run tests (catch logic errors)
mix test && cd native/ecto_libsql && cargo test

# 3. Verify formatting checks (MUST PASS before commit)
mix format --check-formatted && cd native/ecto_libsql && cargo fmt --check

# 4. Lint checks (optional but recommended)
cd native/ecto_libsql && cargo clippy

# 5. Only commit if all checks above passed
git commit -m "feat: descriptive message"

⚠️ Critical: If ANY check fails, fix it and re-run that check before proceeding. Never commit with failing checks.

1. Update version in mix.exs and native/ecto_libsql/Cargo.toml
2. Update CHANGELOG.md
3. Update README.md if needed
4. Run full test suite
5. Merge to main and tag (e.g. git tag 0.9.0 && git push origin 0.9.0)
6. Wait for the NIF build workflow to complete (builds 6 targets, attaches to GitHub release)
7. Download checksums: mix rustler_precompiled.download EctoLibSql.Native --all --print
   • If no checksum file exists yet: ECTO_LIBSQL_BUILD=true mix rustler_precompiled.download EctoLibSql.Native --all --print
8. Commit the updated checksum-Elixir.EctoLibSql.Native.exs file
9. Publish to Hex: mix hex.publish

Hex package includes: lib/, priv/, native/, checksum-*.exs, documentation files Hex package excludes: test/, examples/, build artifacts

Symptoms: ** (EctoLibSql.Error) database is locked

Solutions:

• Use proper transactions with timeout: Repo.transaction(fn -> ... end, timeout: 15_000)
• Ensure connections are closed in try/after blocks
• Use immediate transactions for writes: begin(state, behavior: :immediate)

Symptoms: ** (Ecto.ChangeError) value does not match type

Solutions:

• Verify schema types match database columns
• Check custom types have loaders/dumpers
• Use cast/3 in changesets for automatic conversion

Symptoms: ** (Ecto.MigrationError) cannot alter column type

Cause: SQLite doesn't support ALTER COLUMN; SQLite < 3.35.0 doesn't support DROP COLUMN

Solution: Use table recreation pattern (see ECTO_MIGRATION_GUIDE.md):

1. Create new table with desired schema
2. Copy data with transformation
3. Drop old table
4. Rename new table
5. Recreate indexes

Symptoms: ** (EctoLibSql.Error) connection failed: authentication error

Solutions:

• Verify credentials: turso db show <name> and turso db tokens create <name>
• Check URI includes libsql:// prefix
• Use replica mode for better error handling (local fallback)

Solutions:

• Ensure cursors are deallocated (streams handle this automatically)
• Close connections properly with try/after
• Use connection pooling with appropriate limits

Symptoms: ** (EctoLibSql.Error) no such function: vector

Solutions:

• Verify LibSQL version in native/ecto_libsql/Cargo.toml
• Use correct vector syntax: EctoLibSql.Native.vector_type(128, :f32)
• Insert with vector([...]) wrapper
• Query with distance functions: vector_distance_cos("column", vec)

# Format & checks (ALWAYS before commit)
mix format --check-formatted
cd native/ecto_libsql && cargo fmt

# Tests
mix test                                    # All Elixir
cd native/ecto_libsql && cargo test         # All Rust
mix test test/file.exs:42 --trace           # Specific with trace

# Quality
cd native/ecto_libsql && cargo clippy       # Lint

# Docs
mix docs                                    # Generate docs

• USAGE.md - API reference for library users
• README.md - User-facing documentation
• CHANGELOG.md - Version history
• ECTO_MIGRATION_GUIDE.md - Migrating from PostgreSQL/MySQL
• RUST_ERROR_HANDLING.md - Error pattern reference
• TESTING.md - Testing strategy and organisation

LibSQL & Turso:

• LibSQL Source
• LibSQL Docs
• Turso Rust SDK

Ecto:

• Ecto Docs
• Ecto.Query
• Ecto.Migration

Rust & NIFs:

• Rustler Docs
• Rust Book
• Async Rust

SQLite:

• SQLite Docs
• SQLite Source

1. ✅ Format code: mix format && cargo fmt
2. ✅ Run tests: mix test && cargo test
3. ✅ Verify formatting: mix format --check-formatted
4. ✅ No .unwrap() in production Rust code
5. ✅ Add tests for new features
6. ✅ Update CHANGELOG.md and relevant docs
7. ✅ Follow existing code patterns

EctoLibSql is a production-ready Ecto adapter for LibSQL/Turso with:

• Full Ecto support (schemas, migrations, queries, associations)
• Three connection modes (local, remote, replica)
• Advanced features (vector search, encryption, streaming)
• Zero panic risk (proper error handling throughout)
• Extensive test coverage
• Comprehensive documentation

Key Principle: Safety first. All Rust code uses proper error handling to protect the BEAM VM. Errors are returned as tuples that can be supervised gracefully.

For AI Agents: Follow critical rules (formatting, Rust error handling), use existing patterns, test thoroughly. This is production code.

Last Updated: 2025-12-30
Maintained By: ocean
License: Apache 2.0
Repository: https://github.com/ocean/ecto_libsql