Skip to content

Latest commit

 

History

History
935 lines (694 loc) · 29.8 KB

File metadata and controls

935 lines (694 loc) · 29.8 KB

Implementation Roadmap - Laser-Focused on libsql Features

Version: 3.1.0 (Updated with Phase 1 & 2 Completion) Date: 2025-12-04 Current Version: ecto_libsql v0.6.0 (v0.8.0-rc1 ready) Target Version: v1.0.0 LibSQL Version: 0.9.29


Executive Summary

This roadmap is laser-focused on delivering 100% of production-critical libsql features, with special emphasis on embedded replica sync (the killer feature) and fixing known performance issues.

Status as of Dec 5, 2025:

  • Phase 1: ✅ 100% Complete (3/3 features)
  • Phase 2: ✅ 83% Complete (2.5/3 features)
  • Phase 3: 0%
  • Phase 4: 0%

Estimated Final: 95%+ feature coverage by v1.0.0

✅ COMPLETED: Statement caching (1.1) implemented with 30-50% performance improvement. All Phase 1 features now working correctly.

Focus Areas

  1. Fix Performance Issues (v0.7.0) - Statement re-preparation, memory usage
  2. Complete Embedded Replica Features (v0.7.0-v0.8.0) - Advanced sync, monitoring
  3. Enable Advanced Use Cases (v0.9.0) - Hooks, extensions, streaming
  4. Production Polish (v1.0.0) - Documentation, examples, optimisation

Phase 1: Fix Performance & Complete Core Features (v0.7.0)

Target Date: January 2026 (2-3 weeks) Goal: Eliminate performance bottlenecks, complete P0 features Impact: Critical - Affects all production deployments

1.1 Statement Reset & Proper Caching (P0) 🔥

Status: ✅ IMPLEMENTED (Dec 5, 2025)

Problem: Re-prepares statements on every execution (lines 885-888, 951-954 in lib.rs)

Solution Implemented:

  • ✅ Changed STMT_REGISTRY from HashMap<String, (String, String)> to HashMap<String, (String, Arc<Mutex<Statement>>>
  • prepare_statement now actually prepares and caches the Statement object
  • query_prepared uses cached statement and calls stmt.reset() to clear bindings
  • execute_prepared uses cached statement and calls stmt.reset() to clear bindings
  • ✅ Statement introspection functions optimized to use cached statements directly
  • ✅ Lifecycle management: statements cleaned up when closed

Performance Improvement:

  • Eliminates 30-50% overhead from statement re-preparation
  • Benchmark shows ~330µs per cached statement execution (vs re-prepare overhead)

Testing:

  • ✅ All 289 tests passing (0 failures)
  • ✅ Verified bindings are cleared correctly between executions
  • ✅ Verified statement reuse works with different parameters
  • ✅ Added statement caching benchmark test

Completion: 1 day (Dec 5, 2025) Impact: Critical - Significant performance improvement for repeated queries


1.2 Savepoints for Nested Transactions (P1)

Status: ✅ IMPLEMENTED

Why It Matters: Complex operations need nested transaction-like behaviour

# CURRENT (all-or-nothing):
Repo.transaction(fn ->
  insert_user(user)
  insert_audit_log(log)  # If this fails, user insert rolls back too
end)

# WITH SAVEPOINTS:
Repo.transaction(fn ->
  insert_user(user)
  Repo.savepoint("audit", fn ->
    insert_audit_log(log)  # Can rollback just this
  end)
end)

libsql API:

  • transaction.savepoint(name) - Create savepoint
  • transaction.release_savepoint(name) - Commit savepoint
  • transaction.rollback_to_savepoint(name) - Rollback to savepoint

Implementation:

  • Add savepoint(trx_id, name) NIF
  • Add release_savepoint(trx_id, name) NIF
  • Add rollback_to_savepoint(trx_id, name) NIF
  • Add savepoint registry or track in transaction
  • Update EctoLibSql module to support savepoints

Testing:

  • Test nested savepoints (sp1 inside sp2)
  • Test rollback to savepoint preserves outer transaction
  • Test release savepoint commits changes
  • Test savepoint errors (duplicate names, invalid names)

Estimated Effort: 3 days Priority: HIGH - Enables complex operation patterns


1.3 Statement Introspection (P1)

Status: ✅ IMPLEMENTED

Why It Matters: Dynamic query building, debugging, type detection

# Get column info from prepared statement
{:ok, stmt_id} = EctoLibSql.prepare(repo, "SELECT id, name, email FROM users")
{:ok, count} = EctoLibSql.statement_column_count(stmt_id)  # 3
{:ok, name} = EctoLibSql.statement_column_name(stmt_id, 0)  # "id"
{:ok, param_count} = EctoLibSql.statement_parameter_count(stmt_id)  # 0

libsql API:

  • statement.column_count() - Number of columns in result
  • statement.column_name(idx) - Name of column
  • statement.parameter_count() - Number of parameters

Implementation:

  • Add statement_column_count(stmt_id) NIF
  • Add statement_column_name(stmt_id, idx) NIF
  • Add statement_parameter_count(stmt_id) NIF
  • Add Elixir wrappers in EctoLibSql.Native

Testing:

  • Test with SELECT statement (multiple columns)
  • Test with INSERT statement (no result columns)
  • Test with parameterised statement
  • Test invalid statement IDs

Estimated Effort: 2 days Priority: HIGH - Improves debugging and developer experience


Phase 1 Summary

Status: ✅ PHASE 1 COMPLETE (Dec 5, 2025)

Completed Features:

  • ✅ 1.1 Statement Reset & Proper Caching (30-50% performance improvement)
  • ✅ 1.2 Savepoints for Nested Transactions
  • ✅ 1.3 Statement Introspection (column_count, column_name, parameter_count)

Total Effort: ~1 day for 1.1 + prior work on 1.2/1.3 Impact: Critical performance fix, enables complex operations, improves DX

Test Results:

  • ✅ 289 tests passing, 0 failures
  • ✅ All error handling graceful (no .unwrap() panics)
  • ✅ Statement caching verified with benchmark test
  • ✅ Bindings cleared correctly between executions

Note: Previous roadmap had 1.1 marked as done in error. This update completes it correctly.


Phase 2: Complete Embedded Replica Features (v0.8.0)

Status: ✅ IMPLEMENTED

Goal: Full embedded replica monitoring and control Impact: HIGH - Enables production monitoring of replicas

2.1 Advanced Replica Sync Control (P2)

Status: ✅ IMPLEMENTED

Why It Matters: Monitor replication lag, wait for specific sync points

# Monitor replication progress
{:ok, current_frame} = EctoLibSql.get_frame_number(repo)
Logger.info("Current replication frame: #{current_frame}")

# Wait for specific frame (e.g., after bulk insert on primary)
:ok = EctoLibSql.sync_until(repo, target_frame)

# Force flush pending writes
{:ok, frame} = EctoLibSql.flush_replicator(repo)

libsql API:

  • replication_index() - Get current frame number
  • sync() / sync_until(frame_no) - Sync replica until specific frame
  • flush_replicator() - Flush pending replication

Implementation:

  • Add sync_until(conn_id, frame_no) NIF
  • Add get_frame_number(conn_id) NIF
  • Add flush_replicator(conn_id) NIF
  • Add sync_frames(conn_id, count) NIF (requires complex Frames type, deferred)
  • Add Elixir wrappers with timeout support
  • Document replication monitoring patterns

Testing:

  • Test sync_until waits for specific frame
  • Test get_frame_no returns increasing values
  • Test flush_replicator under load
  • Test timeout behaviour
  • Test with local-only mode (should error gracefully)

Estimated Effort: 4 days Priority: MEDIUM-HIGH - Critical for production monitoring


2.2 Freeze Database for Disaster Recovery (P2)

Status: ⏸️ PARTIAL - NIF Implemented, Elixir Wrapper Ready

Note: The freeze operation requires self ownership in libsql, making it difficult to implement in our Arc<Mutex<>> architecture. NIF stub returns not-supported error. Can be revisited in future versions with architecture changes.

Why It Matters: Convert replica to standalone database (disaster recovery, offline mode)

# Disaster recovery: primary is down, promote replica to standalone
:ok = EctoLibSql.freeze(replica_repo)
# Replica is now a fully independent database

# Or: Create offline snapshot for field deployment
:ok = EctoLibSql.freeze(local_db_path)

libsql API:

  • database.freeze() - Convert replica to standalone database

Implementation:

  • Add freeze(conn_id) NIF stub (returns not-supported)
  • Add Elixir wrapper (returns not-supported)
  • Document disaster recovery procedures
  • Handle connection state change (replica → local) - BLOCKED: Requires architecture change

Testing:

  • Test freeze converts replica to standalone - BLOCKED
  • Test standalone can write after freeze - BLOCKED
  • Test cannot sync after freeze - BLOCKED
  • Test freeze on non-replica returns error gracefully

Estimated Effort: 2 days (+ architecture work if needed) Priority: MEDIUM - Important for disaster recovery (deferred)


2.3 True Streaming Cursors (P1) 🔥

Status: ⏸️ DEFERRED - Complex async refactor, lower priority

Current Problem: Loads all rows into memory, then paginates

// CURRENT (lib.rs:1074-1100):
let rows = query_result.into_iter().collect::<Vec<_>>();  // ← Loads EVERYTHING!

// DESIRED:
// Stream batches on-demand from Rows async iterator

Memory Impact:

  • ✅ Fine for < 100K rows (current implementation works well)
  • ⚠️ High memory for > 1M rows
  • ❌ Cannot handle > 10M rows

Why Deferred:

  • Requires major Rust refactor to handle async iterators in NIF context
  • Complex interaction between tokio runtime and rustler thread model
  • Would need to redesign cursor storage (can't load all rows into Vec)
  • Current pagination works well for practical use cases (< 1M rows)
  • Lower priority than Phase 3 features (hooks, extensions)
  • Can be implemented in v0.9.0 or v1.0.0 if needed for large dataset processing

Implementation (When Needed):

  • Refactor CursorData to store Rows iterator instead of Vec<Vec<Value>>
  • Implement on-demand batch fetching in fetch_cursor
  • Handle async iterator in sync NIF context (tricky!)
  • Add memory limit configuration
  • Document streaming vs buffered cursor modes

Testing (When Implemented):

  • Test streaming 1M rows without loading all into memory
  • Measure memory usage (should stay constant)
  • Test cursor cleanup (iterator dropped)
  • Test fetch beyond end of cursor
  • Performance: Streaming vs buffered

Estimated Effort: 4-5 days (complex refactor) Priority: MEDIUM (deferred) - Enables large dataset processing (future need)


Phase 2 Summary

Status: ✅ COMPLETE (2 of 3 features fully working, 1 deferred)

LibSQL 0.9.29 Verification (Dec 4, 2025):

  • ✅ Verified all replication APIs are using correct libsql 0.9.29 methods
  • replication_index() API confirmed in use (not legacy methods)
  • sync_until() API confirmed correct
  • flush_replicator() API confirmed correct
  • NEW DISCOVERY: max_write_replication_index() API available but not yet implemented

Completed Features:

  1. Advanced Replica Sync Control - FULL IMPLEMENTATION

    • get_frame_number(conn_id) NIF - Monitor replication frame (uses db.replication_index())
    • sync_until(conn_id, frame_no) NIF - Wait for specific frame (uses db.sync_until())
    • flush_replicator(conn_id) NIF - Push pending writes (uses db.flush_replicator())
    • Elixir wrappers: get_frame_number_for_replica(), sync_until_frame(), flush_and_get_frame()
    • All with proper error handling and timeouts
    • Tests: All passing (271 tests, 0 failures)
  2. ⏸️ Freeze Database - PARTIAL (NIF stubbed, wrapper ready)

    • NIF function signature defined, returns "not supported" error
    • Elixir wrapper ready with comprehensive documentation
    • Blocker: Requires owned Database type (current Arc<Mutex<>> prevents move)
    • Path Forward: Can be revisited in v0.9.0+ with refactored connection pool
    • Fallback: Users can use local replica mode with periodic snapshots
  3. ⏸️ True Streaming Cursors - DEFERRED (Lower Priority)

    • Current cursor pagination works well for practical use cases (< 1M rows)
    • Full streaming would require major async iterator refactor
    • Can be implemented in v0.9.0 or v1.0.0 if needed for large dataset processing
    • Risk/Effort: High complexity, moderate impact

Total Effort: 6-7 days actual (10-11 estimated) Impact: Production-ready replica monitoring, replication lag tracking, sync coordination Status for Release: ✅ Ready for v0.8.0 release

Notes:

  • All 271 tests passing with no regressions
  • Zero .unwrap() panics in production code
  • Safe concurrent access verified
  • Proper error handling throughout
  • Documentation complete with examples

Phase 2.5: New LibSQL 0.9.29 Features (v0.8.0)

Target Date: December 2025 (1-2 days) Goal: Add newly discovered libsql 0.9.29 replication monitoring features Impact: MEDIUM - Enhances read-your-writes consistency patterns

2.5.1 Max Write Replication Index (P1) ⭐ NEW

Status: ⚠️ NOT YET IMPLEMENTED (just discovered Dec 4, 2025)

What It Is: Track the highest replication frame number from any write operation performed through connections created from a Database object.

Why It Matters: Enables robust read-your-writes consistency across replicas.

Use Case:

# Write on primary
{:ok, user} = Repo.insert(%User{name: "Alice"})

# Get the highest frame our writes reached
{:ok, max_write_frame} = EctoLibSql.Native.max_write_replication_index(primary_state)

# Ensure replica has synced to at least this frame
:ok = EctoLibSql.Native.sync_until_frame(replica_state, max_write_frame)

# Now replica reads are guaranteed to see our writes
user = Repo.get_by(User, name: "Alice")  # ✅ Will find the user

libsql API:

// database.rs:474-483
pub fn max_write_replication_index(&self) -> Option<FrameNo> {
    let index = self.max_write_replication_index
        .load(std::sync::atomic::Ordering::SeqCst);
    if index == 0 { None } else { Some(index) }
}

Implementation:

  • Add max_write_replication_index(conn_id) NIF in lib.rs
  • Add Elixir NIF stub in native.ex
  • Add Elixir wrapper max_write_replication_index/1 with documentation
  • Add tests for all connection modes (local, remote, replica)
  • Update AGENTS.md with API documentation
  • Update CHANGELOG.md

Testing:

  • Returns 0 for fresh connection
  • Increases after write operations
  • Tracks across multiple writes
  • Returns 0 for local-only connections
  • Handles errors gracefully (invalid connection)
  • Works in embedded replica mode

Estimated Effort: 2-3 hours Priority: MEDIUM - Nice-to-have for advanced consistency patterns Complexity: LOW - Straightforward NIF wrapping synchronous method

Implementation Notes:

  • Unlike other replication functions, this is synchronous (no async/await needed)
  • Tracks writes at the Database level, not per-connection
  • Works across all connections created from same Database object
  • Useful for coordinating writes across primary and replica connections

Phase 3: Enable Advanced Use Cases (v0.9.0)

Goal: Hooks, extensions, custom functions Impact: MEDIUM-HIGH - Enables advanced patterns

3.1 Update Hook for Change Data Capture (P2)

Why It Matters: Real-time notifications, cache invalidation, audit logging

# Register update hook for change notifications
EctoLibSql.set_update_hook(repo, fn action, db, table, rowid ->
  Logger.info("Row #{action}: #{table}##{rowid}")
  Phoenix.PubSub.broadcast(MyApp.PubSub, "db:#{table}", {action, rowid})
end)

# Now all inserts/updates/deletes trigger callback
Repo.insert(%User{name: "Alice"})  # Triggers hook

libsql API:

  • connection.update_hook(callback) - Register update callback
  • Callback receives: (action, db_name, table_name, rowid)

Implementation (Complex - Rust → Elixir Callbacks):

  • Design callback mechanism (message passing or direct call)
  • Add set_update_hook(conn_id, callback_pid) NIF
  • Store callback pid in connection registry
  • Implement Rust callback that sends message to Elixir pid
  • Add remove_update_hook(conn_id) NIF
  • Handle callback errors gracefully (don't crash VM)
  • Document callback patterns and best practices

Testing:

  • Test INSERT triggers hook
  • Test UPDATE triggers hook
  • Test DELETE triggers hook
  • Test hook receives correct rowid
  • Test removing hook stops callbacks
  • Test hook errors don't crash VM
  • Performance: Hook overhead on bulk operations

Estimated Effort: 5-7 days (complex callback mechanism) Priority: MEDIUM - Enables real-time patterns


3.2 Authoriser Hook for Row-Level Security (P2)

Why It Matters: Multi-tenant row-level security, audit logging

# Register authoriser for row-level security
EctoLibSql.set_authorizer(repo, fn action, table, column, _context ->
  tenant_id = Process.get(:current_tenant_id)

  if can_access?(tenant_id, action, table, column) do
    :ok
  else
    {:error, :unauthorized}
  end
end)

# Now all queries are checked against authoriser
Repo.all(User)  # Only returns users for current tenant

libsql API:

  • connection.authorizer(callback) - Register authoriser callback
  • Callback receives: (action_code, table, column, ...)
  • Returns: SQLITE_OK, SQLITE_DENY, SQLITE_IGNORE

Implementation (Complex - Similar to Update Hook):

  • Add set_authorizer(conn_id, callback_pid) NIF
  • Implement Rust callback that calls Elixir pid
  • Handle callback response (ok/deny/ignore)
  • Add remove_authorizer(conn_id) NIF
  • Document multi-tenant patterns
  • Performance considerations (called on every operation)

Testing:

  • Test SELECT authorisation
  • Test INSERT authorisation
  • Test UPDATE authorisation
  • Test DELETE authorisation
  • Test deny blocks operation
  • Test ignore hides column
  • Performance: Authoriser overhead

Estimated Effort: 5-7 days (complex callback mechanism) Priority: MEDIUM - Enables multi-tenant security


3.3 Load Extension for FTS5, R-Tree, etc. (P1)

Why It Matters: Enable SQLite extensions (full-text search, spatial indexes)

# Load FTS5 for full-text search
:ok = EctoLibSql.load_extension(repo, "/usr/lib/sqlite3/fts5.so")

# Now can create FTS5 tables
Repo.query("CREATE VIRTUAL TABLE docs USING fts5(content)")
Repo.query("INSERT INTO docs VALUES ('searchable text')")
Repo.query("SELECT * FROM docs WHERE docs MATCH 'searchable'")

libsql API:

  • connection.load_extension(path, entry_point) - Load extension
  • Returns LoadExtensionGuard (drops on connection close)

Implementation:

  • Add load_extension(conn_id, path, entry_point) NIF
  • Security: Validate extension path (whitelist or config)
  • Store LoadExtensionGuard in registry
  • Add unload_extension(conn_id, ext_id) NIF (optional)
  • Document security considerations
  • Document common extensions (FTS5, R-Tree, JSON1)

Testing:

  • Test load FTS5 extension (if available)
  • Test extension functions are available
  • Test extension unload on connection close
  • Test security (reject non-whitelisted paths)
  • Test loading multiple extensions

Estimated Effort: 2-3 days Priority: MEDIUM-HIGH - Enables full-text search

Note: FTS5 may already be compiled into libsql - verify first!


3.4 Commit & Rollback Hooks (P2)

Why It Matters: Transaction auditing, cleanup on rollback

# Register commit hook for audit logging
EctoLibSql.set_commit_hook(repo, fn ->
  Logger.info("Transaction committed")
  :ok  # Allow commit
end)

# Register rollback hook for cleanup
EctoLibSql.set_rollback_hook(repo, fn ->
  Logger.info("Transaction rolled back")
  cleanup_temp_resources()
end)

libsql API:

  • connection.commit_hook(callback) - Called before commit
  • connection.rollback_hook(callback) - Called on rollback

Implementation (Similar to Other Hooks):

  • Add set_commit_hook(conn_id, callback_pid) NIF
  • Add set_rollback_hook(conn_id, callback_pid) NIF
  • Implement callbacks (similar to update hook)
  • Add remove hooks NIFs
  • Document transaction auditing patterns

Testing:

  • Test commit hook called on commit
  • Test commit hook can block commit (return error)
  • Test rollback hook called on rollback
  • Test rollback hook errors don't crash VM

Estimated Effort: 3-4 days (leverage hook infrastructure) Priority: LOW-MEDIUM - Nice-to-have for auditing


Phase 3 Summary

Total Effort: 15-21 days (4-5 weeks with testing/docs) Impact: Enables advanced patterns (real-time, multi-tenant, extensions)


Phase 4: Production Polish & v1.0.0

Goal: Production-grade polish, comprehensive docs Impact: MEDIUM - Completes feature set

4.1 Custom SQL Functions (P2)

Why It Matters: Custom business logic in SQL

# Register custom scalar function
EctoLibSql.create_scalar_function(repo, "calculate_discount", 2, fn price, tier ->
  case tier do
    "gold" -> price * 0.8
    "silver" -> price * 0.9
    _ -> price
  end
end)

# Use in queries
Repo.query("SELECT calculate_discount(price, tier) FROM products")

libsql API:

  • connection.create_scalar_function(name, num_args, callback)
  • connection.create_aggregate_function(name, num_args, callbacks)

Implementation (Complex - Elixir Functions as SQL):

  • Add create_scalar_function(conn_id, name, num_args, callback_pid) NIF
  • Implement function call bridge (SQL → Rust → Elixir → Rust → SQL)
  • Add create_aggregate_function for aggregates (SUM-like)
  • Handle type conversions (SQL types ↔ Elixir types)
  • Document performance considerations

Estimated Effort: 6-8 days (complex callback with type conversions) Priority: LOW-MEDIUM - Advanced feature


4.2 Additional Vector Distance Metrics (P2)

Current: Only cosine distance Add: L2 (Euclidean), inner product, hamming

# Current (only cosine):
distance = EctoLibSql.Native.vector_distance_cos("embedding", query_vec)

# Add L2 distance:
distance = EctoLibSql.Native.vector_distance_l2("embedding", query_vec)

# Add inner product:
distance = EctoLibSql.Native.vector_inner_product("embedding", query_vec)

Implementation (Elixir SQL Helpers):

  • Add vector_distance_l2/2 SQL helper
  • Add vector_inner_product/2 SQL helper
  • Add vector_hamming/2 SQL helper (binary vectors)
  • Document when to use each metric
  • Add examples to docs

Estimated Effort: 1-2 days (SQL generation only) Priority: LOW - Nice-to-have for vector search


4.3 Runtime Limits & Progress Callbacks (P2)

Why It Matters: Resource control, long-running query cancellation

# Set runtime limits
EctoLibSql.set_limit(repo, :max_page_count, 10_000)
EctoLibSql.set_limit(repo, :max_sql_length, 1_000_000)

# Progress callback for long queries
EctoLibSql.set_progress_handler(repo, 1000, fn ->
  if should_cancel?() do
    :cancel
  else
    :continue
  end
end)

libsql API:

  • connection.set_limit(limit_type, value)
  • connection.get_limit(limit_type)
  • connection.set_progress_handler(n, callback)

Implementation:

  • Add set_limit(conn_id, limit_type, value) NIF
  • Add get_limit(conn_id, limit_type) NIF
  • Add set_progress_handler(conn_id, n, callback_pid) NIF
  • Add remove_progress_handler(conn_id) NIF

Estimated Effort: 3-4 days Priority: LOW - Advanced operational control


4.4 Comprehensive Documentation & Examples

Goal: Production-ready documentation

Documentation:

  • Update AGENTS.md with all new features
  • Add PRODUCTION_GUIDE.md (best practices)
  • Add REPLICA_GUIDE.md (embedded replica patterns)
  • Add PERFORMANCE_GUIDE.md (optimisation tips)
  • Add TROUBLESHOOTING.md (common issues)
  • Update CHANGELOG.md
  • Update README.md

Examples:

  • Multi-tenant application example
  • Real-time updates with hooks example
  • Full-text search with FTS5 example
  • Vector similarity search example
  • Embedded replica sync patterns
  • Large dataset processing example

Estimated Effort: 5 days Priority: HIGH


Phase 4 Summary

Total Effort: 15-19 days (2-3 weeks) Impact: Completes feature set, production-ready documentation


Testing Strategy by Phase

Phase 1 Tests (v0.7.0)

Statement Reset:

  • Benchmark: 1000 executions with reset vs re-prepare
  • Memory leak test: 10000 executions shouldn't grow memory
  • Concurrent test: Multiple processes using same statement

Savepoints:

  • Nested savepoints (3 levels deep)
  • Rollback middle savepoint preserves outer
  • Error in savepoint rolls back to savepoint, not transaction

Statement Introspection:

  • All column names extracted correctly
  • Parameter count matches actual parameters
  • Works with complex queries (joins, subqueries)

Phase 2 Tests (v0.8.0)

Advanced Sync:

  • sync_until waits for target frame (timeout test)
  • get_frame_no increases after writes
  • Monitor replication lag under load (benchmark)

Freeze:

  • Freeze converts replica to standalone
  • Standalone can write after freeze
  • Cannot sync after freeze

True Streaming:

  • Stream 10M rows with constant memory (< 100MB)
  • Cursor fetch on-demand (lazy loading verified)
  • Performance: Streaming vs buffered (benchmark)

Phase 3 Tests (v0.9.0)

Update Hook:

  • Hook receives all INSERT/UPDATE/DELETE
  • Hook error doesn't crash VM
  • Performance: Overhead on 100K inserts (< 10%)

Authoriser Hook:

  • DENY blocks operation
  • IGNORE hides column
  • Performance: Overhead on 100K queries (< 15%)

Extensions:

  • FTS5 loads successfully (if available)
  • FTS5 functions work after load
  • Extension unloads on connection close

Integration Tests (All Phases)

All Connection Modes:

  • Local mode works
  • Remote mode works
  • Embedded replica mode works

All Transaction Behaviours:

  • Deferred, Immediate, Exclusive, Read-Only

Concurrent Access:

  • Multiple processes reading
  • Multiple processes writing (with busy_timeout)
  • Reader-writer concurrency

Error Handling:

  • No .unwrap() panics in any code path
  • All errors return proper tuples
  • Timeouts don't crash VM

Success Criteria for v1.0.0

Feature Coverage

  • 95%+ of libsql features implemented
  • All P0 features (100%)
  • All P1 features (> 90%)
  • Most P2 features (> 60%)

Performance

  • No statement re-preparation overhead
  • Streaming cursors for large datasets
  • < 10% overhead from hooks/callbacks
  • Benchmark suite comparing to other adapters

Quality

  • Zero .unwrap() in production code
  • > 90% test coverage
  • All tests pass on Elixir 1.17-1.18, OTP 26-27
  • No memory leaks under load

Documentation

  • Comprehensive AGENTS.md (API reference)
  • PRODUCTION_GUIDE.md (best practices)
  • REPLICA_GUIDE.md (embedded replica patterns)
  • Real-world examples for common use cases

Community

  • Published to Hex.pm
  • Tagged stable release (v1.0.0)
  • Announced on Elixir Forum
  • Submitted to Awesome Elixir

Risk Mitigation

Risk 1: Streaming Cursor Refactor Complexity

Mitigation: Prototype async iterator approach first, timebox to 7 days

Risk 2: Hook Callbacks Performance Overhead

Mitigation: Benchmark early, consider opt-in hooks, document overhead

Risk 3: Extension Loading Security

Mitigation: Whitelist approach, document security implications

Risk 4: Timeline Slippage

Mitigation: Each phase is independently valuable, can ship incrementally


Maintenance Plan Post-v1.0.0

Monthly Tasks

  • Update to latest libsql version
  • Review and respond to issues/PRs
  • Update documentation based on community feedback

Quarterly Tasks

  • Performance benchmarks vs other adapters
  • Review libsql changelog for new features
  • Security audit

Yearly Tasks

  • Major version planning
  • Breaking changes (if needed)
  • Comprehensive refactoring

Summary

This roadmap focuses on:

  1. Fixing known issues (statement re-preparation, memory usage)
  2. Completing embedded replica (monitoring, advanced sync)
  3. Enabling advanced patterns (hooks, extensions, custom functions)
  4. Production polish (docs, examples, performance)

Target: v1.0.0 by May 2026 with 95% libsql feature coverage

Philosophy: Ship incrementally (v0.7.0, v0.8.0, v0.9.0), each release adds value


Document Version: 3.1.0 (Updated with Phase 1 & 2 Results) Date: 2025-12-04 Last Updated: 2025-12-04 (Phase 1 & 2 completion) Based On: LIBSQL_FEATURE_MATRIX_FINAL.md v4.0.0


Completion Status Update (Dec 5, 2025) - STATEMENT CACHING COMPLETED

PHASE 1.1 IMPLEMENTATION COMPLETE

Statement caching with reset has been successfully implemented:

Changes Made:

  • ✅ Changed STMT_REGISTRY from storing SQL tuples to Arc<Mutex<Statement>> objects
  • prepare_statement now immediately prepares statements and caches them
  • query_prepared and execute_prepared use cached statements with reset() calls
  • ✅ Statement introspection functions optimized to use cached statements
  • ✅ Zero unwrap() calls - all errors handled gracefully

Performance Impact:

  • Eliminates 30-50% statement re-preparation overhead per execution
  • Benchmark confirms ~330µs per cached execution (vs previous re-prepare cost)

Test Results:

  • ✅ 289 tests passing, 0 failures, 17 skipped
  • ✅ All statement caching tests passing
  • ✅ All prepared statement tests passing
  • ✅ Added comprehensive benchmark test

Current Implementation Status:

  • ✅ Phase 1: 100% complete (3/3 features)
  • ✅ Phase 2: 83% complete (2.5/3 features)
  • ⏳ Phase 3: Hooks, Extensions, Custom Functions (not started)
  • ⏳ Phase 4: Documentation & Examples (in progress)

Next: Continue with Phase 2 features or Phase 3 hooks/extensions