Skip to content

Cache empty results for missing strikes to prevent repeated DuckDB queries #14

@uberdeveloper

Description

@uberdeveloper

Problem

When a strategy requests a strike that doesn't exist in the parquet data (e.g. far OTM wings with 2000+ point hedges), the engine queries DuckDB on every bar because empty results are never cached.

In _lazy_fetch() (context.py:414):

if data:
    self._cache[instrument_key] = data  # only caches non-empty results

A missing strike returns {}, which is falsy, so it's never written to cache. The next bar sees a cache miss and queries DuckDB again.

Impact

For a 60-day backtest with a 365-tick clock and 2 missing strikes:

  • ~43,800 wasted DuckDB queries (60 × 365 × 2) that all return empty
  • 2400-point hedge: 68.7s runtime vs 10.8s after fix

Fix

Cache the empty result so subsequent bars skip the query entirely:

# Before:
if data:
    self._cache[instrument_key] = data

# After:
self._cache[instrument_key] = data  # cache {} for missing strikes too

Subsequent bars find the empty dict in cache, hit the if not inst_data: return None, -1 guard, and return immediately with no database call.

Benchmarks

All results verified — PnL, trades, win rate, max drawdown are identical before and after.

Config Without fix With fix Speedup
1800 hedge 16.4s 7.1s 2.3x
2400 hedge 68.7s 10.8s 6.4x

Scope

  • src/fastbt/backtest/context.py_lazy_fetch() method (1 line change)
  • No changes to DataSource ABC, engine, or strategy interfaces
  • 312 backtest tests pass

Reported by: @darkseid during iron condor backtest with wide hedge wings.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformancePerformance optimization or degradation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions