Summary

53% faster combined parse+render time, 61% fewer object allocations on the ThemeRunner benchmark (real Shopify theme templates with production-like data). Zero test regressions — all 974 unit tests pass.

Measured with YJIT enabled on Ruby 3.4, using performance/bench_quick.rb (best of 3 runs, 10 iterations each with GC disabled, after 20-iteration warmup).

Methodology

This PR was developed through ~120 automated experiments using an autoresearch loop: edit → commit → run tests → benchmark → keep/discard. Each change was validated against the full unit test suite before benchmarking. Changes that regressed either correctness or the primary metric were reverted immediately.

The approach was allocation-driven: profile where objects are created, eliminate the ones that aren't needed, and defer the ones that are. With GC consuming 74% of total CPU time, every avoided allocation has outsized impact on wall-clock performance.

Architecture changes

1. Cursor class (lib/liquid/cursor.rb)

A StringScanner wrapper with higher-level methods tuned for Liquid's grammar. One Cursor per ParseContext, reused across all tag/variable/expression parsing:

cursor = parse_context.cursor
cursor.reset(markup)
cursor.skip_ws
tag_name = cursor.scan_tag_name   # C-level regex via StringScanner
cursor.expect_id("in")            # zero-alloc: regex skip + byte compare
cursor.skip_fragment              # zero-alloc: regex skip

Key insight from tenderlove's article on fast tokenizers: C-level StringScanner.scan/skip with compiled regexes is 2-3x faster than Ruby-level peek_byte/scan_byte loops. Methods that previously had 20+ lines of manual byte scanning are now 1-3 line regex delegations.

2. String#byteindex tokenizer

Replaced StringScanner-based tokenizer with String#byteindex for finding {% and {{ delimiters. The tokenizer accounts for ~30% of parse time, and byteindex('{', pos) is ~40% faster than StringScanner#skip_until(/\{[\{\%]/) for single-byte searching. Variable token scanning uses manual byte inspection matching the original tokenizer's exact edge-case handling (unclosed tags, {{ → {% nesting).

3. Zero-Lexer variable parsing

100% of variables in the benchmark (1,197) now parse through Variable#try_fast_parse — a byte-level scanner that extracts the name expression and filter chain without touching the Lexer or Parser. Zero Lexer/Parser fallbacks. Even multi-argument filters like pluralize: 'item', 'items' are scanned directly with comma-separated arg handling. Only keyword arguments (key: value) would fall through (none appear in the benchmark).

What changed (by impact)

Parse optimizations (~61% faster, ~38K fewer allocs)

Replaced StringScanner tokenizer with String#byteindex. Single-byte byteindex searching is ~40% faster than regex-based skip_until. This alone reduced parse time by ~12%.

Pure-byte parse_tag_token. Eliminated the costly StringScanner#string= reset that was called for every {% %} token (878 times). Manual byte scanning for tag name + markup extraction is faster than resetting and re-scanning via StringScanner.

Replaced regex with Cursor scanning in hot paths. FullToken regex → Cursor, VariableParser regex → manual byte scanner, For#Syntax regex → Cursor, If#SIMPLE_CONDITION regex → Cursor, INTEGER_REGEX/FLOAT_REGEX → Cursor scan_number, WhitespaceOrNothing regex → match?.

Fast-path Variable initialization. All variables parse through try_fast_parse which extracts name + filters via byte-level scanning. Cached no-arg filter tuples (NO_ARG_FILTER_CACHE) avoid repeated [name, EMPTY_ARRAY] creation.

Fast-path VariableLookup. simple_lookup? uses match? regex (8x faster than byte scan). Simple identifier chains skip scan_variable entirely.

Avoid unnecessary string allocations. Expression.parse skips strip when no whitespace. Variable fast-path reuses markup string directly when possible. block_delimiter strings cached per tag name.

Render optimizations (~20% faster, ~3K fewer allocs)

Splat-free filter invocation. invoke_single/invoke_two avoid *args array allocation for 90% of filter calls.

Primitive type fast paths. find_variable returns immediately for String/Integer/Float/Array/Hash/nil — skipping to_liquid and respond_to?(:context=). Same in VariableLookup#evaluate. Hash fast-path via instance_of?(Hash) before respond_to? chain.

Cached small integer to_s. Pre-computed frozen strings for 0-999 avoid 267 Integer#to_s allocations per render.

Condition#evaluate fast path. Skip loop do...end block when no child_relation — avoids closure allocation for all benchmark conditions.

While loop for If#@blocks.each. Avoids Proc creation for 1-2 element arrays (YJIT optimizes each better for long arrays, but while wins for short ones).

Lazy initialization. Context defers StringScanner and @interrupts. Registers defers @changes hash. static_environments uses EMPTY_ARRAY when empty.

Code simplified

The Cursor consolidation replaced ~150 scattered getbyte/byteslice calls with a shared vocabulary. Example:

# Before: 15 lines of manual byte scanning
def scan_id
  start = @ss.pos
  b = @ss.peek_byte
  return unless b && ((b >= 97 && b <= 122) || (b >= 65 && b <= 90) || b == USCORE)
  @ss.scan_byte
  while (b = @ss.peek_byte)
    break unless (b >= 97 && b <= 122) || ...
    @ss.scan_byte
  end
  @source.byteslice(start, @ss.pos - start)
end

# After: C-level regex is 2-3x faster
ID_REGEX = /[a-zA-Z_][\w-]*\??/
def scan_id = @ss.scan(ID_REGEX)

What did NOT work

• Split-based tokenizer — String#split with regex is 2.5x faster but can't handle {{ followed by %} (variable-becomes-tag nesting that Liquid supports)
• Tag name interning via byte-based perfect hash — collision issues, and verification loop overhead kills the speed gain
• String#match for name extraction — MatchData creates +5K allocs, far worse than manual scanning
• while loops replacing each in hot render paths — YJIT optimizes each better for many-iteration loops; only wins for short 1-2 element arrays
• Shared expression cache across templates — leaks state between parses, grows unboundedly
• TruthyCondition subclass — YJIT polymorphism at evaluate call site hurts more than 115 saved allocs

Benchmark reproduction

cd performance
bundle exec ruby bench_quick.rb   # single run
# or
./auto/autoresearch.sh            # tests + 3-run best-of

Files changed

• lib/liquid/cursor.rb — new Cursor class (StringScanner wrapper with regex-based methods)
• lib/liquid/tokenizer.rb — String#byteindex-based tokenizer replacing StringScanner
• lib/liquid/block_body.rb — Cursor-based tag/variable parsing, regex blank_string?
• lib/liquid/variable.rb — try_fast_parse with multi-arg filter support, NO_ARG_FILTER_CACHE, invoke_single/invoke_two render dispatch
• lib/liquid/variable_lookup.rb — simple_lookup? regex, parse_simple fast path, primitive type fast paths in evaluate
• lib/liquid/expression.rb — byte-level parse_number, conditional strip
• lib/liquid/context.rb — invoke_single/invoke_two, primitive fast paths in find_variable, lazy init
• lib/liquid/condition.rb — evaluate fast path skipping loop block for simple conditions
• lib/liquid/strainer_template.rb — invoke_single/invoke_two dispatch
• lib/liquid/tags/if.rb — Cursor conditions, while-loop render, inlined to_liquid_value
• lib/liquid/tags/for.rb — Cursor-based lax_parse
• lib/liquid/block.rb — cached block_delimiter strings
• lib/liquid/registers.rb — lazy @changes hash
• lib/liquid/utils.rb — cached small integer to_s, lazy seen hash, slice_collection Array fast path
• lib/liquid/parse_context.rb — Cursor instance
• lib/liquid/resource_limits.rb — expose last_capture_length for render loop optimization