Skip to content

Optimize ConnectionCostMatrix layout and Viterbi hot loop performance#600

Merged
mosuka merged 1 commit intomainfrom
refactoring
Jan 10, 2026
Merged

Optimize ConnectionCostMatrix layout and Viterbi hot loop performance#600
mosuka merged 1 commit intomainfrom
refactoring

Conversation

@mosuka
Copy link
Copy Markdown
Member

@mosuka mosuka commented Jan 10, 2026

Optimize ConnectionCostMatrix layout and Viterbi hot loop performance

  • Transpose ConnectionCostMatrix memory layout from [forward_id][backward_id] to [backward_id][forward_id] to improve cache hit rate in the Viterbi search loop.
  • Introduce a version flag in Compiled ConnectionCostMatrix (matrix.mtx) to support the new layout while maintaining backward compatibility for older dictionary formats.
  • Add #[inline] attributes to hot methods in ConnectionCostMatrix, Mode, and Penalty.
  • Specialize Lattice::add_edge_in_lattice for Mode::Normal to eliminate penalty calculation overhead during standard tokenization.
  • Significant performance improvements observed in IPADIC benchmarks:
    • bench-tokenize-ipadic: about 30 percent faster (15.4 us to 13.3 us)
    • bench-tokenize-long-text-ipadic: about 16 percent faster
    • bench-constructor-ipadic: about 35 percent faster

@mosuka mosuka changed the title Optimize Lattice structure using flat buffer and linked list Optimize ConnectionCostMatrix layout and Viterbi hot loop performance Jan 10, 2026
@mosuka mosuka merged commit d36fd3a into main Jan 10, 2026
8 checks passed
@mosuka mosuka deleted the refactoring branch January 10, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant