… math.

Gardening pass on the base64 library to make it production-ready for
serving workloads. Three categories of changes:

**API improvements**: The encode/decode functions now use proper IREE
span types (iree_const_byte_span_t, iree_byte_span_t,
iree_mutable_string_view_t) instead of separate pointer+length
parameters. This keeps related fields together and matches IREE's
allocator and buffer conventions. Added iree_base64_encode() which was
previously missing — needed for any future base64 output paths.

**Overflow protection**: All size computations now use
iree_host_size_checked_add/mul. The critical fix is in
iree_base64_encoded_size() where ((data_length + 2) / 3) * 4 wraps on
pathological inputs (exploitable on 32-bit targets like WASM where IREE
runs). The function returns IREE_HOST_SIZE_MAX as an overflow sentinel —
never a valid encoded size since the result is always a multiple of 4.
The decode path's (L/4)*3 is provably safe (contraction), but uses
checked math as defense-in-depth policy.

**Testing and benchmarking**: Comprehensive encode tests (RFC 4648
vectors, size calculation, buffer-too-small, single byte values),
full 256-value encode→decode roundtrip, length-sweep roundtrip
(0-33 bytes covering all remainder cases), explicit overflow rejection
tests. Fuzzer now does bidirectional testing: random bytes as base64
input AND random binary through encode→decode roundtrip with
trap-on-mismatch. Benchmark covers encode/decode/roundtrip at 6 sizes
(4B-64KB) — sustained throughput is ~3 GB/s encode, ~2 GB/s decode.

Co-Authored-By: Claude <[email protected]>

…support.

Adds end-to-end validation of the tiktoken loader against real OpenAI
encoding files (cl100k_base, o200k_base, r50k_base, p50k_base) and
cross-validation against equivalent HuggingFace tokenizer.json files.

Fixes a bug where the tiktoken parser rejected files with rank gaps.
Real p50k_base has a gap at rank 50256 (reserved for <|endoftext|>
special token). The parser now fills gaps with placeholder entries and
downstream code (vocab construction, merge reconstruction) skips them.

New files:
- tiktoken_smoketest.py: Downloads .tiktoken files from OpenAI CDN,
  compares IREE output against tiktoken Python library (dev mode) or
  stored goldens (ci/verify mode), with cross-validation support.
- run_tiktoken_smoketest.sh: uvx wrapper with Python dependencies.
- generate_tiktoken_golden_ids.py: Generates golden token IDs using
  the tiktoken Python library for the test corpus.

Changes:
- tiktoken.c: Handle non-contiguous ranks by inserting zero-length
  placeholder entries at gap positions. Use add_token_with_id for
  explicit ID assignment. Skip placeholders in merge reconstruction.
- tiktoken_test.cc: Add ConstructWithRankGap test with synthetic
  gapped data. Rename ParseNonContiguousRanks to ParseBackwardRanks.
- run_benchmarks.sh: Add tiktoken encoding downloads and benchmark
  entries alongside existing HuggingFace tokenizers.
- tokenizer_corpus.json: Add golden IDs for four tiktoken encodings
  (72 test cases). Compact integer arrays to single lines.
- huggingface_smoketest.py: Compact integer arrays when writing corpus.

Co-Authored-By: Claude <[email protected]>

…base speedup.

The pair validation cache previously only stored TRUE results. When
encodings with many overlapping tokens (like p50k_base's 24 whitespace
tokens for 2-25 consecutive spaces) encounter code with long alignment
runs, many candidate token pairs fail validation. These FALSE results
were re-validated on every occurrence — 23,719 redundant calls with
8.35M hash lookup iterations for a 256KB corpus.

The fix stores both TRUE and FALSE results using bit 31 of the cached
token2 field as a validity flag (token IDs never exceed 2^31). Cached
FALSE results are only used when deferred_merge_rank == 0; when deferral
is active, cached FALSE entries trigger re-validation since deferral may
change the result. This is safe because cached TRUE is monotonically
valid (a pair valid with no deferral is valid with any deferral).

Results (O3/march=native/thin_lto, 256KB code corpus):
  p50k_base Code: 1.02 MB/s -> 6.48 MB/s (6.3x speedup)
  All other encodings: within noise (+-3%), no regressions.
  Benchmarked across cl100k_base, o200k_base, r50k_base, gpt2, llama3,
  gemma-2b, qwen2.5, mistral-nemo, deepseek-v3.

Co-Authored-By: Claude <[email protected]>