Commits

Breadcrumbs

History for

onmain

User selector

All users

All time

Commit History

Commits on Mar 15, 2026

• chore: bump version to 0.9.1 and relax regexr version constraint

  Show description for 730a96a
  farhan-syah

  committed
  730a96a

  Copy full SHA for 730a96a

Commits on Feb 27, 2026

• feat: add SentencePiece unigram tokenizer

  Show description for d31fe5c
  farhan-syah

  committed
  d31fe5c

  Copy full SHA for d31fe5c

Commits on Dec 24, 2025

• chore: update version to 0.8.0 and improve version script

  Show description for 2cebef7
  farhan-syah

  committed
  2cebef7

  Copy full SHA for 2cebef7

• test: add comprehensive Mistral V3 tokenizer tests

  Show description for dd80fe9
  farhan-syah

  committed
  dd80fe9

  Copy full SHA for dd80fe9

• docs: add Mistral tokenizer documentation and examples

  Show description for 4deb352
  farhan-syah

  committed
  4deb352

  Copy full SHA for 4deb352

• feat: add Mistral V1/V2/V3 tokenizer support with SentencePiece mode

  Show description for 2d246f3
  farhan-syah

  committed
  2d246f3

  Copy full SHA for 2d246f3

• fix: upgrade regexr to v0.1.0-beta.5 to fix UTF-8 boundary panics

  Show description for 8807f8d
  farhan-syah

  committed
  8807f8d

  Copy full SHA for 8807f8d

• feat: add jit() method and improve regex backend switching

  Show description for fcbdd67
  farhan-syah

  committed
  fcbdd67

  Copy full SHA for fcbdd67

Commits on Dec 2, 2025

• feat: make regexr default backend with optional PCRE2 support

  Show description for cd6ffe6
  farhan-syah

  committed
  cd6ffe6

  Copy full SHA for cd6ffe6

Commits on Nov 26, 2025

• chore: bump version to 0.6.0

  farhan-syah

  committed
  b1761be

  Copy full SHA for b1761be

• feat: add ByteLevel streaming decoder for ByteLevel tokenizers

  Show description for 9e45c14
  farhan-syah

  committed
  9e45c14

  Copy full SHA for 9e45c14

• feat: add DeepSeek V3 tokenizer with ByteLevel BPE support

  Show description for f04b820
  farhan-syah

  committed
  f04b820

  Copy full SHA for f04b820

• feat: add comprehensive benchmark scripts for tokenizers

  Show description for f18227a
  farhan-syah

  committed
  f18227a

  Copy full SHA for f18227a

• chore: bump version to 0.4.0

  Show description for ea0a493
  farhan-syah

  committed
  ea0a493

  Copy full SHA for ea0a493

• test: add comprehensive test suites for all tokenizers

  Show description for d6d836f
  farhan-syah

  committed
  d6d836f

  Copy full SHA for d6d836f

• feat: add complete Llama 3 family support

  Show description for b060c82
  farhan-syah

  committed
  b060c82

  Copy full SHA for b060c82

• chore: update version references to 0.3.0

  farhan-syah

  committed
  f0d176d

  Copy full SHA for f0d176d

• feat: add 54 agent special tokens for conversational AI and reasoning

  Show description for 88e090a
  farhan-syah

  committed
  88e090a

  Copy full SHA for 88e090a

• feat: initial implementation of splintr BPE tokenizer

  Show description for 976bf19
  farhan-syah

  committed
  976bf19

  Copy full SHA for 976bf19

Loading