Release notes from Sudachi

Sudachi version 0.7.5

2024-11-05T05:55:27Z

Highlights

Behavior of the dictionary printer and builder are changed (#234)
- DictionaryPrinter now prints word references in the (Surface, POS, Reading) triple format, instead of the line number format.
- DictionaryBuilder now allows the dictionary form to be written in the triple format, not only the line number format.

Added

Benchmark scripts are added (#235)

Fixed

Tutorial and readme are updated (#237, #240)
Config.Resource.asByteBuffer now always returns ByteBuffer with little endian byte order (#239)
- StringUtil.readAllBytes also now returns ByteBuffer with little endian byte order.

Sudachi version 0.7.4

2024-07-02T07:27:42Z

Highlights

Add Tokenizer.lazyTokenizeSentences(SplitMode mode, Readable input), that performs analysis lazily and saves memory usage (#231)
- Tokenizer.tokenizeSentences(SplitMode mode, Reader input) is marked as deprecated.

Fixed

Do not segfault on tokenizing with closed dictionary (#217)
The default config sudachi.json sets non-existent property joinKanjiNumeric in JoinNumericPlugin (#221)
fix incorrect size calculation when expand (#227)
Update tutorial.md (#226)

Sudachi version 0.7.3

2023-06-26T02:09:00Z

This is a support release for Elasticsearch/OpenSearch integration 3.1.0 release.

Highlights

Added Config.fromResource method for reading Configs vial PathAnchor. (#212)

Internals

Plugin classloading is done by PathAnchor and support multiple classloaders (#210, #209)

Notes about v0.7.2

Release v0.7.2 contains subset of the functionality of this release but did not contain crucial features. It is not a broken release, but there are no user-visible changed from v0.7.1.

v0.7.2

2023-06-15T02:25:54Z

bump version -> v0.7.2

Sudachi version 0.7.1

2023-03-09T09:51:28Z

This is a maintenance release

Highlights

Fixed analysis truncation when using analysis with sentence splitting and the input does not contain data which can be treated as splittable sentences
Fixed O(N^2) performance in sentence splitting when underlying reader does not fill buffer fully at once
Stop calling into reader with full buffer

0.6.4

2023-03-09T09:51:14Z

This is a maintenance release

Highlights

Fixed analysis truncation when using analysis with sentence splitting and the input does not contain data which can be treated as splittable sentences
Fixed O(N^2) performance in sentence splitting when underlying reader does not fill buffer fully at once
Stop calling into reader with full buffer

Sudachi version 0.6.3

2022-08-29T12:50:39Z

Port relaxed boundary mode from 0.7.0 while keeping ABI compatibility with pre-0.7.0 versions.

Sudachi version 0.7.0

2022-08-16T03:00:38Z

Highlights

Tokenizer.tokenize API returns MorphemeList instead of List<Morpheme>. This change is ABI-incompatible with previous versions and applications which use Sudachi require recompilation. The change should be source-compatible with no changes required to the source code which uses Sudachi.
New API: MorphemeList.split: resplit C-mode token sequence to lower level without re-analyzing the whole string.
Added relaxed boundary matching mode for Regex OOV handler

Sudachi version 0.6.2

2022-06-21T01:05:24Z

Highlights

Fixed invalid POS tags which appeared when using user-defined POS tags both in user dictionaries and OOV handlers. You are not affected by this bug if you did not use user-defined POS in OOV handlers.

Sudachi version 0.6.1

2022-06-10T08:45:35Z

Highlights

DO NOT USE 0.6.0, IT IS INCOMPATIBLE WITH 0.6.1
Regex OOV plugin has configurable maximum token length
SettingsAnchor renamed to PathAnchor to make more clear its purpose
Add useful Config methods, e.g. for a common case of loading default configuration with provided PathAnchor to resolve default paths in another directory.
Filesystem-based PathAnchor now plays correctly with SecurityManager present (e.g. in ElasticSearch).

Regex OOV length

Use maxLength field of the plugin configuration object to set maximum allowed length, in utf-8 bytes (by default 32). The unit will change to unicode codepoints in the future.