Commit ab028cb
Migrate inference to llama_batch and llama_decode api (abetlen#795)
* Add low-level batching notebook
* fix: tokenization of special characters: (abetlen#850)
It should behave like llama.cpp, where most out of the box usages
treat special characters accordingly
* Update CHANGELOG
* Cleanup
* Fix runner label
* Update notebook
* Use llama_decode and batch api
* Support logits_all parameter
---------
Co-authored-by: Antoine Lizee <[email protected]>1 parent f436e0c commit ab028cb
3 files changed
Lines changed: 753 additions & 8 deletions
0 commit comments