replace tokenizer with more compatible js-tiktoken
transitive-bullshit
released this
31 May 02:42
·
271 commits
to main
since this release
- Merge pull request #571 from masterkain/js-tiktoken 729b5a4
- switch tokenizer implementation with pure js and more compatible js-tiktoken 3ec62b8
Thanks @masterkain and @dqbd 🙏