2024.1.0.0

What's Changed

• New operations:
  • TrieTokenizer
  • VocabEncoder
  • EqualStr
  • RaggedToSparse
  • RaggedToRagged
  • FuzeRagged
• Update existing operations:
  • Add max_splits argument to RegexSplit
  • Add encoding argument to CaseFold
• Add new and update existing TensorFlow translators for TextVectorization layer partial support.
• RWKV tokenizer support.
• New way to get OpenVINO Tokenizers - build from files. Supports RWKV tokenizer.
• Update tokenizer operation caching mechanism for OpenVINO model caching support
• SentencePiece tokenizer changes and fixes:
  • Update to 0.2.0 version
  • Use constant 0 as mask hide token by @as-suvorov in #90
  • Sentencepiece BOS Token Detection
• Fix multi-input model merging by @yas-sim in #53

New Contributors

• @dependabot made their first contribution in #30
• @yas-sim made their first contribution in #53
• @as-suvorov made their first contribution in #90
• @akladiev made their first contribution in #102

Full Changelog: 2024.0.0.0...2024.1.0.0