Tokenizer

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

What is Tokenizer?

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/microsofttokenizer.svg)](https://yunodb.com/tools/microsofttokenizer?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/microsofttokenizer?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/microsofttokenizer.svg" alt="Featured on YunoDB" height="20"></a>