gpt-tokenizer

Overview of gpt-tokenizer

What it is

gpt-tokenizer is a TypeScript Byte Pair Encoder/Decoder for every OpenAI model family, including GPT-5, GPT-4.1, GPT-4o, o-series reasoning models, gpt-oss, realtime and audio models, and legacy GPT-2/3/4 models. It is a port of OpenAI's tiktoken with additional features. According to the project's Node 26 / o200k_base benchmarks, it is the fastest and lowest-footprint pure-JavaScript GPT tokenizer. The library has been trusted by Microsoft (Teams, GenAIScript), Elastic (Kibana), Effect TS, CodeRabbit, and Rivet by Ironclad.

Key features

• Support for easily tokenizing chats thanks to the encodeChat function - Support for all current OpenAI models, including GPT-5, GPT-4.1, GPT-4o, o1/o3/o4, gpt-oss, realtime, audio, transcription, image, and video models - All OpenAI encodings: r50k_base, p50k_base, p50k_edit, cl100k_base, o200k_base, and o200k_harmony
• Can be loaded and work synchronously (i.e. in non async/await contexts)
• Generator function versions of both the decoder and encoder functions - Provides the ability to decode an asynchronous stream of data (using decodeAsyncGenerator and decodeGenerator with any iterable input)
• Includes a highly performant isWithinTokenLimit function to assess token limit without encoding the entire text/chat

Use cases

Tokenizing text and chat messages for OpenAI models, estimating token counts and API costs with the built-in estimateCost function, decoding model outputs, and checking if input is within a model's token limit.

Getting started

bash npm install gpt-tokenizer

License and pricing

MIT license, open source.

Based on the project's public README and repository metadata.