From fc5af842a22ac4a6383d7e5d27abcd6f337ede9e Mon Sep 17 00:00:00 2001 From: Roger Zurawicki Date: Mon, 16 Oct 2023 13:21:24 -0400 Subject: [PATCH] Add tiktoken-rs to third-party library (#791) --- examples/How_to_count_tokens_with_tiktoken.ipynb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/examples/How_to_count_tokens_with_tiktoken.ipynb b/examples/How_to_count_tokens_with_tiktoken.ipynb index 74e3923..8dd00d1 100644 --- a/examples/How_to_count_tokens_with_tiktoken.ipynb +++ b/examples/How_to_count_tokens_with_tiktoken.ipynb @@ -40,6 +40,7 @@ "- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken), [TiktokenSharp](https://github.com/aiqinxuancai/TiktokenSharp)\n", "- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n", "- Golang: [tiktoken-go](https://github.com/pkoukk/tiktoken-go)\n", + "- Rust: [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs)\n", "\n", "For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n", "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md) (or alternatively [GPT2TokenizerFast](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast))\n", @@ -48,6 +49,7 @@ "- Java: [gpt2-tokenizer-java](https://github.com/hyunwoongko/gpt2-tokenizer-java)\n", "- PHP: [GPT-3-Encoder-PHP](https://github.com/CodeRevolutionPlugins/GPT-3-Encoder-PHP)\n", "- Golang: [tiktoken-go](https://github.com/pkoukk/tiktoken-go)\n", + "- Rust: [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs)\n", "\n", "(OpenAI makes no endorsements or guarantees of third-party libraries.)\n", "\n",