jim/openai-cookbook
1
0
Fork 0
mirror of https://github.com/james-m-jordan/openai-cookbook.git synced 2025-05-09 19:32:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

add tiktoken-go to How_to_count_tokens_with_tiktoken.ipynb (#605)

Browse Source
This commit is contained in:
richzw 2023-07-25 06:54:52 +08:00 committed by GitHub
parent 5e050080ab
commit 964dd85ed3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
examples/How_to_count_tokens_with_tiktoken.ipynb
View File

@ -39,6 +39,7 @@
"- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)\n", "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)\n",
"- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken), [TiktokenSharp](https://github.com/aiqinxuancai/TiktokenSharp)\n", "- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken), [TiktokenSharp](https://github.com/aiqinxuancai/TiktokenSharp)\n",
"- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n", "- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n",
"- Golang: [tiktoken-go](https://github.com/pkoukk/tiktoken-go)\n",
"\n", "\n",
"For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n", "For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n",
"- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md) (or alternatively [GPT2TokenizerFast](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast))\n", "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md) (or alternatively [GPT2TokenizerFast](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast))\n",
@ -46,6 +47,7 @@
"- .NET / C#: [GPT Tokenizer](https://github.com/dluc/openai-tools)\n", "- .NET / C#: [GPT Tokenizer](https://github.com/dluc/openai-tools)\n",
"- Java: [gpt2-tokenizer-java](https://github.com/hyunwoongko/gpt2-tokenizer-java)\n", "- Java: [gpt2-tokenizer-java](https://github.com/hyunwoongko/gpt2-tokenizer-java)\n",
"- PHP: [GPT-3-Encoder-PHP](https://github.com/CodeRevolutionPlugins/GPT-3-Encoder-PHP)\n", "- PHP: [GPT-3-Encoder-PHP](https://github.com/CodeRevolutionPlugins/GPT-3-Encoder-PHP)\n",
"- Golang: [tiktoken-go](https://github.com/pkoukk/tiktoken-go)\n",
"\n", "\n",
"(OpenAI makes no endorsements or guarantees of third-party libraries.)\n", "(OpenAI makes no endorsements or guarantees of third-party libraries.)\n",
"\n", "\n",