jim/openai-cookbook
1
0
Fork 0
mirror of https://github.com/james-m-jordan/openai-cookbook.git synced 2025-05-09 19:32:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add Tiktokenizer link in "How to count tokens" (#604)

Browse Source 
This adds a link to Tiktokenizer webapp as another tool, in
addition to the OpenAI Tokenizer.
This commit is contained in:
Eliah Kagan 2023-08-28 13:28:19 -04:00 committed by GitHub
parent 45c6406e8b
commit 63f95154b1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
examples/How_to_count_tokens_with_tiktoken.ipynb
View File

@ -54,7 +54,7 @@
"\n", "\n",
"## How strings are typically tokenized\n", "## How strings are typically tokenized\n",
"\n", "\n",
"In English, tokens commonly range in length from one character to one word (e.g., `\"t\"` or `\" great\"`), though in some languages tokens can be shorter than one character or longer than one word. Spaces are usually grouped with the starts of words (e.g., `\" is\"` instead of `\"is \"` or `\" \"`+`\"is\"`). You can quickly check how a string is tokenized at the [OpenAI Tokenizer](https://beta.openai.com/tokenizer)." "In English, tokens commonly range in length from one character to one word (e.g., `\"t\"` or `\" great\"`), though in some languages tokens can be shorter than one character or longer than one word. Spaces are usually grouped with the starts of words (e.g., `\" is\"` instead of `\"is \"` or `\" \"`+`\"is\"`). You can quickly check how a string is tokenized at the [OpenAI Tokenizer](https://beta.openai.com/tokenizer), or the third-party [Tiktokenizer](https://tiktokenizer.vercel.app/) webapp."
] ]
}, },
{ {