Merge pull request #350 from eugeneyan/main

Update the section token count from 1,000 to 1,600
This commit is contained in:
Ted Sanders 2023-04-15 21:21:19 -07:00 committed by GitHub
commit d67c4181ab
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -411,7 +411,7 @@
"- Shorter sections allow more sections to be retrieved, which may help with recall\n",
"- Overlapping sections may help prevent answers from being cut by section boundaries\n",
"\n",
"Here, we'll use a simple approach and limit sections to 1,000 tokens each, recursively halving any sections that are too long. To avoid cutting in the middle of useful sentences, we'll split along paragraph boundaries when possible."
"Here, we'll use a simple approach and limit sections to 1,600 tokens each, recursively halving any sections that are too long. To avoid cutting in the middle of useful sentences, we'll split along paragraph boundaries when possible."
]
},
{