jim/openai-cookbook
1
0
Fork 0
mirror of https://github.com/james-m-jordan/openai-cookbook.git synced 2025-05-09 19:32:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update chat finetune data prep notebook (#1074)

Browse Source 
Co-authored-by: andrewpeng@openai.com <Andrew Peng>
This commit is contained in:
Andrew Peng 2024-02-26 17:51:50 -08:00 committed by GitHub
parent 1e1dd5a33d
commit d25d32ed5f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
examples/Chat_finetuning_data_prep.ipynb
View File

@ -83,7 +83,7 @@
"1. **Data Type Check**: Checks whether each entry in the dataset is a dictionary (`dict`). Error type: `data_type`.\n", "1. **Data Type Check**: Checks whether each entry in the dataset is a dictionary (`dict`). Error type: `data_type`.\n",
"2. **Presence of Message List**: Checks if a `messages` list is present in each entry. Error type: `missing_messages_list`.\n", "2. **Presence of Message List**: Checks if a `messages` list is present in each entry. Error type: `missing_messages_list`.\n",
"3. **Message Keys Check**: Validates that each message in the `messages` list contains the keys `role` and `content`. Error type: `message_missing_key`.\n", "3. **Message Keys Check**: Validates that each message in the `messages` list contains the keys `role` and `content`. Error type: `message_missing_key`.\n",
"4. **Unrecognized Keys in Messages**: Logs if a message has keys other than `role`, `content`, and `name`. Error type: `message_unrecognized_key`.\n", "4. **Unrecognized Keys in Messages**: Logs if a message has keys other than `role`, `content`, `weight`, `function_call`, and `name`. Error type: `message_unrecognized_key`.\n",
"5. **Role Validation**: Ensures the `role` is one of \"system\", \"user\", or \"assistant\". Error type: `unrecognized_role`.\n", "5. **Role Validation**: Ensures the `role` is one of \"system\", \"user\", or \"assistant\". Error type: `unrecognized_role`.\n",
"6. **Content Validation**: Verifies that `content` has textual data and is a string. Error type: `missing_content`.\n", "6. **Content Validation**: Verifies that `content` has textual data and is a string. Error type: `missing_content`.\n",
"7. **Assistant Message Presence**: Checks that each conversation has at least one message from the assistant. Error type: `example_missing_assistant_message`.\n", "7. **Assistant Message Presence**: Checks that each conversation has at least one message from the assistant. Error type: `example_missing_assistant_message`.\n",
@ -123,7 +123,7 @@
" if \"role\" not in message or \"content\" not in message:\n", " if \"role\" not in message or \"content\" not in message:\n",
" format_errors[\"message_missing_key\"] += 1\n", " format_errors[\"message_missing_key\"] += 1\n",
" \n", " \n",
" if any(k not in (\"role\", \"content\", \"name\", \"function_call\") for k in message):\n", " if any(k not in (\"role\", \"content\", \"name\", \"function_call\", \"weight\") for k in message):\n",
" format_errors[\"message_unrecognized_key\"] += 1\n", " format_errors[\"message_unrecognized_key\"] += 1\n",
" \n", " \n",
" if message.get(\"role\", None) not in (\"system\", \"user\", \"assistant\", \"function\"):\n", " if message.get(\"role\", None) not in (\"system\", \"user\", \"assistant\", \"function\"):\n",