{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Meta Prompting: A Guide to Automated Prompt Optimization\n", "\n", "Welcome to our cookbook on meta prompting! In this guide, we'll explore how to take a basic prompt and refine it to enhance the quality of outputs from a language model. We'll use the example of summarizing news reports to illustrate the process.\n", "\n", "\n", "Meta-prompting is a technique where you use an LLM to generate or improve prompts. Typically this is done using a higher intelligent model that optimizes prompts for a model with less intelligence. It’s a process of using prompts to guide, structure, and optimize other prompts, helping ensure they’re more effective in guiding the LLM towards high-quality, relevant outputs. We'll be leveraging the capabilities of `o1-preview`, a more intelligent model with advanced reasoning skills, to improve a prompt for `gpt-4o`.\n", "\n", "We're committed to making your development journey with LLMs smoother and more accessible through this technique. Don't forget to check out our [Generate Anything](https://platform.openai.com/docs/guides/prompt-generation) feature in the playground — it's a fantastic starting point to dive into meta prompting.\n", "\n", "In this example, we'll begin with a simple prompt for summarizing news articles and then enhance it to see how the outputs improve. We'll use `o1-preview` to analyze and refine our prompt, adding more detail and clarity along the way. Finally, we'll evaluate the outputs systematically to understand the impact of our refinements." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] } ], "source": [ "import pandas as pd\n", "import openai \n", "from concurrent.futures import ThreadPoolExecutor, as_completed\n", "from tqdm import tqdm\n", "from pydantic import BaseModel\n", "from datasets import load_dataset\n", "\n", "client = openai.Client()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Importing the Data\n", "\n", "Let's kick things off by importing the `bbc_news_alltime` dataset from [HuggingFace](https://huggingface.co/datasets/RealTimeData/bbc_news_alltime). This dataset contains all BBC News articles, capturing everything published monthly from 2017 up to the latest complete month. For our experiment, we'll focus exclusively on a sample from a recent month—August 2024—to keep things current and manageable.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | title | \n", "published_date | \n", "authors | \n", "description | \n", "section | \n", "content | \n", "link | \n", "top_image | \n", "
---|---|---|---|---|---|---|---|---|
2662 | \n", "Laura Whitmore: I was gaslighted after raising... | \n", "2024-08-04 | \n", "https://www.facebook.com/bbcnews | \n", "The former Love Island host said that things s... | \n", "Culture | \n", "Television presenter Laura Whitmore has said t... | \n", "http://www.bbc.co.uk/news/articles/c9wvwvzm7x7o | \n", "https://ichef.bbci.co.uk/ace/standard/2560/cps... | \n", "
1865 | \n", "Errollyn Wallen appointed as Master of the Kin... | \n", "2024-08-25 | \n", "https://www.facebook.com/bbcnews | \n", "She is best known for her work on the 2012 Par... | \n", "Culture | \n", "Celebrated composer and singer-songwriter Erro... | \n", "http://www.bbc.co.uk/news/articles/c4gl758g7zgo | \n", "https://ichef.bbci.co.uk/ace/standard/2560/cps... | \n", "
2554 | \n", "SDLP: Matthew O'Toole endorses Claire Hanna fo... | \n", "2024-08-30 | \n", "https://www.facebook.com/bbcnews | \n", "Matthew O'Toole had been named by some as a po... | \n", "Northern Ireland Politics | \n", "Matthew O'Toole leads his party's official opp... | \n", "http://www.bbc.co.uk/news/articles/cvg41j7xrzdo | \n", "https://ichef.bbci.co.uk/ace/standard/3840/cps... | \n", "
1338 | \n", "Rotherham rioters among those jailed - BBC News | \n", "2024-08-20 | \n", "https://www.facebook.com/bbcnews | \n", "Two men who were part of a mob targeting a Hol... | \n", "South Yorkshire | \n", "Rotherham pair among those jailed for UK rioti... | \n", "http://www.bbc.co.uk/news/articles/cwywggd7qw6o | \n", "https://ichef.bbci.co.uk/ace/standard/2560/cps... | \n", "
1232 | \n", "BBC News - BBC iPlayer | \n", "2024-08-02 | \n", "None | \n", "None | \n", "None | \n", "JavaScript seems to be disabled. Please enable... | \n", "http://www.bbc.co.uk/news/10318089 | \n", "\n", " |
\n", " | title | \n", "published_date | \n", "authors | \n", "description | \n", "section | \n", "content | \n", "link | \n", "top_image | \n", "simple_summary | \n", "complex_summary | \n", "
---|---|---|---|---|---|---|---|---|---|---|
2662 | \n", "Laura Whitmore: I was gaslighted after raising... | \n", "2024-08-04 | \n", "https://www.facebook.com/bbcnews | \n", "The former Love Island host said that things s... | \n", "Culture | \n", "Television presenter Laura Whitmore has said t... | \n", "http://www.bbc.co.uk/news/articles/c9wvwvzm7x7o | \n", "https://ichef.bbci.co.uk/ace/standard/2560/cps... | \n", "Television presenter Laura Whitmore has spoken... | \n", "1. **Type of News**: Entertainment/Television\\... | \n", "
1865 | \n", "Errollyn Wallen appointed as Master of the Kin... | \n", "2024-08-25 | \n", "https://www.facebook.com/bbcnews | \n", "She is best known for her work on the 2012 Par... | \n", "Culture | \n", "Celebrated composer and singer-songwriter Erro... | \n", "http://www.bbc.co.uk/news/articles/c4gl758g7zgo | \n", "https://ichef.bbci.co.uk/ace/standard/2560/cps... | \n", "Errollyn Wallen has been appointed Master of t... | \n", "1. **Type of News**: Arts/Music\\n\\n2. **Summar... | \n", "
2554 | \n", "SDLP: Matthew O'Toole endorses Claire Hanna fo... | \n", "2024-08-30 | \n", "https://www.facebook.com/bbcnews | \n", "Matthew O'Toole had been named by some as a po... | \n", "Northern Ireland Politics | \n", "Matthew O'Toole leads his party's official opp... | \n", "http://www.bbc.co.uk/news/articles/cvg41j7xrzdo | \n", "https://ichef.bbci.co.uk/ace/standard/3840/cps... | \n", "Matthew O'Toole, the leader of the official op... | \n", "1. **Type of News**: Politics\\n\\n2. **Summary*... | \n", "
1338 | \n", "Rotherham rioters among those jailed - BBC News | \n", "2024-08-20 | \n", "https://www.facebook.com/bbcnews | \n", "Two men who were part of a mob targeting a Hol... | \n", "South Yorkshire | \n", "Rotherham pair among those jailed for UK rioti... | \n", "http://www.bbc.co.uk/news/articles/cwywggd7qw6o | \n", "https://ichef.bbci.co.uk/ace/standard/2560/cps... | \n", "Two men, Nathan Palmer (29) and Niven Matthewm... | \n", "1. **Type of News**: Politics / Crime and Just... | \n", "
1232 | \n", "BBC News - BBC iPlayer | \n", "2024-08-02 | \n", "None | \n", "None | \n", "None | \n", "JavaScript seems to be disabled. Please enable... | \n", "http://www.bbc.co.uk/news/10318089 | \n", "\n", " | The article discusses the need to enable JavaS... | \n", "I cannot provide a summary of the article as t... | \n", "
\n", " | title | \n", "published_date | \n", "authors | \n", "description | \n", "section | \n", "content | \n", "link | \n", "top_image | \n", "simple_summary | \n", "complex_summary | \n", "simple_evaluation | \n", "complex_evaluation | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|
2662 | \n", "Laura Whitmore: I was gaslighted after raising... | \n", "2024-08-04 | \n", "https://www.facebook.com/bbcnews | \n", "The former Love Island host said that things s... | \n", "Culture | \n", "Television presenter Laura Whitmore has said t... | \n", "http://www.bbc.co.uk/news/articles/c9wvwvzm7x7o | \n", "https://ichef.bbci.co.uk/ace/standard/2560/cps... | \n", "Television presenter Laura Whitmore has spoken... | \n", "1. **Type of News**: Entertainment/Television\\... | \n", "categorization=4 keyword_extraction=3 sentimen... | \n", "categorization=5 keyword_extraction=5 sentimen... | \n", "
1865 | \n", "Errollyn Wallen appointed as Master of the Kin... | \n", "2024-08-25 | \n", "https://www.facebook.com/bbcnews | \n", "She is best known for her work on the 2012 Par... | \n", "Culture | \n", "Celebrated composer and singer-songwriter Erro... | \n", "http://www.bbc.co.uk/news/articles/c4gl758g7zgo | \n", "https://ichef.bbci.co.uk/ace/standard/2560/cps... | \n", "Errollyn Wallen has been appointed Master of t... | \n", "1. **Type of News**: Arts/Music\\n\\n2. **Summar... | \n", "categorization=4 keyword_extraction=4 sentimen... | \n", "categorization=5 keyword_extraction=5 sentimen... | \n", "
2554 | \n", "SDLP: Matthew O'Toole endorses Claire Hanna fo... | \n", "2024-08-30 | \n", "https://www.facebook.com/bbcnews | \n", "Matthew O'Toole had been named by some as a po... | \n", "Northern Ireland Politics | \n", "Matthew O'Toole leads his party's official opp... | \n", "http://www.bbc.co.uk/news/articles/cvg41j7xrzdo | \n", "https://ichef.bbci.co.uk/ace/standard/3840/cps... | \n", "Matthew O'Toole, the leader of the official op... | \n", "1. **Type of News**: Politics\\n\\n2. **Summary*... | \n", "categorization=5 keyword_extraction=4 sentimen... | \n", "categorization=5 keyword_extraction=5 sentimen... | \n", "
1338 | \n", "Rotherham rioters among those jailed - BBC News | \n", "2024-08-20 | \n", "https://www.facebook.com/bbcnews | \n", "Two men who were part of a mob targeting a Hol... | \n", "South Yorkshire | \n", "Rotherham pair among those jailed for UK rioti... | \n", "http://www.bbc.co.uk/news/articles/cwywggd7qw6o | \n", "https://ichef.bbci.co.uk/ace/standard/2560/cps... | \n", "Two men, Nathan Palmer (29) and Niven Matthewm... | \n", "1. **Type of News**: Politics / Crime and Just... | \n", "categorization=3 keyword_extraction=3 sentimen... | \n", "categorization=5 keyword_extraction=4 sentimen... | \n", "
1232 | \n", "BBC News - BBC iPlayer | \n", "2024-08-02 | \n", "None | \n", "None | \n", "None | \n", "JavaScript seems to be disabled. Please enable... | \n", "http://www.bbc.co.uk/news/10318089 | \n", "\n", " | The article discusses the need to enable JavaS... | \n", "I cannot provide a summary of the article as t... | \n", "categorization=2 keyword_extraction=3 sentimen... | \n", "categorization=1 keyword_extraction=1 sentimen... | \n", "