{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Multi-Tool Orchestration with RAG approach using OpenAI's Responses API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "This cookbook guides you through building dynamic, multi-tool workflows using OpenAI's Responses API. It demonstrates how to implement a Retrieval-Augmented Generation (RAG) approach that intelligently routes user queries to the appropriate in-built or external tools. Whether your query calls for general knowledge or requires accessing specific internal context from a vector database (like Pinecone), this guide shows you how to integrate function calls, web searches in-built tool, and leverage document retrieval to generate accurate, context-aware responses." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", "Note: you may need to restart the kernel to use updated packages.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/shikhar/openai_projects/github_repos/success-git/success_new/success/oneoffs/shikhar/responses_rag_cookbook/env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] } ], "source": [ "#%pip install datasets tqdm pandas pinecone openai --quiet\n", "\n", "import os\n", "import time\n", "from tqdm.auto import tqdm\n", "from pandas import DataFrame\n", "from datasets import load_dataset\n", "import random\n", "import string\n", "\n", "\n", "# Import OpenAI client and initialize with your API key.\n", "from openai import OpenAI\n", "\n", "client = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n", "\n", "# Import Pinecone client and related specifications.\n", "from pinecone import Pinecone\n", "from pinecone import ServerlessSpec" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example we use a sample medical reasoning dataset from Hugging Face. We convert the dataset into a Pandas DataFrame and merge the “Question” and “Response” columns into a single string. This merged text is used for embedding and later stored as metadata." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Example merged text: Question: A 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions? Answer: Cystometry in this case of stress urinary incontinence would most likely reveal a normal post-void residual volume, as stress incontinence typically does not involve issues with bladder emptying. Additionally, since stress urinary incontinence is primarily related to physical exertion and not an overactive bladder, you would not expect to see any involuntary detrusor contractions during the test.\n" ] } ], "source": [ "# Load the dataset (ensure you're logged in with huggingface-cli if needed)\n", "ds = load_dataset(\"FreedomIntelligence/medical-o1-reasoning-SFT\", \"en\", split='train[:100]', trust_remote_code=True)\n", "ds_dataframe = DataFrame(ds)\n", "\n", "# Merge the Question and Response columns into a single string.\n", "ds_dataframe['merged'] = ds_dataframe.apply(\n", " lambda row: f\"Question: {row['Question']} Answer: {row['Response']}\", axis=1\n", ")\n", "print(\"Example merged text:\", ds_dataframe['merged'].iloc[0])" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
QuestionComplex_CoTResponsemerged
0A 61-year-old woman with a long history of inv...Okay, let's think about this step by step. The...Cystometry in this case of stress urinary inco...Question: A 61-year-old woman with a long hist...
1A 45-year-old man with a history of alcohol us...Alright, let’s break this down. We have a 45-y...Considering the clinical presentation of sudde...Question: A 45-year-old man with a history of ...
2A 45-year-old man presents with symptoms inclu...Okay, so here's a 45-year-old guy who's experi...Based on the clinical findings presented—wide-...Question: A 45-year-old man presents with symp...
3A patient with psoriasis was treated with syst...I'm thinking about this patient with psoriasis...The development of generalized pustules in a p...Question: A patient with psoriasis was treated...
4What is the most likely diagnosis for a 2-year...Okay, so we're dealing with a 2-year-old child...Based on the described symptoms and the unusua...Question: What is the most likely diagnosis fo...
...............
95An electrical current flows along a flat plate...Alright, to find out the temperature at the ce...The correct answer is F. 1549°F.Question: An electrical current flows along a ...
96A herpetologist bitten by a poisonous snake is...Alright, so we're dealing with a case where a ...The snake venom is most likely affecting the a...Question: A herpetologist bitten by a poisonou...
97A 34 years old person has rapidly developing c...Alright, let's break down what's happening wit...The symptoms described in the question fit mos...Question: A 34 years old person has rapidly de...
98What is the term used to describe the type of ...Okay, so I need to figure out what kind of inj...The term used to describe the type of injury c...Question: What is the term used to describe th...
99During the process of chlorination of water, t...Alright, let's think this through starting fro...The effective disinfecting action during the c...Question: During the process of chlorination o...
\n", "

100 rows × 4 columns

\n", "
" ], "text/plain": [ " Question \\\n", "0 A 61-year-old woman with a long history of inv... \n", "1 A 45-year-old man with a history of alcohol us... \n", "2 A 45-year-old man presents with symptoms inclu... \n", "3 A patient with psoriasis was treated with syst... \n", "4 What is the most likely diagnosis for a 2-year... \n", ".. ... \n", "95 An electrical current flows along a flat plate... \n", "96 A herpetologist bitten by a poisonous snake is... \n", "97 A 34 years old person has rapidly developing c... \n", "98 What is the term used to describe the type of ... \n", "99 During the process of chlorination of water, t... \n", "\n", " Complex_CoT \\\n", "0 Okay, let's think about this step by step. The... \n", "1 Alright, let’s break this down. We have a 45-y... \n", "2 Okay, so here's a 45-year-old guy who's experi... \n", "3 I'm thinking about this patient with psoriasis... \n", "4 Okay, so we're dealing with a 2-year-old child... \n", ".. ... \n", "95 Alright, to find out the temperature at the ce... \n", "96 Alright, so we're dealing with a case where a ... \n", "97 Alright, let's break down what's happening wit... \n", "98 Okay, so I need to figure out what kind of inj... \n", "99 Alright, let's think this through starting fro... \n", "\n", " Response \\\n", "0 Cystometry in this case of stress urinary inco... \n", "1 Considering the clinical presentation of sudde... \n", "2 Based on the clinical findings presented—wide-... \n", "3 The development of generalized pustules in a p... \n", "4 Based on the described symptoms and the unusua... \n", ".. ... \n", "95 The correct answer is F. 1549°F. \n", "96 The snake venom is most likely affecting the a... \n", "97 The symptoms described in the question fit mos... \n", "98 The term used to describe the type of injury c... \n", "99 The effective disinfecting action during the c... \n", "\n", " merged \n", "0 Question: A 61-year-old woman with a long hist... \n", "1 Question: A 45-year-old man with a history of ... \n", "2 Question: A 45-year-old man presents with symp... \n", "3 Question: A patient with psoriasis was treated... \n", "4 Question: What is the most likely diagnosis fo... \n", ".. ... \n", "95 Question: An electrical current flows along a ... \n", "96 Question: A herpetologist bitten by a poisonou... \n", "97 Question: A 34 years old person has rapidly de... \n", "98 Question: What is the term used to describe th... \n", "99 Question: During the process of chlorination o... \n", "\n", "[100 rows x 4 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds_dataframe" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create a Pinecone Index Based on the Dataset\n", "Use the dataset itself to determine the embedding dimensionality. For example, compute one embedding from the merged column and then create the index accordingly." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Embedding dimension: 1536\n" ] } ], "source": [ "MODEL = \"text-embedding-3-small\" # Replace with your production embedding model if needed\n", "# Compute an embedding for the first document to obtain the embedding dimension.\n", "sample_embedding_resp = client.embeddings.create(\n", " input=[ds_dataframe['merged'].iloc[0]],\n", " model=MODEL\n", ")\n", "embed_dim = len(sample_embedding_resp.data[0].embedding)\n", "print(f\"Embedding dimension: {embed_dim}\")\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Index stats: {'dimension': 1536,\n", " 'index_fullness': 0.0,\n", " 'metric': 'dotproduct',\n", " 'namespaces': {},\n", " 'total_vector_count': 0,\n", " 'vector_type': 'dense'}\n" ] } ], "source": [ "\n", "# Initialize Pinecone using your API key.\n", "pc = Pinecone(api_key=os.getenv(\"PINECONE_API_KEY\"))\n", "\n", "# Define the Pinecone serverless specification.\n", "AWS_REGION = \"us-east-1\"\n", "spec = ServerlessSpec(cloud=\"aws\", region=AWS_REGION)\n", "\n", "# Create a random index name with lower case alphanumeric characters and '-'\n", "index_name = 'pinecone-index-' + ''.join(random.choices(string.ascii_lowercase + string.digits, k=10))\n", "\n", "# Create the index if it doesn't already exist.\n", "if index_name not in pc.list_indexes().names():\n", " pc.create_index(\n", " index_name,\n", " dimension=embed_dim,\n", " metric='dotproduct',\n", " spec=spec\n", " )\n", "\n", "# Connect to the index.\n", "index = pc.Index(index_name)\n", "time.sleep(1)\n", "print(\"Index stats:\", index.describe_index_stats())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Upsert the Dataset into Pinecone index\n", "\n", "Process the dataset in batches, generate embeddings for each merged text, prepare metadata (including separate Question and Answer fields), and upsert each batch into the index. You may also update metadata for specific entries if needed." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Upserting to Pinecone: 100%|██████████| 4/4 [00:06<00:00, 1.64s/it]\n" ] } ], "source": [ "batch_size = 32\n", "for i in tqdm(range(0, len(ds_dataframe['merged']), batch_size), desc=\"Upserting to Pinecone\"):\n", " i_end = min(i + batch_size, len(ds_dataframe['merged']))\n", " lines_batch = ds_dataframe['merged'][i: i_end]\n", " ids_batch = [str(n) for n in range(i, i_end)]\n", " \n", " # Create embeddings for the current batch.\n", " res = client.embeddings.create(input=[line for line in lines_batch], model=MODEL)\n", " embeds = [record.embedding for record in res.data]\n", " \n", " # Prepare metadata by extracting original Question and Answer.\n", " meta = []\n", " for record in ds_dataframe.iloc[i:i_end].to_dict('records'):\n", " q_text = record['Question']\n", " a_text = record['Response']\n", " # Optionally update metadata for specific entries.\n", " meta.append({\"Question\": q_text, \"Answer\": a_text})\n", " \n", " # Upsert the batch into Pinecone.\n", " vectors = list(zip(ids_batch, embeds, meta))\n", " index.upsert(vectors=vectors)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![Pinecone Image](../../images/responses_pinecone_rag.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Query the Pinecone Index\n", "\n", "Create a natural language query, compute its embedding, and perform a similarity search on the Pinecone index. The returned results include metadata that provides context for generating answers." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "def query_pinecone_index(client, index, model, query_text):\n", " # Generate an embedding for the query.\n", " query_embedding = client.embeddings.create(input=query_text, model=model).data[0].embedding\n", "\n", " # Query the index and return top 5 matches.\n", " res = index.query(vector=[query_embedding], top_k=5, include_metadata=True)\n", " print(\"Query Results:\")\n", " for match in res['matches']:\n", " print(f\"{match['score']:.2f}: {match['metadata'].get('Question', 'N/A')} - {match['metadata'].get('Answer', 'N/A')}\")\n", " return res" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Query Results:\n", "0.70: A 45-year-old man with a history of alcohol use, who has been abstinent for the past 10 years, presents with sudden onset dysarthria, shuffling gait, and intention tremors. Given this clinical presentation and history, what is the most likely diagnosis? - Considering the clinical presentation of sudden onset dysarthria, shuffling gait, and intention tremors in a 45-year-old man with a history of alcohol use who has been abstinent for the past 10 years, the most likely diagnosis is acquired hepatocerebral degeneration.\n", "\n", "This condition is associated with chronic liver disease, which can often be a consequence of long-term alcohol use. Despite the patient's abstinence from alcohol for a decade, previous alcohol use may have led to underlying liver dysfunction. This dysfunction, even if subclinical, can cause encephalopathy due to the accumulation of neurotoxic substances that affect the brain. The sudden onset of these neurological symptoms aligns with how acquired hepatocerebral degeneration can manifest, making it a probable diagnosis in this scenario.\n", "0.55: A 45-year-old man presents with symptoms including a wide-based gait, a blank facial expression, hallucinations, memory issues, a resting tremor that resolves with movement, and bradykinesia. Based on these clinical findings, what is most likely to be observed in the histological specimen of his brain? - Based on the clinical findings presented—wide-based gait, blank facial expression, hallucinations, memory issues, resting tremor that resolves with movement, and bradykinesia—it is likely that the 45-year-old man is experiencing a condition related to Parkinsonism, possibly Parkinson's disease or dementia with Lewy bodies. Both of these conditions are associated with the presence of Lewy bodies in the brain. Lewy bodies are abnormal aggregates of protein, primarily alpha-synuclein, which can cause both the motor and cognitive symptoms observed in this patient. Therefore, in the histological specimen of his brain, you would most likely observe the presence of Lewy bodies.\n", "0.53: A 73-year-old man is evaluated for increasing forgetfulness, getting lost while walking, irritability, and difficulty recalling recent events while retaining detailed memories from over 20 years ago. On examination, he is oriented to person and place but disoriented to time, and an MRI of the brain reveals significant changes. Considering these symptoms and the imaging findings, what is the most likely underlying pathological process contributing to the patient's condition? - The symptoms and MRI findings of this 73-year-old man suggest the most likely underlying pathological process is the buildup of amyloid-beta plaques and tau protein tangles, which are characteristic of Alzheimer's disease. These changes often begin in brain regions involved in memory, such as the hippocampus and temporal lobes, leading to the gradual memory decline, disorientation, and personality changes observed in the patient.\n", "0.42: A 2-day-old male newborn delivered at 36 weeks presents with generalized convulsions, lethargy, feeding difficulties, icterus, purpura, posterior uveitis, and failed auditory screening. Cranial ultrasonography shows ventricular dilatation and hyperechoic foci in multiple brain areas. Considering these clinical signs and history, what is the most likely diagnosis? - The symptoms and findings you've described in this 2-day-old newborn point towards congenital Toxoplasmosis. The combination of neurological symptoms (such as convulsions and ventricular dilatation with hyperechoic foci), the presence of posterior uveitis, and the skin manifestations like purpura, all fit into the classic presentation of a TORCH infection. Toxoplasmosis, specifically, is known to cause widespread calcifications in the brain, not limited to the periventricular areas, which matches the ultrasound findings. Additionally, while hearing loss is more traditionally associated with CMV, it can also occur in Toxoplasmosis. Thus, the most likely diagnosis given this clinical picture is congenital Toxoplasmosis.\n", "0.42: A 45-year-old male patient experiences double vision specifically when walking upstairs. Considering his well-controlled history of Type-II diabetes, which cranial nerve is most likely involved in his symptoms? - Based on the symptoms described, the cranial nerve most likely involved in the double vision experienced by this patient while walking upstairs is the trochlear nerve, or cranial nerve IV. This nerve controls the superior oblique muscle, which plays a role in stabilizing the eye during certain movements, including the coordination required when looking upwards while walking upstairs. Given the patient's history of diabetes, cranial neuropathies can occur, and CN IV involvement can lead to vertical diplopia that becomes noticeable during specific activities like walking up stairs. Therefore, the trochlear nerve is a likely candidate for the involvement in these symptoms.\n" ] }, { "data": { "text/plain": [ "{'matches': [{'id': '1',\n", " 'metadata': {'Answer': 'Considering the clinical presentation of '\n", " 'sudden onset dysarthria, shuffling gait, '\n", " 'and intention tremors in a 45-year-old '\n", " 'man with a history of alcohol use who '\n", " 'has been abstinent for the past 10 '\n", " 'years, the most likely diagnosis is '\n", " 'acquired hepatocerebral degeneration.\\n'\n", " '\\n'\n", " 'This condition is associated with '\n", " 'chronic liver disease, which can often '\n", " 'be a consequence of long-term alcohol '\n", " \"use. Despite the patient's abstinence \"\n", " 'from alcohol for a decade, previous '\n", " 'alcohol use may have led to underlying '\n", " 'liver dysfunction. This dysfunction, '\n", " 'even if subclinical, can cause '\n", " 'encephalopathy due to the accumulation '\n", " 'of neurotoxic substances that affect the '\n", " 'brain. The sudden onset of these '\n", " 'neurological symptoms aligns with how '\n", " 'acquired hepatocerebral degeneration can '\n", " 'manifest, making it a probable diagnosis '\n", " 'in this scenario.',\n", " 'Question': 'A 45-year-old man with a history of '\n", " 'alcohol use, who has been abstinent '\n", " 'for the past 10 years, presents with '\n", " 'sudden onset dysarthria, shuffling '\n", " 'gait, and intention tremors. Given '\n", " 'this clinical presentation and '\n", " 'history, what is the most likely '\n", " 'diagnosis?'},\n", " 'score': 0.697534442,\n", " 'values': []},\n", " {'id': '2',\n", " 'metadata': {'Answer': 'Based on the clinical findings '\n", " 'presented—wide-based gait, blank facial '\n", " 'expression, hallucinations, memory '\n", " 'issues, resting tremor that resolves '\n", " 'with movement, and bradykinesia—it is '\n", " 'likely that the 45-year-old man is '\n", " 'experiencing a condition related to '\n", " \"Parkinsonism, possibly Parkinson's \"\n", " 'disease or dementia with Lewy bodies. '\n", " 'Both of these conditions are associated '\n", " 'with the presence of Lewy bodies in the '\n", " 'brain. Lewy bodies are abnormal '\n", " 'aggregates of protein, primarily '\n", " 'alpha-synuclein, which can cause both '\n", " 'the motor and cognitive symptoms '\n", " 'observed in this patient. Therefore, in '\n", " 'the histological specimen of his brain, '\n", " 'you would most likely observe the '\n", " 'presence of Lewy bodies.',\n", " 'Question': 'A 45-year-old man presents with '\n", " 'symptoms including a wide-based gait, '\n", " 'a blank facial expression, '\n", " 'hallucinations, memory issues, a '\n", " 'resting tremor that resolves with '\n", " 'movement, and bradykinesia. Based on '\n", " 'these clinical findings, what is most '\n", " 'likely to be observed in the '\n", " 'histological specimen of his brain?'},\n", " 'score': 0.55345,\n", " 'values': []},\n", " {'id': '19',\n", " 'metadata': {'Answer': 'The symptoms and MRI findings of this '\n", " '73-year-old man suggest the most likely '\n", " 'underlying pathological process is the '\n", " 'buildup of amyloid-beta plaques and tau '\n", " 'protein tangles, which are '\n", " \"characteristic of Alzheimer's disease. \"\n", " 'These changes often begin in brain '\n", " 'regions involved in memory, such as the '\n", " 'hippocampus and temporal lobes, leading '\n", " 'to the gradual memory decline, '\n", " 'disorientation, and personality changes '\n", " 'observed in the patient.',\n", " 'Question': 'A 73-year-old man is evaluated for '\n", " 'increasing forgetfulness, getting lost '\n", " 'while walking, irritability, and '\n", " 'difficulty recalling recent events '\n", " 'while retaining detailed memories from '\n", " 'over 20 years ago. On examination, he '\n", " 'is oriented to person and place but '\n", " 'disoriented to time, and an MRI of the '\n", " 'brain reveals significant changes. '\n", " 'Considering these symptoms and the '\n", " 'imaging findings, what is the most '\n", " 'likely underlying pathological process '\n", " \"contributing to the patient's \"\n", " 'condition?'},\n", " 'score': 0.526201367,\n", " 'values': []},\n", " {'id': '38',\n", " 'metadata': {'Answer': \"The symptoms and findings you've \"\n", " 'described in this 2-day-old newborn '\n", " 'point towards congenital Toxoplasmosis. '\n", " 'The combination of neurological symptoms '\n", " '(such as convulsions and ventricular '\n", " 'dilatation with hyperechoic foci), the '\n", " 'presence of posterior uveitis, and the '\n", " 'skin manifestations like purpura, all '\n", " 'fit into the classic presentation of a '\n", " 'TORCH infection. Toxoplasmosis, '\n", " 'specifically, is known to cause '\n", " 'widespread calcifications in the brain, '\n", " 'not limited to the periventricular '\n", " 'areas, which matches the ultrasound '\n", " 'findings. Additionally, while hearing '\n", " 'loss is more traditionally associated '\n", " 'with CMV, it can also occur in '\n", " 'Toxoplasmosis. Thus, the most likely '\n", " 'diagnosis given this clinical picture is '\n", " 'congenital Toxoplasmosis.',\n", " 'Question': 'A 2-day-old male newborn delivered at '\n", " '36 weeks presents with generalized '\n", " 'convulsions, lethargy, feeding '\n", " 'difficulties, icterus, purpura, '\n", " 'posterior uveitis, and failed auditory '\n", " 'screening. Cranial ultrasonography '\n", " 'shows ventricular dilatation and '\n", " 'hyperechoic foci in multiple brain '\n", " 'areas. Considering these clinical '\n", " 'signs and history, what is the most '\n", " 'likely diagnosis?'},\n", " 'score': 0.422916651,\n", " 'values': []},\n", " {'id': '31',\n", " 'metadata': {'Answer': 'Based on the symptoms described, the '\n", " 'cranial nerve most likely involved in '\n", " 'the double vision experienced by this '\n", " 'patient while walking upstairs is the '\n", " 'trochlear nerve, or cranial nerve IV. '\n", " 'This nerve controls the superior oblique '\n", " 'muscle, which plays a role in '\n", " 'stabilizing the eye during certain '\n", " 'movements, including the coordination '\n", " 'required when looking upwards while '\n", " \"walking upstairs. Given the patient's \"\n", " 'history of diabetes, cranial '\n", " 'neuropathies can occur, and CN IV '\n", " 'involvement can lead to vertical '\n", " 'diplopia that becomes noticeable during '\n", " 'specific activities like walking up '\n", " 'stairs. Therefore, the trochlear nerve '\n", " 'is a likely candidate for the '\n", " 'involvement in these symptoms.',\n", " 'Question': 'A 45-year-old male patient experiences '\n", " 'double vision specifically when '\n", " 'walking upstairs. Considering his '\n", " 'well-controlled history of Type-II '\n", " 'diabetes, which cranial nerve is most '\n", " 'likely involved in his symptoms?'},\n", " 'score': 0.420719624,\n", " 'values': []}],\n", " 'namespace': '',\n", " 'usage': {'read_units': 6}}" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Example usage with a different query from the train/test set\n", "query = (\n", " \"A 45-year-old man with a history of alcohol use presents with symptoms including confusion, ataxia, and ophthalmoplegia. \"\n", " \"What is the most likely diagnosis and the recommended treatment?\"\n", ")\n", "query_pinecone_index(client, index, MODEL, query)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Generate a Response Using the Retrieved Context\n", "\n", "Select the best matching result from your query results and use the OpenAI Responses API to generate a final answer by combining the retrieved context with the original question." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Final Answer:\n", "The presentation of confusion, ataxia, and ophthalmoplegia in a 45-year-old man with a history of alcohol use is suggestive of Wernicke's encephalopathy. This condition is caused by thiamine (vitamin B1) deficiency, often associated with chronic alcohol use.\n", "\n", "The recommended treatment is the immediate administration of thiamine, typically given intravenously or intramuscularly, to prevent progression to more severe neurological damage or Korsakoff syndrome.\n" ] } ], "source": [ "# Retrieve and concatenate top 3 match contexts.\n", "matches = index.query(\n", " vector=[client.embeddings.create(input=query, model=MODEL).data[0].embedding],\n", " top_k=3,\n", " include_metadata=True\n", ")['matches']\n", "\n", "context = \"\\n\\n\".join(\n", " f\"Question: {m['metadata'].get('Question', '')}\\nAnswer: {m['metadata'].get('Answer', '')}\"\n", " for m in matches\n", ")\n", "# Use the context to generate a final answer.\n", "response = client.responses.create(\n", " model=\"gpt-4o\",\n", " input=f\"Provide the answer based on the context: {context} and the question: {query} as per the internal knowledge base\",\n", ")\n", "print(\"\\nFinal Answer:\")\n", "print(response.output_text)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Orchestrate Multi-Tool Calls\n", "\n", "Now, we'll define the built-in function available through the Responses API, including the ability to invoke the external Vector Store - Pinecone as an example.\n", "\n", "*Web Search Preview Tool*: Enables the model to perform live web searches and preview the results. This is ideal for retrieving real-time or up-to-date information from the internet.\n", "\n", "*Pinecone Search Tool*: Allows the model to query a vector database using semantic search. This is especially useful for retrieving relevant documents—such as medical literature or other domain-specific content—that have been stored in a vectorized format." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# Tools definition: The list of tools includes:\n", "# - A web search preview tool.\n", "# - A Pinecone search tool for retrieving medical documents.\n", "\n", "# Define available tools.\n", "tools = [ \n", " {\"type\": \"web_search_preview\",\n", " \"user_location\": {\n", " \"type\": \"approximate\",\n", " \"country\": \"US\",\n", " \"region\": \"California\",\n", " \"city\": \"SF\"\n", " },\n", " \"search_context_size\": \"medium\"},\n", " {\n", " \"type\": \"function\",\n", " \"name\": \"PineconeSearchDocuments\",\n", " \"description\": \"Search for relevant documents based on the medical question asked by the user that is stored within the vector database using a semantic query.\",\n", " \"parameters\": {\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"query\": {\n", " \"type\": \"string\",\n", " \"description\": \"The natural language query to search the vector database.\"\n", " },\n", " \"top_k\": {\n", " \"type\": \"integer\",\n", " \"description\": \"Number of top results to return.\",\n", " \"default\": 3\n", " }\n", " },\n", " \"required\": [\"query\"],\n", " \"additionalProperties\": False\n", " }\n", " }\n", "]\n" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "# Example queries that the model should route appropriately.\n", "queries = [\n", " {\"query\": \"Who won the cricket world cup in 1983?\"},\n", " {\"query\": \"What is the most common cause of death in the United States according to the internet?\"},\n", " {\"query\": (\"A 7-year-old boy with sickle cell disease is experiencing knee and hip pain, \"\n", " \"has been admitted for pain crises in the past, and now walks with a limp. \"\n", " \"His exam shows a normal, cool hip with decreased range of motion and pain with ambulation. \"\n", " \"What is the most appropriate next step in management according to the internal knowledge base?\")}\n", "]" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "🌟--- Processing Query ---🌟\n", "🔍 **User Query:** Who won the cricket world cup in 1983?\n", "\n", "✨ **Initial Response Output:**\n", "[ResponseOutputMessage(id='msg_67e6e7a9f7508191a9d18c3ff25310290811a0720cf47168', content=[ResponseOutputText(annotations=[], text='India won the Cricket World Cup in 1983.', type='output_text')], role='assistant', status='completed', type='message')]\n", "💡 **Final Answer:**\n", "India won the Cricket World Cup in 1983.\n", "\n", "🌟--- Processing Query ---🌟\n", "🔍 **User Query:** What is the most common cause of death in the United States according to the internet?\n", "\n", "✨ **Initial Response Output:**\n", "[ResponseFunctionWebSearch(id='ws_67e6e7aad0248191ab974d4b09b460c90537f90023d2dd32', status='completed', type='web_search_call'), ResponseOutputMessage(id='msg_67e6e7ace08081918f06b5cac32e8c0e0537f90023d2dd32', content=[ResponseOutputText(annotations=[AnnotationURLCitation(end_index=363, start_index=225, title='10 Leading Causes of Death in the U.S.', type='url_citation', url='https://www.usnews.com/news/healthiest-communities/slideshows/top-10-causes-of-death-in-america?slide=11&utm_source=openai'), AnnotationURLCitation(end_index=753, start_index=625, title='Top causes of death in the US — see the CDC’s latest list - Rifnote', type='url_citation', url='https://rifnote.com/health/2024/08/11/top-causes-of-death-in-the-us-see-the-cdcs-latest-list/?utm_source=openai'), AnnotationURLCitation(end_index=1014, start_index=886, title='Top causes of death in the US — see the CDC’s latest list - Rifnote', type='url_citation', url='https://rifnote.com/health/2024/08/11/top-causes-of-death-in-the-us-see-the-cdcs-latest-list/?utm_source=openai'), AnnotationURLCitation(end_index=1216, start_index=1061, title='US deaths are down and life expectancy is up, but improvements are slowing', type='url_citation', url='https://apnews.com/article/be061f9f14c883178eea6dddc9550e60?utm_source=openai'), AnnotationURLCitation(end_index=1394, start_index=1219, title='A Mysterious Health Wave Is Breaking Out Across the U.S.', type='url_citation', url='https://www.theatlantic.com/ideas/archive/2024/12/violence-obesity-overdoses-health-covid/681079/?utm_source=openai')], text='According to the Centers for Disease Control and Prevention (CDC), heart disease was the leading cause of death in the United States in 2023, accounting for 680,980 deaths, which is approximately 22% of all deaths that year. ([usnews.com](https://www.usnews.com/news/healthiest-communities/slideshows/top-10-causes-of-death-in-america?slide=11&utm_source=openai))\\n\\nThe top 10 causes of death in the U.S. for 2023 were:\\n\\n1. Heart disease\\n2. Cancer\\n3. Unintentional injury\\n4. Stroke\\n5. Chronic lower respiratory diseases\\n6. Alzheimer’s disease\\n7. Diabetes\\n8. Kidney disease\\n9. Chronic liver disease and cirrhosis\\n10. COVID-19\\n\\n([rifnote.com](https://rifnote.com/health/2024/08/11/top-causes-of-death-in-the-us-see-the-cdcs-latest-list/?utm_source=openai))\\n\\nNotably, COVID-19, which was the fourth leading cause of death in 2022, dropped to the tenth position in 2023, with 76,446 deaths. ([rifnote.com](https://rifnote.com/health/2024/08/11/top-causes-of-death-in-the-us-see-the-cdcs-latest-list/?utm_source=openai))\\n\\n\\n## Recent Trends in U.S. Mortality Rates:\\n- [US deaths are down and life expectancy is up, but improvements are slowing](https://apnews.com/article/be061f9f14c883178eea6dddc9550e60?utm_source=openai)\\n- [A Mysterious Health Wave Is Breaking Out Across the U.S.](https://www.theatlantic.com/ideas/archive/2024/12/violence-obesity-overdoses-health-covid/681079/?utm_source=openai) ', type='output_text')], role='assistant', status='completed', type='message')]\n", "💡 **Final Answer:**\n", "According to the Centers for Disease Control and Prevention (CDC), heart disease was the leading cause of death in the United States in 2023, accounting for 680,980 deaths, which is approximately 22% of all deaths that year. ([usnews.com](https://www.usnews.com/news/healthiest-communities/slideshows/top-10-causes-of-death-in-america?slide=11&utm_source=openai))\n", "\n", "The top 10 causes of death in the U.S. for 2023 were:\n", "\n", "1. Heart disease\n", "2. Cancer\n", "3. Unintentional injury\n", "4. Stroke\n", "5. Chronic lower respiratory diseases\n", "6. Alzheimer’s disease\n", "7. Diabetes\n", "8. Kidney disease\n", "9. Chronic liver disease and cirrhosis\n", "10. COVID-19\n", "\n", "([rifnote.com](https://rifnote.com/health/2024/08/11/top-causes-of-death-in-the-us-see-the-cdcs-latest-list/?utm_source=openai))\n", "\n", "Notably, COVID-19, which was the fourth leading cause of death in 2022, dropped to the tenth position in 2023, with 76,446 deaths. ([rifnote.com](https://rifnote.com/health/2024/08/11/top-causes-of-death-in-the-us-see-the-cdcs-latest-list/?utm_source=openai))\n", "\n", "\n", "## Recent Trends in U.S. Mortality Rates:\n", "- [US deaths are down and life expectancy is up, but improvements are slowing](https://apnews.com/article/be061f9f14c883178eea6dddc9550e60?utm_source=openai)\n", "- [A Mysterious Health Wave Is Breaking Out Across the U.S.](https://www.theatlantic.com/ideas/archive/2024/12/violence-obesity-overdoses-health-covid/681079/?utm_source=openai) \n", "\n", "🌟--- Processing Query ---🌟\n", "🔍 **User Query:** A 7-year-old boy with sickle cell disease is experiencing knee and hip pain, has been admitted for pain crises in the past, and now walks with a limp. His exam shows a normal, cool hip with decreased range of motion and pain with ambulation. What is the most appropriate next step in management according to the internal knowledge base?\n", "\n", "✨ **Initial Response Output:**\n", "[ResponseFunctionToolCall(arguments='{\"query\":\"7-year-old sickle cell disease knee hip pain limp normal cool hip decreased range of motion\"}', call_id='call_ds0ETZbYtX71U2bQZXTBEWxN', name='PineconeSearchDocuments', type='function_call', id='fc_67e6e7b03ee48191bb400c13c359c35e0aeeec60d0806312', status='completed')]\n", "\n", "🔧 **Model triggered a tool call:** PineconeSearchDocuments\n", "🔍 **Invoking PineconeSearchDocuments tool...**\n", "Query Results:\n", "0.87: A 7-year-old boy with sickle cell disease is experiencing knee and hip pain, has been admitted for pain crises in the past, and now walks with a limp. His physical exam shows a normal and cool hip to the touch, with decreased range of motion at the hip and pain with ambulation. Given these findings, what is the most appropriate next step in the management of this patient's hip pain? - In managing the hip pain of a 7-year-old boy with sickle cell disease, who presents with knee and hip pain, a limp, and decreased range of motion in the hip, the most appropriate next step is to obtain an X-ray of the hip. This will help evaluate the possibility of avascular necrosis (AVN) or other structural abnormalities. X-rays are typically the first-line imaging technique in such cases due to their accessibility and ability to reveal gross pathological changes. If the X-ray does not provide conclusive information and clinical suspicion of AVN remains high, an MRI may subsequently be considered for a more detailed assessment.\n", "0.55: What is the most likely diagnosis for a 2-year-old 70 kg child who presents with limitation of abduction and internal rotation, tenderness in Scarpa's triangle, and abduction of the limb upon flexing the hip? - Based on the described symptoms and the unusual weight for a 2-year-old child, the most likely diagnosis is Slipped Capital Femoral Epiphysis (SCFE). Even though SCFE typically occurs in older children, mainly adolescents, the combination of excessive weight, limited hip abduction and internal rotation, tenderness in the hip area, and the characteristic limb movement (abduction upon hip flexion) strongly points towards SCFE as the most plausible diagnosis in this scenario.\n", "0.48: A 5-year-old boy has recurrent abdominal pain primarily occurring during school hours, with no significant findings on physical examination and normal stool characteristics. His symptoms resolve at home, and his laboratory tests and abdominal exam are unremarkable. Considering the psychological factors involved, what is the most appropriate next step in managing his condition? - Given the symptoms and the context you've provided, it seems quite possible that the boy's recurrent abdominal pain is linked to psychological stressors related to school. Since all medical tests and examinations have returned normal results, this suggests that the pain might be stress-induced, possibly due to anxiety or stress at school. \n", "\n", "The most appropriate next step is to focus on addressing any potential psychological or emotional factors. Consulting a psychologist or school counselor would be beneficial. They can work with the boy to explore any underlying emotional issues or anxieties about school. Through conversation, play, or other therapeutic techniques suitable for his age, they can help identify and manage any stressors he might be facing. This approach could not only help alleviate his abdominal pain but also improve his overall well-being by addressing the source of his anxiety.\n", "0.44: In a patient who, five days post-open colectomy for colon cancer, develops severe pain and swelling of the left calf along with necrotic lesions, a fever, and thrombocytopenia while on unfractionated heparin, what is the most appropriate next step in management? - In this clinical scenario, the presentation of severe pain and swelling in the calf, necrotic skin lesions, fever, and thrombocytopenia in a patient receiving unfractionated heparin strongly suggests heparin-induced thrombocytopenia (HIT). HIT is a prothrombotic disorder caused by antibodies against heparin-platelet factor 4 complexes, leading to platelet activation, thrombocytopenia, and an increased risk of thrombosis.\n", "\n", "The most appropriate next step in management is to immediately discontinue the unfractionated heparin to prevent further complications related to thrombosis. Simultaneously, it's crucial to initiate an alternative anticoagulant that does not cross-react with HIT antibodies to manage the thrombotic risk. Argatroban or fondaparinux are commonly used anticoagulants in this context as they are safe and effective for patients with HIT. Direct-acting oral anticoagulants (DOACs) are also potential options, but argatroban is often preferred initially due to its intravenous route and ability to be titrated easily in acute care settings. This dual approach addresses both the cause and the risk effectively.\n", "0.44: In a patient with sickle cell anaemia presenting with multiple non-suppurative osteomyelitic dactylitis, what is the most likely causative organism? - In a patient with sickle cell anemia presenting with multiple non-suppurative osteomyelitic dactylitis, the most likely causative organism is Salmonella species. In individuals with sickle cell disease, Salmonella is particularly notorious for causing osteomyelitis. The relationship between sickle cell anemia and Salmonella infections, especially in the bone, is well-documented, and their presentations can often be less typical and less suppurative than those caused by other common bacteria like Staphylococcus aureus.\n", "✅ **PineconeSearchDocuments tool invoked successfully.**\n", "\n", "💡 **Final Answer:**\n", "The most appropriate next step in the management of this 7-year-old boy with sickle cell disease and hip pain is to obtain an X-ray of the hip. This will help evaluate for potential avascular necrosis or other structural issues. If the X-ray is inconclusive and there is still a high suspicion of avascular necrosis, further imaging with an MRI may be considered.\n" ] } ], "source": [ "# Process each query dynamically.\n", "for item in queries:\n", " input_messages = [{\"role\": \"user\", \"content\": item[\"query\"]}]\n", " print(\"\\n🌟--- Processing Query ---🌟\")\n", " print(f\"🔍 **User Query:** {item['query']}\")\n", " \n", " # Call the Responses API with tools enabled and allow parallel tool calls.\n", " response = client.responses.create(\n", " model=\"gpt-4o\",\n", " input=[\n", " {\"role\": \"system\", \"content\": \"When prompted with a question, select the right tool to use based on the question.\"\n", " },\n", " {\"role\": \"user\", \"content\": item[\"query\"]}\n", " ],\n", " tools=tools,\n", " parallel_tool_calls=True\n", " )\n", " \n", " print(\"\\n✨ **Initial Response Output:**\")\n", " print(response.output)\n", " \n", " # Determine if a tool call is needed and process accordingly.\n", " if response.output:\n", " tool_call = response.output[0]\n", " if tool_call.type in [\"web_search_preview\", \"function_call\"]:\n", " tool_name = tool_call.name if tool_call.type == \"function_call\" else \"web_search_preview\"\n", " print(f\"\\n🔧 **Model triggered a tool call:** {tool_name}\")\n", " \n", " if tool_name == \"PineconeSearchDocuments\":\n", " print(\"🔍 **Invoking PineconeSearchDocuments tool...**\")\n", " res = query_pinecone_index(client, index, MODEL, item[\"query\"])\n", " if res[\"matches\"]:\n", " best_match = res[\"matches\"][0][\"metadata\"]\n", " result = f\"**Question:** {best_match.get('Question', 'N/A')}\\n**Answer:** {best_match.get('Answer', 'N/A')}\"\n", " else:\n", " result = \"**No matching documents found in the index.**\"\n", " print(\"✅ **PineconeSearchDocuments tool invoked successfully.**\")\n", " else:\n", " print(\"🔍 **Invoking simulated web search tool...**\")\n", " result = \"**Simulated web search result.**\"\n", " print(\"✅ **Simulated web search tool invoked successfully.**\")\n", " \n", " # Append the tool call and its output back into the conversation.\n", " input_messages.append(tool_call)\n", " input_messages.append({\n", " \"type\": \"function_call_output\",\n", " \"call_id\": tool_call.call_id,\n", " \"output\": str(result)\n", " })\n", " \n", " # Get the final answer incorporating the tool's result.\n", " final_response = client.responses.create(\n", " model=\"gpt-4o\",\n", " input=input_messages,\n", " tools=tools,\n", " parallel_tool_calls=True\n", " )\n", " print(\"\\n💡 **Final Answer:**\")\n", " print(final_response.output_text)\n", " else:\n", " # If no tool call is triggered, print the response directly.\n", " print(\"💡 **Final Answer:**\")\n", " print(response.output_text)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As shown above, depending on the query, appropriate tool is invoked in order to determine the optimal response.\n", "\n", "For instance, looking at the third example, when the model triggers the tool named \"PineconeSearchDocuments\", the code calls `query_pinecone_index` with the current query and then extracts the best match (or an appropriate context) as the result. For non health related inqueries or queries where explicit internet search is asked, the code calls the web_search_call function and for other queries, it may choose to not call any tool and rather provide a response based on the question under consideration.\n", "\n", "Finally, the tool call and its output are appended to the conversation, and the final answer is generated by the Responses API." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Multi-tool orchestration flow\n", "\n", "Now let us try to modify the input query and the system instructions to the responses API in order to follow a tool calling sequence and generate the output. " ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "🌟--- Processing Query ---🌟\n", "🔍 **User Query:** What is the most common cause of death in the United States\n", "\n", "🔧 **Calling Responses API with Tools Enabled**\n", "\n", "🕵️‍♂️ **Step 1: Web Search Call**\n", " - Initiating web search to gather initial information.\n", "\n", "📚 **Step 2: Pinecone Search Call**\n", " - Querying Pinecone to find relevant examples from the internal knowledge base.\n", "input_messages [{'role': 'user', 'content': 'What is the most common cause of death in the United States'}]\n", "\n", "✨ **Initial Response Output:**\n", "[ResponseFunctionWebSearch(id='ws_67e6e83241ac81918f93ffc96491ec390fdddafaeefcefc1', status='completed', type='web_search_call'), ResponseOutputMessage(id='msg_67e6e833a2cc8191a9df22f324a876b00fdddafaeefcefc1', content=[ResponseOutputText(annotations=[AnnotationURLCitation(end_index=698, start_index=613, title='Products - Data Briefs - Number 521 - December 2024', type='url_citation', url='https://www.cdc.gov/nchs/products/databriefs/db521.htm?utm_source=openai'), AnnotationURLCitation(end_index=984, start_index=891, title='US deaths are down and life expectancy is up, but improvements are slowing', type='url_citation', url='https://apnews.com/article/be061f9f14c883178eea6dddc9550e60?utm_source=openai'), AnnotationURLCitation(end_index=1186, start_index=1031, title='US deaths are down and life expectancy is up, but improvements are slowing', type='url_citation', url='https://apnews.com/article/be061f9f14c883178eea6dddc9550e60?utm_source=openai')], text=\"As of 2023, the leading causes of death in the United States are:\\n\\n1. **Heart Disease**: 680,981 deaths\\n2. **Cancer**: 613,352 deaths\\n3. **Unintentional Injuries**: 222,698 deaths\\n4. **Stroke**: 162,639 deaths\\n5. **Chronic Lower Respiratory Diseases**: 145,357 deaths\\n6. **Alzheimer's Disease**: 114,034 deaths\\n7. **Diabetes**: 95,190 deaths\\n8. **Kidney Disease**: 55,253 deaths\\n9. **Chronic Liver Disease and Cirrhosis**: 52,222 deaths\\n10. **COVID-19**: 49,932 deaths\\n\\nNotably, COVID-19 has dropped from the fourth leading cause in 2022 to the tenth in 2023, reflecting a significant decrease in related deaths. ([cdc.gov](https://www.cdc.gov/nchs/products/databriefs/db521.htm?utm_source=openai))\\n\\nOverall, the U.S. experienced a decline in total deaths and a modest increase in life expectancy in 2023, attributed to reductions in deaths from COVID-19, heart disease, and drug overdoses. ([apnews.com](https://apnews.com/article/be061f9f14c883178eea6dddc9550e60?utm_source=openai))\\n\\n\\n## Recent Trends in U.S. Mortality Rates:\\n- [US deaths are down and life expectancy is up, but improvements are slowing](https://apnews.com/article/be061f9f14c883178eea6dddc9550e60?utm_source=openai) \", type='output_text')], role='assistant', status='completed', type='message'), ResponseFunctionToolCall(arguments='{\"query\":\"most common cause of death in the United States\",\"top_k\":3}', call_id='call_6YWhEw3QSI7wGZBlNs5Pz4zI', name='PineconeSearchDocuments', type='function_call', id='fc_67e6e8364e4c819198501fba5d3f155b0fdddafaeefcefc1', status='completed')]\n" ] } ], "source": [ "# Process one query as an example to understand the tool calls and function calls as part of the response output\n", "item = \"What is the most common cause of death in the United States\"\n", "\n", "# Initialize input messages with the user's query.\n", "input_messages = [{\"role\": \"user\", \"content\": item}]\n", "print(\"\\n🌟--- Processing Query ---🌟\")\n", "print(f\"🔍 **User Query:** {item}\")\n", " \n", " # Call the Responses API with tools enabled and allow parallel tool calls.\n", "print(\"\\n🔧 **Calling Responses API with Tools Enabled**\")\n", "print(\"\\n🕵️‍♂️ **Step 1: Web Search Call**\")\n", "print(\" - Initiating web search to gather initial information.\")\n", "print(\"\\n📚 **Step 2: Pinecone Search Call**\")\n", "print(\" - Querying Pinecone to find relevant examples from the internal knowledge base.\")\n", " \n", "response = client.responses.create(\n", " model=\"gpt-4o\",\n", " input=[\n", " {\"role\": \"system\", \"content\": \"Every time it's prompted with a question, first call the web search tool for results, then call `PineconeSearchDocuments` to find real examples in the internal knowledge base.\"},\n", " {\"role\": \"user\", \"content\": item}\n", " ],\n", " tools=tools,\n", " parallel_tool_calls=True\n", " )\n", " \n", "# Print the initial response output.\n", "print(\"input_messages\", input_messages)\n", "\n", "print(\"\\n✨ **Initial Response Output:**\")\n", "print(response.output)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TypeCall IDOutputName
0web_search_callws_67e6e83241ac81918f93ffc96491ec390fdddafaeef...N/AN/A
1messagemsg_67e6e833a2cc8191a9df22f324a876b00fdddafaee...N/AN/A
2function_callcall_6YWhEw3QSI7wGZBlNs5Pz4zIN/APineconeSearchDocuments
\n", "
" ], "text/plain": [ " Type Call ID Output \\\n", "0 web_search_call ws_67e6e83241ac81918f93ffc96491ec390fdddafaeef... N/A \n", "1 message msg_67e6e833a2cc8191a9df22f324a876b00fdddafaee... N/A \n", "2 function_call call_6YWhEw3QSI7wGZBlNs5Pz4zI N/A \n", "\n", " Name \n", "0 N/A \n", "1 N/A \n", "2 PineconeSearchDocuments " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Understand the tool calls and function calls as part of the response output\n", "\n", "import pandas as pd\n", "\n", "# Create a list to store the tool call and function call details\n", "tool_calls = []\n", "\n", "# Iterate through the response output and collect the details\n", "for i in response.output:\n", " tool_calls.append({\n", " \"Type\": i.type,\n", " \"Call ID\": i.call_id if hasattr(i, 'call_id') else i.id if hasattr(i, 'id') else \"N/A\",\n", " \"Output\": str(i.output) if hasattr(i, 'output') else \"N/A\",\n", " \"Name\": i.name if hasattr(i, 'name') else \"N/A\"\n", " })\n", "\n", "# Convert the list to a DataFrame for tabular display\n", "df_tool_calls = pd.DataFrame(tool_calls)\n", "\n", "# Display the DataFrame\n", "df_tool_calls" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ResponseFunctionWebSearch(id='ws_67e6e83241ac81918f93ffc96491ec390fdddafaeefcefc1', status='completed', type='web_search_call')\n", "ws_67e6e83241ac81918f93ffc96491ec390fdddafaeefcefc1\n", "ResponseFunctionToolCall(arguments='{\"query\":\"most common cause of death in the United States\",\"top_k\":3}', call_id='call_6YWhEw3QSI7wGZBlNs5Pz4zI', name='PineconeSearchDocuments', type='function_call', id='fc_67e6e8364e4c819198501fba5d3f155b0fdddafaeefcefc1', status='completed')\n", "call_6YWhEw3QSI7wGZBlNs5Pz4zI\n" ] } ], "source": [ "tool_call_1 = response.output[0]\n", "print(tool_call_1)\n", "print(tool_call_1.id)\n", "\n", "tool_call_2 = response.output[2]\n", "print(tool_call_2)\n", "print(tool_call_2.call_id)" ] }, { "cell_type": "code", "execution_count": 166, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[{'role': 'user', 'content': 'What is the most common cause of death in the United States'}, ResponseFunctionToolCall(arguments='{\"query\":\"most common cause of death in the United States\"}', call_id='call_8Vzsn4RwMOgXyX98UpZY8hls', name='PineconeSearchDocuments', type='function_call', id='fc_67e348f36f7c81919d0aeef1855df3f20d0bd7f2a5744b88', status='completed')]\n", "[{'role': 'user', 'content': 'What is the most common cause of death in the United States'}, ResponseFunctionToolCall(arguments='{\"query\":\"most common cause of death in the United States\"}', call_id='call_8Vzsn4RwMOgXyX98UpZY8hls', name='PineconeSearchDocuments', type='function_call', id='fc_67e348f36f7c81919d0aeef1855df3f20d0bd7f2a5744b88', status='completed'), {'type': 'function_call_output', 'call_id': 'call_8Vzsn4RwMOgXyX98UpZY8hls', 'output': \"**Question:** A 7-year-old boy with sickle cell disease is experiencing knee and hip pain, has been admitted for pain crises in the past, and now walks with a limp. His physical exam shows a normal and cool hip to the touch, with decreased range of motion at the hip and pain with ambulation. Given these findings, what is the most appropriate next step in the management of this patient's hip pain?\\n**Answer:** In managing the hip pain of a 7-year-old boy with sickle cell disease, who presents with knee and hip pain, a limp, and decreased range of motion in the hip, the most appropriate next step is to obtain an X-ray of the hip. This will help evaluate the possibility of avascular necrosis (AVN) or other structural abnormalities. X-rays are typically the first-line imaging technique in such cases due to their accessibility and ability to reveal gross pathological changes. If the X-ray does not provide conclusive information and clinical suspicion of AVN remains high, an MRI may subsequently be considered for a more detailed assessment.\"}]\n" ] } ], "source": [ "# append the tool call and its output back into the conversation.\n", "input_messages.append(response.output[2])\n", "input_messages.append({\n", " \"type\": \"function_call_output\",\n", " \"call_id\": tool_call_2.call_id,\n", " \"output\": str(result)\n", "})\n", "print(input_messages)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "🔧 **Calling Responses API for Final Answer**\n", "Response(id='resp_67e6e886ac7081918b07224fb1ed38ab05c4a598f9697c7c', created_at=1743186054.0, error=None, incomplete_details=None, instructions=None, metadata={}, model='gpt-4o-2024-08-06', object='response', output=[ResponseOutputMessage(id='msg_67e6e8872ddc81918e92c9e4508abbe005c4a598f9697c7c', content=[ResponseOutputText(annotations=[], text='The most common cause of death in the United States is heart disease.', type='output_text')], role='assistant', status='completed', type='message')], parallel_tool_calls=True, temperature=1.0, tool_choice='auto', tools=[], top_p=1.0, max_output_tokens=None, previous_response_id=None, reasoning=Reasoning(effort=None, generate_summary=None), status='completed', text=ResponseTextConfig(format=ResponseFormatText(type='text')), truncation='disabled', usage=ResponseUsage(input_tokens=37, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=15, output_tokens_details=OutputTokensDetails(reasoning_tokens=0), total_tokens=52), user=None, store=False)\n" ] } ], "source": [ "\n", "# Get the final answer incorporating the tool's result.\n", "print(\"\\n🔧 **Calling Responses API for Final Answer**\")\n", "\n", "response_2 = client.responses.create(\n", " model=\"gpt-4o\",\n", " input=input_messages,\n", ")\n", "print(response_2)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The most common cause of death in the United States is heart disease.\n" ] } ], "source": [ "# print the final answer\n", "print(response_2.output_text)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Here, we have seen how to utilize OpenAI's Responses API to implement a Retrieval-Augmented Generation (RAG) approach with multi-tool calling capabilities. It showcases an example where the model selects the appropriate tool based on the input query: general questions may be handled by built-in tools such as web-search, while specific medical inquiries related to internal knowledge are addressed by retrieving context from a vector database (such as Pinecone) via function calls. Additonally, we have showcased how multiple tool calls can be sequentially combined to generate a final response based on our instructions provided to responses API. Happy coding! " ] } ], "metadata": { "kernelspec": { "display_name": "env", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" } }, "nbformat": 4, "nbformat_minor": 2 }