2024-04-10 08:53:14 -07:00
{
"cells": [
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "7b3d17451440ca82",
"metadata": {
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"source": [
2025-01-09 13:13:27 +00:00
"# How to use GPT-4o Vision with Function Calling \n",
2024-04-10 08:53:14 -07:00
"\n",
2025-01-09 13:13:27 +00:00
"The GPT-4o, available as gpt-4o-2024-11-20 as of Novemeber 2024, now enables function calling with vision capabilities, better reasoning and a knowledge cutoff date of Oct 2023. Using images with function calling will unlock multimodal use cases and the ability to use reasoning, allowing you to go beyond OCR and image descriptions.\n",
2024-04-10 08:53:14 -07:00
"\n",
2025-01-09 13:13:27 +00:00
"We will go through two examples to demonstrate the use of function calling with GPT-4o with Vision:\n",
2024-04-10 08:53:14 -07:00
"\n",
"1. Simulating a customer service assistant for delivery exception support\n",
"2. Analyzing an organizational chart to extract employee information"
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "feffa794bc28be22",
2024-04-10 08:53:14 -07:00
"metadata": {
"collapsed": false
},
2025-01-09 13:13:27 +00:00
"source": [
"### Installation and Setup"
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "code",
2025-01-09 13:13:27 +00:00
"execution_count": 1,
"id": "6ce24dfd4e4bbc47",
"metadata": {
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"outputs": [],
"source": [
"!pip install pymupdf --quiet\n",
"!pip install openai --quiet\n",
"!pip install matplotlib --quiet\n",
"# instructor makes it easy to work with function calling\n",
"!pip install instructor --quiet"
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "code",
2025-01-09 13:13:27 +00:00
"execution_count": 2,
2024-04-10 08:53:14 -07:00
"id": "initial_id",
"metadata": {
"ExecuteTime": {
"end_time": "2024-04-10T03:50:37.564145Z",
"start_time": "2024-04-10T03:50:37.560040Z"
2025-01-09 13:13:27 +00:00
},
"collapsed": true
2024-04-10 08:53:14 -07:00
},
2025-01-09 13:13:27 +00:00
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
}
],
2024-04-10 08:53:14 -07:00
"source": [
"import base64\n",
"import os\n",
"from enum import Enum\n",
"from io import BytesIO\n",
"from typing import Iterable\n",
"from typing import List\n",
"from typing import Literal, Optional\n",
"\n",
"import fitz\n",
"# Instructor is powered by Pydantic, which is powered by type hints. Schema validation, prompting is controlled by type annotations\n",
"import instructor\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"from IPython.display import display\n",
"from PIL import Image\n",
"from openai import OpenAI\n",
"from pydantic import BaseModel, Field"
]
},
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "14ab8f931cc759d",
"metadata": {
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"source": [
"## 1. Simulating a customer service assistant for delivery exception support\n",
"We will simulate a customer service assistant for a delivery service that is equipped to analyze images of packages. The assistant will perform the following actions based on the image analysis:\n",
"- If a package appears damaged in the image, automatically process a refund according to policy.\n",
"- If the package looks wet, initiate a replacement.\n",
"- If the package appears normal and not damaged, escalate to an agent."
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "ca35604f27f93e77",
2024-04-10 08:53:14 -07:00
"metadata": {
"collapsed": false
},
2025-01-09 13:13:27 +00:00
"source": [
"Let's look at the sample images of packages that the customer service assistant will analyze to determine the appropriate action. We will encode the images as base64 strings for processing by the model."
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "code",
2025-01-09 13:13:27 +00:00
"execution_count": 3,
"id": "2f2066d40dbbfbe8",
"metadata": {
"ExecuteTime": {
"end_time": "2024-04-10T03:50:38.120670Z",
"start_time": "2024-04-10T03:50:37.565555Z"
},
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Encoded image: wet_package.jpg\n",
"Encoded image: damaged_package.jpg\n",
"Encoded image: normal_package.jpg\n"
]
},
{
"data": {
2025-01-09 13:13:27 +00:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAABv4AAAGsCAYAAAAVEtBaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/GU6VOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOy9ecBkRX33+6k653T3s86+MDDsiyDuehUNMi4v4PqiRtT3uhIjqEFNgrnqGxSIYtSorxcThai4YTRRMUZZRKNXEg2aiCKLOAMzgMy+PPvT3eecqvtHVZ1T53T3M88AOgmpL/Q83Wep9Vfb71u/XwmttSYgICAgICAgICAgICAgICAgICAgICAgICAgIOC/NOTBTkBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQMCDRyD+AgICAgICAgICAgICAgICAgICAgICAgICAgIeBgjEX0BAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEDAwwCB+AsICAgICAgICAgICAgICAgICAgICAgICAgIeBggEH8BAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQ8DBOIvICAgICAgICAgICAgICAgICAgICAgICAgIOBhgED8BQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQ8DBCIv4CAgICAgICAgICAgICAgICAgICAgICAgICAhwEC8RcQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQ8DBAIP4CAgICAgICAgICAgICAgICAgICAgL+W2DDhg1s2LDhYCdjv9iyZQtCCP7qr/7qYCclICDgvxgC8RcQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQ8DBAIP4CAgICAgICAgICAgICAgICAgICAgICAgICAh4GCMRfQEBAQEBAQEBAQEBAQEBAQEBAQEDA7xxaa+bn5w92MgICAgIeVgjEX0BAQEBAQEBAQEBAQEBAQEBAQEBAQA8uuugihBBs2rSJ1772tSxdupQlS5bwute9jrm5ueK5LMv4i7/4C4455hiazSZHHnkk73rXu+h0OpXwjjzySJ7//Odz/fXX88QnPpGhoSEuv/xyfvCDHyCE4O///u+5+OKLOfTQQxkbG+P3f//3mZycpNPp8La3vY3Vq1czOjrK6173up6wr7zySp75zGeyevVqms0mJ510Ep/4xCceknJw6f7Od77DYx/7WFqtFieddBJf//rXK8/t3buXCy64gEc96lGMjo4yPj7Oc57zHH7xi1/0hNlut7nooos4/vjjabVaHHLIIbz4xS/mrrvuGpgOrTVveMMbaDQaRdy33HILr33tazn66KNptVqsXbuWc845hz179vS8/4Mf/IAnPvGJtFotjjnmGC6//PKijuv44he/yBOe8ASGhoZYvnw5L3/5y7nvvvsOtOgCAgIOAuKDnYCAgICAgICAgICAgICAgICAgICAgID/vDj77LM56qijeP/738/PfvYzPvWpT7F69Wo+8IEPAPD617+ez33uc/z+7/8+f/qnf8pNN93E+9//fu644w6uvvrqSlh33nknr3jFKzj33HP5wz/8Q0444YTi3vvf/36GhoZ4xzvewaZNm7jssstIkgQpJfv27eOiiy7i3/7t3/jsZz/LUUcdxbvf/e7i3U984hM88pGP5IUvfCFxHPNP//RPvOlNb0IpxZvf/OYHXQYbN27kZS97Geeddx6vec1ruPLKK3npS1/Kddddx//4H/8DgLvvvptvfOMbvPSlL+Woo45ix44dXH755Zx22mncfvvtrFu3DoA8z3n+85/P9773PV7+8pfz1re+lenpaW644QZuvfVWjjnmmJ748zznnHPO4Stf+QpXX301z3ve8wC44YYbuPvuu3nd617H2rVrue2227jiiiu47bbb+Ld/+7eC1Lv55ps588wzOeSQQ7j44ovJ85xLLrmEVatW9cT1vve9jwsvvJCzzz6b17/+9ezatYvLLruMpz/96dx8880sXbr0QZdnQEDAbxE6ICAgICAgICAgICAgICAgICAgICAgoIb3vOc9GtDnnHNO5fqLXvQivWLFCq211j//+c81oF//+tdXnrngggs0oP/5n/+5uHbEEUdoQF933XWVZ7///e9rQJ988sm62+0W11/xildoIYR+znOeU3n+lFNO0UcccUTl2tzcXE/6zzjjDH300UdXrp122mn6tNNOWzjjNbh0f+1rXyuuTU5O6kMOOUQ/7nGPK661222d53nl3c2bN+tms6kvueSS4tpnPvMZDeiPfOQjPXEppYr3AP2hD31Ip2mqX/ayl+mhoSF9/fXXV57vl++/+7u/04D+4Q9/WFx7wQteoIeHh/X9999fXNu4caOO41j7NMGWLVt0FEX6fe97XyXMX/7ylzqO457rAQEB//kQXH0GBAQEBAQEBAQEBAQEBAQEBAQEBAQMxHnnnVf5feqpp7Jnzx6mpqa45pprAPiTP/mTyjN/+qd/CsC3v/3tyvWjjjqKM844o288r371q0mSpPj95Cc/Ga0155xzTuW5Jz/5ydx3331kWVZcGxoaKr5PTk6ye/duTjvtNO6++24mJycXm9WBWLduHS960YuK3+Pj47z61a/m5ptvZvv27QA0m02kNCr3PM/Zs2cPo6OjnHDCCfzsZz8r3v3a177GypUrOf/883viqbvd7Ha7vPSlL+Vb3/oW11xzDaeffnrlvp/vdrvN7t27ecpTngJQxJnnOd/97nc566yzCqtDgGOPPZbnPOc5lfC+/vWvo5Ti7LPPZvfu3cVn7dq1HHfccXz/+99ffKEFBAQcFARXnwEBAQEBAQEBAQEBAQEBAQEBAQEBAQNx+OGHV34vW7YMgH379nHPPfcgpeTYY4+tPLN27VqWLl3KPffcU7l+1FFHLTqeJUuWALB+/fqe60opJicnWbFiBQD/+q//ynve8x5+/OMfV84fBEMEurAeKI499tgeUu74448HYMuWLaxduxalFB/72Mf4m7/5GzZv3kye58WzLp0Ad911FyeccAJxvH/1/Pvf/35mZma49tpr2bBhQ8/9vXv3cvHFF/PlL3+ZnTt3Vu45wnPnzp3Mz8/31JHLl4+NGzeitea4447rmx6fmA0ICPjPiUD8BQQEBAQEBAQEBAQEBAQEBAQEBAQEDEQURX2va62L73VSbBB8C7XFxrO/+O+66y6e9axn8YhHPIKPfOQjrF+/nkajwTXXXMNHP/pRlFKLStuDxaWXXsqFF17IOeecw1/8xV+wfPlypJS87W1ve8BpOOOMM7juuuv44Ac/yIYNG2i1WpX7Z599Nj/60Y94+9vfzmMf+1hGR0dRSnHmmWc+oDiVUgghuPbaa/uW++jo6APKR0BAwO8OgfgLCAgICAgICAgICAgICAgICAgICAh4QDjiiCNQSrFx40ZOPPHE4vqOHTuYmJjgiCOO+K2n4Z/+6Z/odDp885vfrFgNPpRuKTdt2oTWukJw/vrXvwbgyCOPBOCrX/0qz3jGM/j0pz9deXdiYoKVK1cWv4855hhuuukm0jTdrwXdU57yFM477zye//zn89KXvpSrr766sBTct28f3/ve97j44ot597vfXbyzcePGShirV6+m1WqxadOmvvnyccwxx6C15qijjiosGgMCAv5rIZzxFxAQEBAQEBAQEBAQEBAQEBAQEBAQ8IDw3Oc+F4D/83/+T+X6Rz7yEQCe97zn/dbT4CzTfAvEyclJrrzyyocsjq1bt3L11VcXv6empvj85z/PYx/7WNauXVukw08DwD/8wz9w//33V6695CUvYffu3Xz84x/viaf+PsCzn/1svvzlL3Pdddfxqle9qrDk65dv6K2LKIp49rOfzTe+8Q22bt1aXN+0aRPXXntt5dkXv/jFRFHExRdf3BOu1po9e/b0pC8gIOA/F4LFX0BAQEBAQEBAQEBAQEBAQEBAQEBAwAPCYx7zGF7zmtdwxRVXMDExwWmnncZPfvITPve5z3HWWWfxjGc847eehtNPP51Go8ELXvACzj33XGZmZvjbv/1bVq9ezbZt2x6SOI4//nj+4A/+gJ/+9KesWbO
"text/plain": [
"<Figure size 1800x600 with 3 Axes>"
]
2024-04-10 08:53:14 -07:00
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Function to encode the image as base64\n",
"def encode_image(image_path: str):\n",
" # check if the image exists\n",
" if not os.path.exists(image_path):\n",
" raise FileNotFoundError(f\"Image file not found: {image_path}\")\n",
" with open(image_path, \"rb\") as image_file:\n",
" return base64.b64encode(image_file.read()).decode('utf-8')\n",
"\n",
"\n",
"# Sample images for testing\n",
"image_dir = \"images\"\n",
"\n",
"# encode all images within the directory\n",
"image_files = os.listdir(image_dir)\n",
"image_data = {}\n",
"for image_file in image_files:\n",
" image_path = os.path.join(image_dir, image_file)\n",
" # encode the image with key as the image file name\n",
" image_data[image_file.split('.')[0]] = encode_image(image_path)\n",
" print(f\"Encoded image: {image_file}\")\n",
"\n",
"\n",
"def display_images(image_data: dict):\n",
" fig, axs = plt.subplots(1, 3, figsize=(18, 6))\n",
" for i, (key, value) in enumerate(image_data.items()):\n",
" img = Image.open(BytesIO(base64.b64decode(value)))\n",
" ax = axs[i]\n",
" ax.imshow(img)\n",
" ax.axis(\"off\")\n",
" ax.set_title(key)\n",
" plt.tight_layout()\n",
" plt.show()\n",
"\n",
"\n",
"display_images(image_data)"
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "f4b16c5219dff6db",
"metadata": {
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"source": [
"We have successfully encoded the sample images as base64 strings and displayed them. The customer service assistant will analyze these images to determine the appropriate action based on the package condition.\n",
"\n",
"Let's now define the functions/tools for order processing, such as escalating an order to an agent, refunding an order, and replacing an order. We will create placeholder functions to simulate the processing of these actions based on the identified tools. We will be using Pydantic models to define the structure of the data for order actions.\n"
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "code",
2025-01-09 13:13:27 +00:00
"execution_count": 4,
"id": "115b5a0bf208b054",
"metadata": {
"ExecuteTime": {
"end_time": "2024-04-10T03:50:38.135826Z",
"start_time": "2024-04-10T03:50:38.122420Z"
},
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"outputs": [],
"source": [
2025-01-09 13:13:27 +00:00
"MODEL = \"gpt-4o-2024-11-20\"\n",
2024-04-10 08:53:14 -07:00
"\n",
"class Order(BaseModel):\n",
" \"\"\"Represents an order with details such as order ID, customer name, product name, price, status, and delivery date.\"\"\"\n",
" order_id: str = Field(..., description=\"The unique identifier of the order\")\n",
" product_name: str = Field(..., description=\"The name of the product\")\n",
" price: float = Field(..., description=\"The price of the product\")\n",
" status: str = Field(..., description=\"The status of the order\")\n",
" delivery_date: str = Field(..., description=\"The delivery date of the order\")\n",
"# Placeholder functions for order processing\n",
"\n",
"def get_order_details(order_id):\n",
" # Placeholder function to retrieve order details based on the order ID\n",
" return Order(\n",
" order_id=order_id,\n",
" product_name=\"Product X\",\n",
" price=100.0,\n",
" status=\"Delivered\",\n",
" delivery_date=\"2024-04-10\",\n",
" )\n",
"\n",
"def escalate_to_agent(order: Order, message: str):\n",
" # Placeholder function to escalate the order to a human agent\n",
" return f\"Order {order.order_id} has been escalated to an agent with message: `{message}`\"\n",
"\n",
"def refund_order(order: Order):\n",
" # Placeholder function to process a refund for the order\n",
" return f\"Order {order.order_id} has been refunded successfully.\"\n",
"\n",
"def replace_order(order: Order):\n",
" # Placeholder function to replace the order with a new one\n",
" return f\"Order {order.order_id} has been replaced with a new order.\"\n",
"\n",
"class FunctionCallBase(BaseModel):\n",
" rationale: Optional[str] = Field(..., description=\"The reason for the action.\")\n",
" image_description: Optional[str] = Field(\n",
" ..., description=\"The detailed description of the package image.\"\n",
" )\n",
" action: Literal[\"escalate_to_agent\", \"replace_order\", \"refund_order\"]\n",
" message: Optional[str] = Field(\n",
" ...,\n",
" description=\"The message to be escalated to the agent if action is escalate_to_agent\",\n",
" )\n",
" # Placeholder functions to process the action based on the order ID\n",
" def __call__(self, order_id):\n",
" order: Order = get_order_details(order_id=order_id)\n",
" if self.action == \"escalate_to_agent\":\n",
" return escalate_to_agent(order, self.message)\n",
" if self.action == \"replace_order\":\n",
" return replace_order(order)\n",
" if self.action == \"refund_order\":\n",
" return refund_order(order)\n",
"\n",
"class EscalateToAgent(FunctionCallBase):\n",
" \"\"\"Escalate to an agent for further assistance.\"\"\"\n",
" pass\n",
"\n",
"class OrderActionBase(FunctionCallBase):\n",
" pass\n",
"\n",
"class ReplaceOrder(OrderActionBase):\n",
" \"\"\"Tool call to replace an order.\"\"\"\n",
" pass\n",
"\n",
"class RefundOrder(OrderActionBase):\n",
" \"\"\"Tool call to refund an order.\"\"\"\n",
" pass"
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "5716c84cb57e3fb3",
2024-04-10 08:53:14 -07:00
"metadata": {
"collapsed": false
},
2025-01-09 13:13:27 +00:00
"source": [
"### Simulating user messages and processing the package images\n",
"\n",
"We will simulate user messages containing the package images and process the images using the GPT-4o with Vision model. The model will identify the appropriate tool call based on the image analysis and the predefined actions for damaged, wet, or normal packages. We will then process the identified action based on the order ID and display the results."
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "code",
2025-01-09 13:13:27 +00:00
"execution_count": 11,
"id": "dacd602d9e2728d9",
"metadata": {
"ExecuteTime": {
"end_time": "2024-04-10T03:50:53.927476Z",
"start_time": "2024-04-10T03:50:38.136857Z"
},
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Processing delivery exception support for different package images...\n",
"\n",
"===================== Simulating user message 1 =====================\n",
"- Tool call: refund_order for provided img: damaged_package\n",
2025-01-09 13:13:27 +00:00
"- Parameters: rationale='The package appears damaged as it is visibly crushed and deformed.' image_description='A package that is visibly crushed and deformed, with torn and wrinkled packaging material.' action='refund_order' message=None\n",
2024-04-10 08:53:14 -07:00
">> Action result: Order 12345 has been refunded successfully.\n",
"\n",
"===================== Simulating user message 2 =====================\n",
"- Tool call: escalate_to_agent for provided img: normal_package\n",
2025-01-09 13:13:27 +00:00
"- Parameters: rationale='The package appears normal and undamaged in the image.' image_description='A cardboard box placed on a wooden floor, showing no visible signs of damage or wetness.' action='escalate_to_agent' message='The package appears normal and undamaged. Please review further.'\n",
">> Action result: Order 12345 has been escalated to an agent with message: `The package appears normal and undamaged. Please review further.`\n",
2024-04-10 08:53:14 -07:00
"\n",
"===================== Simulating user message 3 =====================\n",
"- Tool call: replace_order for provided img: wet_package\n",
2025-01-09 13:13:27 +00:00
"- Parameters: rationale='The package appears wet, which may compromise its contents.' image_description=\"A cardboard box labeled 'Fragile' with visible wet spots on its surface.\" action='replace_order' message=None\n",
2024-04-10 08:53:14 -07:00
">> Action result: Order 12345 has been replaced with a new order.\n"
]
}
],
"source": [
"# extract the tool call from the response\n",
"ORDER_ID = \"12345\" # Placeholder order ID for testing\n",
"INSTRUCTION_PROMPT = \"You are a customer service assistant for a delivery service, equipped to analyze images of packages. If a package appears damaged in the image, automatically process a refund according to policy. If the package looks wet, initiate a replacement. If the package appears normal and not damaged, escalate to agent. For any other issues or unclear images, escalate to agent. You must always use tools!\"\n",
"\n",
"def delivery_exception_support_handler(test_image: str):\n",
" payload = {\n",
" \"model\": MODEL,\n",
" \"response_model\": Iterable[RefundOrder | ReplaceOrder | EscalateToAgent],\n",
" \"tool_choice\": \"auto\", # automatically select the tool based on the context\n",
" \"temperature\": 0.0, # for less diversity in responses\n",
" \"seed\": 123, # Set a seed for reproducibility\n",
" }\n",
" payload[\"messages\"] = [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": INSTRUCTION_PROMPT,\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\n",
" \"type\": \"image_url\",\n",
" \"image_url\": {\n",
" \"url\": f\"data:image/jpeg;base64,{image_data[test_image]}\"\n",
" }\n",
" },\n",
" ],\n",
" }\n",
" ]\n",
" function_calls = instructor.from_openai(\n",
" OpenAI(), mode=instructor.Mode.PARALLEL_TOOLS\n",
" ).chat.completions.create(**payload)\n",
2025-01-09 13:13:27 +00:00
"\n",
2024-04-10 08:53:14 -07:00
" for tool in function_calls:\n",
" print(f\"- Tool call: {tool.action} for provided img: {test_image}\")\n",
" print(f\"- Parameters: {tool}\")\n",
" print(f\">> Action result: {tool(ORDER_ID)}\")\n",
" return tool\n",
"\n",
"\n",
"print(\"Processing delivery exception support for different package images...\")\n",
"\n",
"print(\"\\n===================== Simulating user message 1 =====================\")\n",
"assert delivery_exception_support_handler(\"damaged_package\").action == \"refund_order\"\n",
"\n",
"print(\"\\n===================== Simulating user message 2 =====================\")\n",
"assert delivery_exception_support_handler(\"normal_package\").action == \"escalate_to_agent\"\n",
"\n",
"print(\"\\n===================== Simulating user message 3 =====================\")\n",
"assert delivery_exception_support_handler(\"wet_package\").action == \"replace_order\""
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "78604bb7ff61b472",
"metadata": {
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"source": [
"## 2. Analyzing an organizational chart to extract employee information\n",
"\n",
2025-01-09 13:13:27 +00:00
"For the second example, we will analyze an organizational chart image to extract employee information, such as employee names, roles, managers, and manager roles. We will use GPT-4o with Vision to process the organizational chart image and extract structured data about the employees in the organization. Indeed, function calling lets us go beyond OCR to actually deduce and translate hierarchical relationships within the chart.\n",
2024-04-10 08:53:14 -07:00
"\n",
"We will start with a sample organizational chart in PDF format that we want to analyze and convert the first page of the PDF to a JPEG image for analysis."
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "code",
2025-01-09 13:13:27 +00:00
"execution_count": 12,
"id": "c3fc6f64b2a2f8e6",
"metadata": {
"ExecuteTime": {
"end_time": "2024-04-10T03:50:54.033536Z",
"start_time": "2024-04-10T03:50:53.929732Z"
},
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"outputs": [
{
"data": {
2025-01-09 13:13:27 +00:00
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAJAAwADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDpgMjNOHXFIo4p4546UmdaY5eOlSAcUwDingf/AK6kq44DGBTx2IpoHPapAKljTHDrTwB1po+6KeOOKllXHU5fSkApwHPWpaHccBipB9aaOntThioaHckAxUyduKiWplBFS0S2TpVhO1VkNWEpWM5FhKmXtUC1OtWkZMmU1IKiWpBWsTNkgp1MFOFaIkcKKQUtUIKKKKYBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAeUqKfjC0gA/wD108cgUmb3FWnimjjingelIq45R7U8Cmin4GBipHccBx7U8cU0ehp4GOnWkykxwz+FOHSk7U4DvU2HccKeDTQDTx2qWh3HjNTL+tQrUq9RU2FcmSrCGq6VOppWIZZWpkqulTrVJGbJlqUVCtSrWiIZIKcKaKcK0RItLSClpiCiiimAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHlwHNOwMjtSgU8DFIu4gApwHegDingUDTAdjTl/wAigdcCnAce9SUmOA5pQOaAP0pQMnpQO44etO6Ug9qcKloakOHtTh9O9NHWndcUrDuSA81InrUannFSIcVNguTrU6nkVXX1qdDRYTZOlTrVdDzU6dqaRDJ1qVahU1KtWiWSCnimCniqRItLSClqhBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHmmOtOUZFAGacF/CgVwAPpTulAFPAOPagLgB/8AWpQOePzoA/Gnhe1Kw7iDkU8DHNIBinAUWKuHU04elGOlGPSlYaYtOHUGm9BTunNKxVx4/UVIpzUQJ6VKvalYdyZKnWq61OnalYLlhOKnWoEqZKdiSdalWoVqZaZJIKcKaKcKoQtOpop1MQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQB5yB0p4HPNKq04Lg0jK4gHFKBTgOtOC0wuIq8+9LjrT9vrTgOtMdxgFLt+angUYxQNMbgkc0vOO1KRR9aLDuN+tKKOetHelYpMcDipVIqEY4qVaTRVydTUyGoFNTJSsO5ZSp0qsnarCdqLCuTqalWoVqZaYEgp4pgp4piHUtNFOFAgoopCQASTgDuaAFoqtBqFldSNHb3lvM69VjkViPwBqzQAUUUUAFFFFABRRRQAUUUUAFFUNW1nTdCsje6rfQWdsGC+ZM4UFj0A9T7e1Wba5gvLaK5tpkmglUPHJGwZXU9CCOooAmooooAKKKKACiis7V9d0nw/bJc6vqNtZQu+xHnkChm9BnrQBo0UyKSOeJJYnWSN1DI6HIYHkEEdRT6ACiiigAooooAKKKZJNHEAZJETPTcwGaAH0UgYMoYEEHkEd6WgAoqsNQsmuPs63luZ848oSLu/LOas0AFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBwQWlCdxTwvX2p4X2pIwGBKeF5p6rTwlUBFtFO21IExTtnNMZFt9aTbU2OKTZTC5CRSEVKVppFA7kZGKM07HNIRRYdwA49qkXrxUYHSpFpWKTJV6ip0OcVAtTJSsVcsIanSq6VOlKw7lhamWoUqVaAJRTxTBTxQIcKUU2nCgAr5o8Wa7r/wAWviNJ4T0W7MGkwSPGAGIjZUOGlfH3ueg9x3JNfS9fMvwFddO+KeqWN4Ql29pNCobqXWRCy/XCsfwoAueIfgDf+HtGk1fQ9ekub2zQzGMQ+SzBRklGDHBHYd/Wu4+C3xCuvFmgXdlq8pl1HTApMx+9NEc4J9WBBBPfjuTXp19dQWNhcXd0yrbwRNJKzdAoGT+lfOH7O0Era/r9woPkpp+xvQMzAr+itQB6db/HLwXcadeXvn3kcdrsBWSDDSM2cKgB5Pyn0HFdHpHjvRNW8Ht4p82Sz0pd26S7TYRtO08AnPPAxnJ4618+/AnwrpHiXxNfvrFol3FZ24eOCQZQszYyR3wM8Hjmuy/aCWPRfBug6LpsCWuntcu/kwrtQbRkDA7Zdjj1oA6/Sfjf4M1jWU02K4uoHkbZFNcQ7I3bsAckjPuBWv4O+JOg+Obq5ttH+1eZbIJJPPiCDBOOOTXOeFPhx4Vu/hJYRz6ZZvNe6ctxLetEDKsjpu3B8ZG0nj6fWuD/AGbCBr+uknA+yp/6FQB6l4o+L/hLwnqL6fd3U9zeRnEsNpHvMZ9GJIGfbORV7wj8SvDPjWV7fSrtxdou821wmyTb6jsfwJxXkcPiXwJYeL7yXwj4H1DxJqDF98jM0qMS2WdAwc8n+LA6+9c74Znul/aCsZpdIGiTS3mXsEGBCHj6dB1Bz0HXoKAPe9V+Jvh3RvF8fhm9e4S+coN/ljyl3DIJYngY61if8L58Df2r9i+1Xfl7tv2v7OfK+vXdj3215d8TtPh1X9oKz065yYLueyglAOCVfap/Qmuj+PHg3w9ovg7T9Q0rSbWxuEvVty1vGE3oUc/NjqcqOTz1oA7L4wN4Su/B1i/iWa9+wy3KtbT6ftZt5RiDzwVK5/Sr9j4n8M+DvhfperW4vBoaRJHBlN0pDE4LDPUnJNeReN5pJ/2dPBjyMWYXITJ9FWUAfkBXr3w902w1b4S+H7XUrK3vLc
2024-04-10 08:53:14 -07:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAwAAAAJACAIAAAC1zJYBAAEAAElEQVR4Aez9B6BtSXYW9p9873vdEyUkkFBAQgEEiCCQBMgoEEzOmGByNP5jDDLGNgjBDDYYYxNtgm1MFiaKIEtgkDQzRiBAIiMUAIFQQhrNTKf37j3p//vW2mff8+57r6d7pif17OrX+9auWrVq1Tr77PWdVauq5l/4h/7U8ji/Xu9nh93ueDgulk/sV/PF6vq4vVisDsvjYrebL2bz1XK5318f9/PDxXI5ny2Oi9XicNgpnh8X0m4/W8yOauazw/F4nM3ne80W89nV1Wqz1Gy1WM8Py/nusF7NZvP9YT07Xu+W85W2R23ms9nssF4s91rOj4fDYbNcHXb742x/ebG6vn7uYn93vzwetZ3N8JcwOy5Xd69Xu9318g6R9ofjTsPV8vKwOy4PYTRfRJJiPlvO5uv54v7F/Tvb5f643C+ItMdQ7+trAmxIdVzsFwtFy5nRKFscrmez5W62ms2Xy8VxtdAu5TNybRYroyXDfr48llh0sYr0x+OexlQvF+5kpdl6narQzQx5tsQ/aX/ILcroATnNRhULpAvKpfXZ8VBKIVgaUukRTaeIoklJq3H4z+eEd10cwl5K+emKOLVjaj7DLUk1rJvi09mR9lZmkP9W6XQ7aWDSwKSBSQOTBt4nNLDaAxWDiZ7DHyDFar5ifC8XG+DhCLMs10x+W8TlYrlbKFnOD8fVXtqxxoDO9gjsHI/z1YGxPUAAs8PsADDMmP8Vs81Gz9luJLP5AagJLYSwBnGCZMig3eV6k352uwWrvz3sQJTVkrHfHo6H+er+xSzoZrcDYi7my9WCkLPDnrmHdLAN5Rz+WOwCAyCbY9AAvJCuVUYmvR1gnZ16OAlegmtmM4wgDMM+EppcUCDclOvBoDYGaxR7lQRHStJCOQEhURwsE+YZW7CUvwdkgVAlFwEIovCwC9Ap4dC7D+KZ6zeYrtoXG1lEIEvBjxQFZc0jV/3D6vmByfvEYzcJOWlg0sCkgUkDkwbesxoAJAJFIBpWls+Avd0fDqDNmrfjcNwvD/CD4u2eE+fIUbEJnIkPgpODh0Ri6jk3DkuOHy4b1hoKAUaw2h33u8XsEhiBVcorES66C2zZbzerNT6BU1DUnmdltt1uV8fFegOMsPfollv4arZcXvDCxO+hdgEP7fwXYeY8SnMNuW7glvlqzSkE54BFMItOSQjEEClwJAACADrOtwsenYWRXWV0Mwhqvzhu8a2PYgkxYY6dKg3KY8JdFG7xKsFDFGCUu7hu4kEK4tMRuBUMxosUP01cOnpEHE1oQGEZgRxFRSg4KbpNwySFQTsPohvun/wXLkknyoEuei2UFRj2mBQUGj0M1UFU+QDTbLhWVcpJcOpClZKxx5BOadLApIFJA5MGJg28jDSw2oILrKeZGBafh4e5ZCALjrDrTCBD7spKB1yYuAFnQB9mPzMvcUeY/wEejss1GBCTz86z/iDIHoyIjea4yVQUo4u3RqZ3+Hb4nQCXGWQVzw5CTpMVSMQxI8UqBxXtM/2jrakyjPZq49AxnbTk6tHfkf9GXtfy82N8SDHxprGIVZBGcQGXfGjgxhaC4rIiXdxYZv/SFxR1x5Di34nPKFNchJwft8vlaqtzLZUrK5xS91BP9Rr4AsUFb5FQPYXoPeMswTEtlBMwcdKi8WSIBpxmBYBKzwVAgpdMsDVoC8dKjUXS6oRRHpc/tei/+ezkdFMqrWs+E2NIeTKnquG+S6frpIFJA5MGJg1MGnhZa8D8S9ALi33wX5thNtPNwVxSJnd2ux3kUvEh3CDHw2rFB7LYAxiFFOazLewBLYjnYfnhFGYdSop3aLNcXB7hjQMOATGFkIJVwIP9dUKO1pkgE7wTn9P19TUB7jxxR0xPpp8QHWcXS90t9rutiTj4Z3CpLOb8TwnGIXkCh9j1XZqAOvDGAbja7GbXwVvBGpnhgkI6D4rxVfEL8SytDwEGnFT6fdaQDkAeD0/ib4xYG4hliUoF340xhryRTToiPy9ZkFEgRfw3iT0K6MsYB6QCNBWygAk9SKlyG/qU1l0wSrmCApR0k/JTbQj1mKbdC5mKnYJHp5Dh54r00SQvrPTtdfTCuExUkwYmDUwamDQwaeC9UgOibICbuDti7mvahs0NCMgcWIyg2a8Y7Lht2paDQWVhIQouISTuQIpheAlNZoLjiAGLBPLCDxV4e+C8gZTi4OCR2a+X6/32WrPMkyWgeS4EabPcXO1314Ka+VFALLHK4oqAseNuSyCOH8LNBAMJMwZ3EsOTGTUsk1KKeZoG1cXfI0WOMySgEBezYOpNhwFqW/NY0EKDq8zmGdURGirssbiTeOYMJpAl4Us6zLRcwphNH1YQslDxEiAXPYYGi1Ja0E0m0QQRlTAqAwkL0hTTdrAl6z8pn0PFTo8czzMFv84Lnjcf3NSpxEzW4ILAztJ4WyM8q5iykwYmDUwamDQwaeDlq4GVIBqByKy61V2JJN4LiLGQanY1T7hMJo/8Lx0SayOMZXl9LWpa8LPElF+Idqk4mcw+xatiQVdmiiS3IMlcPHNKYIVAH36QQKAZmAMRFDzg4uFYisvEUrJtYoIWa1NtQRLik2Kwj+sLHp2KQ0qciiht0EO0zV680nZt9k6H/gWZQTLkP86uTVEp0irCBHsUTQX3XAM0xnMwdCMHZyxmE7MdyWE+s2I8ORX5E3bWgMWfk5k9qCWQwtip4iLCZ9UXJZW7iARCqciVCCSd0Voyy3jACt+AmYGQXEYRaRAq2KrzQT8ZbFBocYgnLY21KWxUXMshpCAOplRI6ezxSfNb6dR/io11FIZEEwa6pavpdtLApIFJA5MGXq4aMAUlEphhz6r1TFztd7vD3mqug+jeQJTFOs6fmSXoPB6HFR8LiCLIBTgx/zNbJoZaGLFJontZJg4DJKo4Rn0pnHgtWDp2PJNRwUlmq6y0D1CBZDar5Xa/ZeQzs2aR1PoSAFod17wzZsTYdbNt6La7vVkuVJBBgnsW8y2Est+tDjPr08RFZwouM3K8SFaei9nhbdke3Mavk/Bo4xPdFMgQBLK4NhaL1A8zfYN1xn693K/jPDJkvIIQ4sMCDYJxAJ40rQ6yYouOaCydpgMAJQBIMjop03/BNNjRT9aVgSBUWmBJfXATgnCEnWAp+ijEU4gouqpHjaApqEtddVUV3bay8Rg1zdltZx971etj6x6sOO/owZrpbtLApIFJA5MGJg28z2tgJfbFeqYdgFHLssTLXHByMObX28XmEgzgvlnZyee4v9qJ0Vmy6YKW7zLHT2TmK2AiYcz7tXAdK6Pcl8/BzBVks726t9zaYcfuOlvzOhaHH46mxGJbV5e77XZvnms9Ewh0AFh2+2tIYLu4D/LAIAKiQQTrvQIUMh8FU8ARQWvxOXHTbGb3kSU6xnY9e5vm7A735hvwQkebDfQBuaws5jeBZuEY304CgYCr5fXcBBzYkVjveFsWi+us7TcZJx4pZBkmaHPcgFfX5sDi24FMtiTM3jw2NIKvtMV4xx20AmvgJuBnvrtvTVxYEwOkJFrQzUVP1cFM8bXhFRRSLiJ/5fMPXT9NhVECr9wSOnU1ixZvTSBXApEKYMFS+knKZGPwWvlwRj7hQRUNthKA3SmwKSkNC+t18c18Weqqvisec8XvgSaPIZuKJw1MGpg0MGlg0sB7owZsQ2PWxnKo2MMgh3KUuFnZGJCR5uhIYvGtPN8vBPkuLirGWVROQaGltecVnAOqhPK6FoBz8mjourp7J/NQGLDaQAXvEDCRHYC4SLib9msYB20hG5YZaBE2ZDk74GCPovQel0l5amJyAwKCHGonwZoZMgOmSe2mU4AgVjnumQCPWPrgIeigbsRaJ5aJKyvSBPTgv9tnRZe5qsYaIrK10tj/iutOp4UJlEhiusX0COxO8HUSHxP2aWZ0eiunjpVupIfydJc47ZsUNaRFkFd2bwzLIU7bxxB4GNCSDyS9umn5McjM14lTak+jQ5GbStV4vLspDA0pB/QzlI9/HtlqrJ0ykwYmDUwamDQwaeBlpoHacSe
2025-01-09 13:13:27 +00:00
"text/plain": [
"<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=768x576>"
]
2024-04-10 08:53:14 -07:00
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Function to convert a single page PDF page to a JPEG image\n",
"def convert_pdf_page_to_jpg(pdf_path: str, output_path: str, page_number=0):\n",
" if not os.path.exists(pdf_path):\n",
" raise FileNotFoundError(f\"PDF file not found: {pdf_path}\")\n",
" doc = fitz.open(pdf_path)\n",
" page = doc.load_page(page_number) # 0 is the first page\n",
" pix = page.get_pixmap()\n",
" # Save the pixmap as a JPEG\n",
" pix.save(output_path)\n",
"\n",
"\n",
"def display_img_local(image_path: str):\n",
" img = Image.open(image_path)\n",
" display(img)\n",
"\n",
"\n",
"pdf_path = 'data/org-chart-sample.pdf'\n",
"output_path = 'org-chart-sample.jpg'\n",
"\n",
"convert_pdf_page_to_jpg(pdf_path, output_path)\n",
"display_img_local(output_path)"
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "5f6b4299c99721bc",
2024-04-10 08:53:14 -07:00
"metadata": {
"collapsed": false
},
2025-01-09 13:13:27 +00:00
"source": [
"The organizational chart image has been successfully extracted from the PDF file and displayed. Let's now define a function to analyze the organizational chart image using the new GPT4o with Vision. The function will extract information about the employees, their roles, and their managers from the image. We will use function/tool calling to specify the input parameters for the organizational structure, such as the employee name, role, and manager's name and role. We will use Pydantic models to define the structure of the data.\n"
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "code",
2025-01-09 13:13:27 +00:00
"execution_count": 15,
"id": "b11469486fc95c23",
"metadata": {
"ExecuteTime": {
"end_time": "2024-04-10T03:50:54.043273Z",
"start_time": "2024-04-10T03:50:54.034475Z"
},
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"outputs": [],
"source": [
"base64_img = encode_image(output_path)\n",
"\n",
"class RoleEnum(str, Enum):\n",
" \"\"\"Defines possible roles within an organization.\"\"\"\n",
" CEO = \"CEO\"\n",
" CTO = \"CTO\"\n",
" CFO = \"CFO\"\n",
" COO = \"COO\"\n",
" EMPLOYEE = \"Employee\"\n",
" MANAGER = \"Manager\"\n",
" INTERN = \"Intern\"\n",
" OTHER = \"Other\"\n",
"\n",
"class Employee(BaseModel):\n",
" \"\"\"Represents an employee, including their name, role, and optional manager information.\"\"\"\n",
" employee_name: str = Field(..., description=\"The name of the employee\")\n",
" role: RoleEnum = Field(..., description=\"The role of the employee\")\n",
" manager_name: Optional[str] = Field(None, description=\"The manager's name, if applicable\")\n",
" manager_role: Optional[RoleEnum] = Field(None, description=\"The manager's role, if applicable\")\n",
"\n",
"\n",
"class EmployeeList(BaseModel):\n",
" \"\"\"A list of employees within the organizational structure.\"\"\"\n",
" employees: List[Employee] = Field(..., description=\"A list of employees\")\n",
"\n",
"def parse_orgchart(base64_img: str) -> EmployeeList:\n",
" response = instructor.from_openai(OpenAI()).chat.completions.create(\n",
2025-01-09 13:13:27 +00:00
" model=MODEL,\n",
2024-04-10 08:53:14 -07:00
" response_model=EmployeeList,\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": 'Analyze the given organizational chart and very carefully extract the information.',\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\n",
" \"type\": \"image_url\",\n",
" \"image_url\": {\n",
" \"url\": f\"data:image/jpeg;base64,{base64_img}\"\n",
" }\n",
" },\n",
" ],\n",
" }\n",
" ],\n",
" )\n",
" return response"
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "73619ee88ed5cd2",
2024-04-10 08:53:14 -07:00
"metadata": {
"collapsed": false
},
2025-01-09 13:13:27 +00:00
"source": [
"Now, we will define a function to parse the response from GPT-4o with vision and extract the employee data. We will tabulate the extracted data for easy visualization. Please note that the accuracy of the extracted data may vary based on the complexity and clarity of the input image."
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "code",
2025-01-09 13:13:27 +00:00
"execution_count": 16,
"id": "962c5ea24b8a9dc9",
"metadata": {
"ExecuteTime": {
"end_time": "2024-04-10T03:51:03.255172Z",
"start_time": "2024-04-10T03:50:54.044659Z"
},
"collapsed": false
},
2024-04-10 08:53:14 -07:00
"outputs": [
{
"data": {
2025-01-09 13:13:27 +00:00
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>employee_name</th>\n",
" <th>role</th>\n",
" <th>manager_name</th>\n",
" <th>manager_role</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Juliana Silva</td>\n",
" <td>CEO</td>\n",
" <td>None</td>\n",
" <td>None</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Kim Chun Hei</td>\n",
" <td>CFO</td>\n",
" <td>Juliana Silva</td>\n",
" <td>CEO</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Cahaya Dewi</td>\n",
" <td>Manager</td>\n",
" <td>Kim Chun Hei</td>\n",
" <td>CFO</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Drew Feig</td>\n",
" <td>Employee</td>\n",
" <td>Cahaya Dewi</td>\n",
" <td>Manager</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Richard Sanchez</td>\n",
" <td>Employee</td>\n",
" <td>Cahaya Dewi</td>\n",
" <td>Manager</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Sacha Dubois</td>\n",
" <td>Intern</td>\n",
" <td>Cahaya Dewi</td>\n",
" <td>Manager</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Chad Gibbons</td>\n",
" <td>CTO</td>\n",
" <td>Juliana Silva</td>\n",
" <td>CEO</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Shawn Garcia</td>\n",
" <td>Manager</td>\n",
" <td>Chad Gibbons</td>\n",
" <td>CTO</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Olivia Wilson</td>\n",
" <td>Employee</td>\n",
" <td>Shawn Garcia</td>\n",
" <td>Manager</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Matt Zhang</td>\n",
" <td>Intern</td>\n",
" <td>Shawn Garcia</td>\n",
" <td>Manager</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Chiaki Sato</td>\n",
" <td>COO</td>\n",
" <td>Juliana Silva</td>\n",
" <td>CEO</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Aaron Loeb</td>\n",
" <td>Manager</td>\n",
" <td>Chiaki Sato</td>\n",
" <td>COO</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12</th>\n",
" <td>Avery Davis</td>\n",
" <td>Employee</td>\n",
" <td>Aaron Loeb</td>\n",
" <td>Manager</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13</th>\n",
" <td>Harper Russo</td>\n",
" <td>Employee</td>\n",
" <td>Aaron Loeb</td>\n",
" <td>Manager</td>\n",
" </tr>\n",
" <tr>\n",
" <th>14</th>\n",
" <td>Taylor Alonso</td>\n",
" <td>Intern</td>\n",
" <td>Aaron Loeb</td>\n",
" <td>Manager</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" employee_name role manager_name manager_role\n",
"0 Juliana Silva CEO None None\n",
"1 Kim Chun Hei CFO Juliana Silva CEO\n",
"2 Cahaya Dewi Manager Kim Chun Hei CFO\n",
"3 Drew Feig Employee Cahaya Dewi Manager\n",
"4 Richard Sanchez Employee Cahaya Dewi Manager\n",
"5 Sacha Dubois Intern Cahaya Dewi Manager\n",
"6 Chad Gibbons CTO Juliana Silva CEO\n",
"7 Shawn Garcia Manager Chad Gibbons CTO\n",
"8 Olivia Wilson Employee Shawn Garcia Manager\n",
"9 Matt Zhang Intern Shawn Garcia Manager\n",
"10 Chiaki Sato COO Juliana Silva CEO\n",
"11 Aaron Loeb Manager Chiaki Sato COO\n",
"12 Avery Davis Employee Aaron Loeb Manager\n",
"13 Harper Russo Employee Aaron Loeb Manager\n",
"14 Taylor Alonso Intern Aaron Loeb Manager"
]
2024-04-10 08:53:14 -07:00
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# call the functions to analyze the organizational chart and parse the response\n",
"result = parse_orgchart(base64_img)\n",
"\n",
"# tabulate the extracted data\n",
"df = pd.DataFrame([{\n",
" 'employee_name': employee.employee_name,\n",
" 'role': employee.role.value,\n",
" 'manager_name': employee.manager_name,\n",
" 'manager_role': employee.manager_role.value if employee.manager_role else None\n",
"} for employee in result.employees])\n",
"\n",
"display(df)"
2025-01-09 13:13:27 +00:00
]
2024-04-10 08:53:14 -07:00
},
{
"cell_type": "markdown",
2025-01-09 13:13:27 +00:00
"id": "c55b5fbf97ae7ba8",
2024-04-10 08:53:14 -07:00
"metadata": {
"collapsed": false
},
2025-01-09 13:13:27 +00:00
"source": [
"The extracted data from the organizational chart has been successfully parsed and displayed in a DataFrame. This approach allows us to leverage GPT-4o with Vision capabilities to extract structured information from images, such as organizational charts and diagrams, and process the data for further analysis. By using function calling, we can extend the functionality of multimodal models to perform specific tasks or call external functions."
]
2024-04-10 08:53:14 -07:00
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
2025-01-09 13:13:27 +00:00
"version": 3
2024-04-10 08:53:14 -07:00
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
2025-01-09 13:13:27 +00:00
"pygments_lexer": "ipython3",
"version": "3.13.0"
2024-04-10 08:53:14 -07:00
}
},
"nbformat": 4,
"nbformat_minor": 5
}
