jim/openai-cookbook
1
0
Fork 0
mirror of https://github.com/james-m-jordan/openai-cookbook.git synced 2025-05-09 19:32:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
openai-cookbook/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb

575 lines
1.4 MiB
Plaintext
Raw Normal View History

Adds notebook for GPTV with function calling (#1139)
2024-04-10 08:53:14 -07:00
{
"cells": [
{
"cell_type": "markdown",
"source": [
"# How to use GPT-4 Vision with Function Calling \n",
"\n",
"The new GPT-4 Turbo model, available as gpt-4-turbo-2024-04-09 as of April 2024, now enables function calling with vision capabilities, better reasoning and a knowledge cutoff date of Dec 2023. Using images with function calling will unlock multimodal use cases and the ability to use reasoning, allowing you to go beyond OCR and image descriptions.\n",
"\n",
"We will go through two examples to demonstrate the use of function calling with GPT-4 Turbo with Vision:\n",
"\n",
"1. Simulating a customer service assistant for delivery exception support\n",
"2. Analyzing an organizational chart to extract employee information"
],
"metadata": {
"collapsed": false
},
"id": "7b3d17451440ca82"
},
{
"cell_type": "markdown",
"source": [
"### Installation and Setup"
],
"metadata": {
"collapsed": false
},
"id": "feffa794bc28be22"
},
{
"cell_type": "code",
"outputs": [],
"source": [
"!pip install pymupdf --quiet\n",
"!pip install openai --quiet\n",
"!pip install matplotlib --quiet\n",
"# instructor makes it easy to work with function calling\n",
"!pip install instructor --quiet"
],
"metadata": {
"collapsed": false
},
"id": "6ce24dfd4e4bbc47",
"execution_count": null
},
{
"cell_type": "code",
"execution_count": 44,
"id": "initial_id",
"metadata": {
"collapsed": true,
"ExecuteTime": {
"end_time": "2024-04-10T03:50:37.564145Z",
"start_time": "2024-04-10T03:50:37.560040Z"
}
},
"outputs": [],
"source": [
"import base64\n",
"import os\n",
"from enum import Enum\n",
"from io import BytesIO\n",
"from typing import Iterable\n",
"from typing import List\n",
"from typing import Literal, Optional\n",
"\n",
"import fitz\n",
"# Instructor is powered by Pydantic, which is powered by type hints. Schema validation, prompting is controlled by type annotations\n",
"import instructor\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"from IPython.display import display\n",
"from PIL import Image\n",
"from openai import OpenAI\n",
"from pydantic import BaseModel, Field"
]
},
{
"cell_type": "markdown",
"source": [
"## 1. Simulating a customer service assistant for delivery exception support\n",
"We will simulate a customer service assistant for a delivery service that is equipped to analyze images of packages. The assistant will perform the following actions based on the image analysis:\n",
"- If a package appears damaged in the image, automatically process a refund according to policy.\n",
"- If the package looks wet, initiate a replacement.\n",
"- If the package appears normal and not damaged, escalate to an agent."
],
"metadata": {
"collapsed": false
},
"id": "14ab8f931cc759d"
},
{
"cell_type": "markdown",
"source": [
"Let's look at the sample images of packages that the customer service assistant will analyze to determine the appropriate action. We will encode the images as base64 strings for processing by the model."
],
"metadata": {
"collapsed": false
},
"id": "ca35604f27f93e77"
},
{
"cell_type": "code",
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Encoded image: wet_package.jpg\n",
"Encoded image: damaged_package.jpg\n",
"Encoded image: normal_package.jpg\n"
]
},
{
"data": {
"text/plain": "<Figure size 1800x600 with 3 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAABv4AAAGsCAYAAAAVEtBaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOy9ecBkRX33+6k653T3s86+MDDsiyDuehUNMi4v4PqiRtT3uhIjqEFNgrnqGxSIYtSorxcThai4YTRRMUZZRKNXEg2aiCKLOAMzgMy+PPvT3eecqvtHVZ1T53T3M88AOgmpL/Q83Wep9Vfb71u/XwmttSYgICAgICAgICAgICAgICAgICAgICAgICAgIOC/NOTBTkBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQMCDRyD+AgICAgICAgICAgICAgICAgICAgICAgICAgIeBgjEX0BAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEDAwwCB+AsICAgICAgICAgICAgICAgICAgICAgICAgIeBggEH8BAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQ8DBOIvICAgICAgICAgICAgICAgICAgICAgICAgIOBhgED8BQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQ8DBCIv4CAgICAgICAgICAgICAgICAgICAgICAgICAhwEC8RcQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQ8DBAIP4CAgICAgICAgICAgICAgICAgICAgL+W2DDhg1s2LDhYCdjv9iyZQtCCP7qr/7qYCclICDgvxgC8RcQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQ8DBAIP4CAgICAgICAgICAgICAgICAgICAgICAgICAh4GCMRfQEBAQEBAQEBAQEBAQEBAQEBAQEDA7xxaa+bn5w92MgICAgIeVgjEX0BAQEBAQEBAQEBAQEBAQEBAQEBAQA8uuugihBBs2rSJ1772tSxdupQlS5bwute9jrm5ueK5LMv4i7/4C4455hiazSZHHnkk73rXu+h0OpXwjjzySJ7//Odz/fXX88QnPpGhoSEuv/xyfvCDHyCE4O///u+5+OKLOfTQQxkbG+P3f//3mZycpNPp8La3vY3Vq1czOjrK6173up6wr7zySp75zGeyevVqms0mJ510Ep/4xCceknJw6f7Od77DYx/7WFqtFieddBJf//rXK8/t3buXCy64gEc96lGMjo4yPj7Oc57zHH7xi1/0hNlut7nooos4/vjjabVaHHLIIbz4xS/mrrvuGpgOrTVveMMbaDQaRdy33HILr33tazn66KNptVqsXbuWc845hz179vS8/4Mf/IAnPvGJtFotjjnmGC6//PKijuv44he/yBOe8ASGhoZYvnw5L3/5y7nvvvsOtOgCAgIOAuKDnYCAgICAgICAgICAgICAgICAgICAgID/vDj77LM56qijeP/738/PfvYzPvWpT7F69Wo+8IEPAPD617+ez33uc/z+7/8+f/qnf8pNN93E+9//fu644w6uvvrqSlh33nknr3jFKzj33HP5wz/8Q0444YTi3vvf/36GhoZ4xzvewaZNm7jssstIkgQpJfv27eOiiy7i3/7t3/jsZz/LUUcdxbvf/e7i3U984hM88pGP5IUvfCFxHPNP//RPvOlNb0IpxZvf/OYHXQYbN27kZS97Geeddx6vec1ruPLKK3npS1/Kddddx//4H/8DgLvvvptvfOMbvPSlL+Woo45ix44dXH755Zx22mncfvvtrFu3DoA8z3n+85/P9773PV7+8pfz1re+lenpaW644QZuvfVWjjnmmJ748zznnHPO4Stf+QpXX301z3ve8wC44YYbuPvuu3nd617H2rVrue2227jiiiu47bbb+Ld/+7eC1Lv55ps588wzOeSQQ7j44ovJ85xLLrmEVatW9cT1vve9jwsvvJCzzz6b17/+9ezatYvLLruMpz/96dx8880sXbr0QZdnQEDAbxE6ICAgICAgICAgICAgICAgICAgICAgoIb3vOc9GtDnnHNO5fqLXvQivWLFCq211j//+c81oF//+tdXnrngggs0oP/5n/+5uHbEEUdoQF933XWVZ7///e9rQJ988sm62+0W11/xildoIYR+znOeU3n+lFNO0UcccUTl2tzcXE/6zzjjDH300UdXrp122mn6tNNOWzjjNbh0f+1rXyuuTU5O6kMOOUQ/7nGPK661222d53nl3c2bN+tms6kvueSS4tpnPvMZDeiPfOQjPXEppYr3AP2hD31Ip2mqX/ayl+mhoSF9/fXXV57vl++/+7u/04D+4Q9/WFx7wQteoIeHh/X9999fXNu4caOO41j7NMGWLVt0FEX6fe97XyXMX/7ylzqO457rAQEB//kQXH0GBAQEBAQEBAQEBAQEBAQEBAQEBAQMxHnnnVf5feqpp7Jnzx6mpqa45pprAPiTP/mTyjN/+qd/CsC3v/3tyvWjjjqKM844o288r371q0mSpPj95Cc/Ga0155xzTuW5Jz/5ydx3331kWVZcGxoaKr5PTk6ye/duTjvtNO6++24mJycXm9WBWLduHS960YuK3+Pj47z61a/m5ptvZvv27QA0m02kNCr3PM/Zs2cPo6OjnHDCCfzsZz8r3v3a177GypUrOf/883viqbvd7Ha7vPSlL+Vb3/oW11xzDaeffnrlvp/vdrvN7t27ecpTngJQxJnnOd/97nc566yzCqtDgGOPPZbnPOc5lfC+/vWvo5Ti7LPPZvfu3cVn7dq1HHfccXz/+99ffKEFBAQcFARXnwEBAQEBAQEBAQEBAQEBAQEBAQEBAQNx+OGHV34vW7YMgH379nHPPfcgpeTYY4+tPLN27VqWLl3KPffcU7l+1FFHLTqeJUuWALB+/fqe60opJicnWbFiBQD/+q//ynve8x5+/OMfV84fBEMEurAeKI499tgeUu74448HYMuWLaxduxalFB/72Mf4m7/5GzZv3kye58WzLp0Ad911FyeccAJxvH/1/Pvf/35mZma49tpr2bBhQ8/9vXv3cvHFF/PlL3+ZnTt3Vu45wnPnzp3Mz8/31JHLl4+NGzeitea4447rmx6fmA0ICPjPiUD8BQQEBAQEBAQEBAQEBAQEBAQEBAQEDEQURX2va62L73VSbBB8C7XFxrO/+O+66y6e9axn8YhHPIKPfOQjrF+/nkajwTXXXMNHP/pRlFKLStuDxaWXXsqFF17IOeecw1/8xV+wfPlypJS87W1ve8BpOOOMM7juuuv44Ac/yIYNG2i1WpX7Z599Nj/60Y94+9vfzmMf+1hGR0dRSnHmmWc+oDiVUgghuPbaa/uW++jo6APKR0BAwO8OgfgLCAgICAgICAgICAgICAgICAgICAh4QDjiiCNQSrFx40ZOPPHE4vqOHTuYmJjgiCOO+K2n4Z/+6Z/odDp885vfrFgNPpRuKTdt2oTWukJw/vrXvwbgyCOPBOCrX/0qz3jGM/j0pz9deXdiYoKVK1cWv4855hhuuukm0jTdrwXdU57yFM477zye//zn89KXvpSrr766sBTct28f3/ve97j44ot597vfXbyzcePGShirV6+m1WqxadOmvvnyccwxx6C15qijjiosGgMCAv5rIZzxFxAQEBAQEBAQEBAQEBAQEBAQEBAQ8IDw3Oc+F4D/83/+T+X6Rz7yEQCe97zn/dbT4CzTfAvEyclJrrzyyocsjq1bt3L11VcXv6empvj85z/PYx/7WNauXVukw08DwD/8wz9w//33V6695CUvYffu3Xz84x/viaf+PsCzn/1svvzlL3Pdddfxqle9qrDk65dv6K2LKIp49rOfzTe+8Q22bt1aXN+0aRPXXntt5dkXv/jFRFHExRdf3BOu1po9e/b0pC8gIOA/F4LFX0BAQEBAQEBAQEBAQEBAQEBAQEBAwAPCYx7zGF7zmtdwxRVXMDExwWmnncZPfvITPve5z3HWWWfxjGc847eehtNPP51Go8ELXvACzj33XGZmZvjbv/1bVq9ezbZt2x6SOI4//nj+4A/+gJ/+9KesWbO
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Function to encode the image as base64\n",
"def encode_image(image_path: str):\n",
" # check if the image exists\n",
" if not os.path.exists(image_path):\n",
" raise FileNotFoundError(f\"Image file not found: {image_path}\")\n",
" with open(image_path, \"rb\") as image_file:\n",
" return base64.b64encode(image_file.read()).decode('utf-8')\n",
"\n",
"\n",
"# Sample images for testing\n",
"image_dir = \"images\"\n",
"\n",
"# encode all images within the directory\n",
"image_files = os.listdir(image_dir)\n",
"image_data = {}\n",
"for image_file in image_files:\n",
" image_path = os.path.join(image_dir, image_file)\n",
" # encode the image with key as the image file name\n",
" image_data[image_file.split('.')[0]] = encode_image(image_path)\n",
" print(f\"Encoded image: {image_file}\")\n",
"\n",
"\n",
"def display_images(image_data: dict):\n",
" fig, axs = plt.subplots(1, 3, figsize=(18, 6))\n",
" for i, (key, value) in enumerate(image_data.items()):\n",
" img = Image.open(BytesIO(base64.b64decode(value)))\n",
" ax = axs[i]\n",
" ax.imshow(img)\n",
" ax.axis(\"off\")\n",
" ax.set_title(key)\n",
" plt.tight_layout()\n",
" plt.show()\n",
"\n",
"\n",
"display_images(image_data)"
],
"metadata": {
"collapsed": false,
"ExecuteTime": {
"end_time": "2024-04-10T03:50:38.120670Z",
"start_time": "2024-04-10T03:50:37.565555Z"
}
},
"id": "2f2066d40dbbfbe8",
"execution_count": 45
},
{
"cell_type": "markdown",
"source": [
"We have successfully encoded the sample images as base64 strings and displayed them. The customer service assistant will analyze these images to determine the appropriate action based on the package condition.\n",
"\n",
"Let's now define the functions/tools for order processing, such as escalating an order to an agent, refunding an order, and replacing an order. We will create placeholder functions to simulate the processing of these actions based on the identified tools. We will be using Pydantic models to define the structure of the data for order actions.\n"
],
"metadata": {
"collapsed": false
},
"id": "f4b16c5219dff6db"
},
{
"cell_type": "code",
"outputs": [],
"source": [
"MODEL = \"gpt-4-turbo-2024-04-09\"\n",
"\n",
"class Order(BaseModel):\n",
" \"\"\"Represents an order with details such as order ID, customer name, product name, price, status, and delivery date.\"\"\"\n",
" order_id: str = Field(..., description=\"The unique identifier of the order\")\n",
" product_name: str = Field(..., description=\"The name of the product\")\n",
" price: float = Field(..., description=\"The price of the product\")\n",
" status: str = Field(..., description=\"The status of the order\")\n",
" delivery_date: str = Field(..., description=\"The delivery date of the order\")\n",
"# Placeholder functions for order processing\n",
"\n",
"def get_order_details(order_id):\n",
" # Placeholder function to retrieve order details based on the order ID\n",
" return Order(\n",
" order_id=order_id,\n",
" product_name=\"Product X\",\n",
" price=100.0,\n",
" status=\"Delivered\",\n",
" delivery_date=\"2024-04-10\",\n",
" )\n",
"\n",
"def escalate_to_agent(order: Order, message: str):\n",
" # Placeholder function to escalate the order to a human agent\n",
" return f\"Order {order.order_id} has been escalated to an agent with message: `{message}`\"\n",
"\n",
"def refund_order(order: Order):\n",
" # Placeholder function to process a refund for the order\n",
" return f\"Order {order.order_id} has been refunded successfully.\"\n",
"\n",
"def replace_order(order: Order):\n",
" # Placeholder function to replace the order with a new one\n",
" return f\"Order {order.order_id} has been replaced with a new order.\"\n",
"\n",
"class FunctionCallBase(BaseModel):\n",
" rationale: Optional[str] = Field(..., description=\"The reason for the action.\")\n",
" image_description: Optional[str] = Field(\n",
" ..., description=\"The detailed description of the package image.\"\n",
" )\n",
" action: Literal[\"escalate_to_agent\", \"replace_order\", \"refund_order\"]\n",
" message: Optional[str] = Field(\n",
" ...,\n",
" description=\"The message to be escalated to the agent if action is escalate_to_agent\",\n",
" )\n",
" # Placeholder functions to process the action based on the order ID\n",
" def __call__(self, order_id):\n",
" order: Order = get_order_details(order_id=order_id)\n",
" if self.action == \"escalate_to_agent\":\n",
" return escalate_to_agent(order, self.message)\n",
" if self.action == \"replace_order\":\n",
" return replace_order(order)\n",
" if self.action == \"refund_order\":\n",
" return refund_order(order)\n",
"\n",
"class EscalateToAgent(FunctionCallBase):\n",
" \"\"\"Escalate to an agent for further assistance.\"\"\"\n",
" pass\n",
"\n",
"class OrderActionBase(FunctionCallBase):\n",
" pass\n",
"\n",
"class ReplaceOrder(OrderActionBase):\n",
" \"\"\"Tool call to replace an order.\"\"\"\n",
" pass\n",
"\n",
"class RefundOrder(OrderActionBase):\n",
" \"\"\"Tool call to refund an order.\"\"\"\n",
" pass"
],
"metadata": {
"collapsed": false,
"ExecuteTime": {
"end_time": "2024-04-10T03:50:38.135826Z",
"start_time": "2024-04-10T03:50:38.122420Z"
}
},
"id": "115b5a0bf208b054",
"execution_count": 46
},
{
"cell_type": "markdown",
"source": [
"### Simulating user messages and processing the package images\n",
"\n",
"We will simulate user messages containing the package images and process the images using the GPT-4 Turbo with Vision model. The model will identify the appropriate tool call based on the image analysis and the predefined actions for damaged, wet, or normal packages. We will then process the identified action based on the order ID and display the results."
],
"metadata": {
"collapsed": false
},
"id": "5716c84cb57e3fb3"
},
{
"cell_type": "code",
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Processing delivery exception support for different package images...\n",
"\n",
"===================== Simulating user message 1 =====================\n",
"- Tool call: refund_order for provided img: damaged_package\n",
"- Parameters: rationale='The package is visibly damaged with significant tears and crushing, indicating potential harm to the contents.' image_description='The package in the image shows extensive damage, including deep creases and tears in the cardboard. The package is also wrapped with extra tape, suggesting prior attempts to secure it after damage.' action='refund_order' message=None\n",
">> Action result: Order 12345 has been refunded successfully.\n",
"\n",
"===================== Simulating user message 2 =====================\n",
"- Tool call: escalate_to_agent for provided img: normal_package\n",
"- Parameters: rationale='The package appears normal and not damaged, requiring further assistance for any potential issues not visible in the image.' image_description='A cardboard box on a wooden floor, appearing intact and undamaged, with no visible signs of wear, tear, or wetness.' action='escalate_to_agent' message='Please review this package for any issues not visible in the image. The package appears normal and undamaged.'\n",
">> Action result: Order 12345 has been escalated to an agent with message: `Please review this package for any issues not visible in the image. The package appears normal and undamaged.`\n",
"\n",
"===================== Simulating user message 3 =====================\n",
"- Tool call: replace_order for provided img: wet_package\n",
"- Parameters: rationale='The package appears wet, which may have compromised the contents, especially since it is labeled as fragile.' image_description=\"The package in the image shows significant wetness on the top surface, indicating potential water damage. The box is labeled 'FRAGILE', which suggests that the contents are delicate and may be more susceptible to damage from moisture.\" action='replace_order' message=None\n",
">> Action result: Order 12345 has been replaced with a new order.\n"
]
}
],
"source": [
"# extract the tool call from the response\n",
"ORDER_ID = \"12345\" # Placeholder order ID for testing\n",
"INSTRUCTION_PROMPT = \"You are a customer service assistant for a delivery service, equipped to analyze images of packages. If a package appears damaged in the image, automatically process a refund according to policy. If the package looks wet, initiate a replacement. If the package appears normal and not damaged, escalate to agent. For any other issues or unclear images, escalate to agent. You must always use tools!\"\n",
"\n",
"def delivery_exception_support_handler(test_image: str):\n",
" payload = {\n",
" \"model\": MODEL,\n",
" \"response_model\": Iterable[RefundOrder | ReplaceOrder | EscalateToAgent],\n",
" \"tool_choice\": \"auto\", # automatically select the tool based on the context\n",
" \"temperature\": 0.0, # for less diversity in responses\n",
" \"seed\": 123, # Set a seed for reproducibility\n",
" }\n",
" payload[\"messages\"] = [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": INSTRUCTION_PROMPT,\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\n",
" \"type\": \"image_url\",\n",
" \"image_url\": {\n",
" \"url\": f\"data:image/jpeg;base64,{image_data[test_image]}\"\n",
" }\n",
" },\n",
" ],\n",
" }\n",
" ]\n",
" function_calls = instructor.from_openai(\n",
" OpenAI(), mode=instructor.Mode.PARALLEL_TOOLS\n",
" ).chat.completions.create(**payload)\n",
" for tool in function_calls:\n",
" print(f\"- Tool call: {tool.action} for provided img: {test_image}\")\n",
" print(f\"- Parameters: {tool}\")\n",
" print(f\">> Action result: {tool(ORDER_ID)}\")\n",
" return tool\n",
"\n",
"\n",
"print(\"Processing delivery exception support for different package images...\")\n",
"\n",
"print(\"\\n===================== Simulating user message 1 =====================\")\n",
"assert delivery_exception_support_handler(\"damaged_package\").action == \"refund_order\"\n",
"\n",
"print(\"\\n===================== Simulating user message 2 =====================\")\n",
"assert delivery_exception_support_handler(\"normal_package\").action == \"escalate_to_agent\"\n",
"\n",
"print(\"\\n===================== Simulating user message 3 =====================\")\n",
"assert delivery_exception_support_handler(\"wet_package\").action == \"replace_order\""
],
"metadata": {
"collapsed": false,
"ExecuteTime": {
"end_time": "2024-04-10T03:50:53.927476Z",
"start_time": "2024-04-10T03:50:38.136857Z"
}
},
"id": "dacd602d9e2728d9",
"execution_count": 47
},
{
"cell_type": "markdown",
"source": [
"## 2. Analyzing an organizational chart to extract employee information\n",
"\n",
"For the second example, we will analyze an organizational chart image to extract employee information, such as employee names, roles, managers, and manager roles. We will use GPT-4 Turbo with Vision to process the organizational chart image and extract structured data about the employees in the organization. Indeed, function calling lets us go beyond OCR to actually deduce and translate hierarchical relationships within the chart.\n",
"\n",
"We will start with a sample organizational chart in PDF format that we want to analyze and convert the first page of the PDF to a JPEG image for analysis."
],
"metadata": {
"collapsed": false
},
"id": "78604bb7ff61b472"
},
{
"cell_type": "code",
"outputs": [
{
"data": {
"text/plain": "<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=768x576>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAwAAAAJACAIAAAC1zJYBAAEAAElEQVR4Aez9B6BtSXYW9p9873vdEyUkkFBAQgEEiCCQBMgoEEzOmGByNP5jDDLGNgjBDDYYYxNtgm1MFiaKIEtgkDQzRiBAIiMUAIFQQhrNTKf37j3p//vW2mff8+57r6d7pif17OrX+9auWrVq1Tr77PWdVauq5l/4h/7U8ji/Xu9nh93ueDgulk/sV/PF6vq4vVisDsvjYrebL2bz1XK5318f9/PDxXI5ny2Oi9XicNgpnh8X0m4/W8yOauazw/F4nM3ne80W89nV1Wqz1Gy1WM8Py/nusF7NZvP9YT07Xu+W85W2R23ms9nssF4s91rOj4fDYbNcHXb742x/ebG6vn7uYn93vzwetZ3N8JcwOy5Xd69Xu9318g6R9ofjTsPV8vKwOy4PYTRfRJJiPlvO5uv54v7F/Tvb5f643C+ItMdQ7+trAmxIdVzsFwtFy5nRKFscrmez5W62ms2Xy8VxtdAu5TNybRYroyXDfr48llh0sYr0x+OexlQvF+5kpdl6narQzQx5tsQ/aX/ILcroATnNRhULpAvKpfXZ8VBKIVgaUukRTaeIoklJq3H4z+eEd10cwl5K+emKOLVjaj7DLUk1rJvi09mR9lZmkP9W6XQ7aWDSwKSBSQOTBt4nNLDaAxWDiZ7DHyDFar5ifC8XG+DhCLMs10x+W8TlYrlbKFnOD8fVXtqxxoDO9gjsHI/z1YGxPUAAs8PsADDMmP8Vs81Gz9luJLP5AagJLYSwBnGCZMig3eV6k352uwWrvz3sQJTVkrHfHo6H+er+xSzoZrcDYi7my9WCkLPDnrmHdLAN5Rz+WOwCAyCbY9AAvJCuVUYmvR1gnZ16OAlegmtmM4wgDMM+EppcUCDclOvBoDYGaxR7lQRHStJCOQEhURwsE+YZW7CUvwdkgVAlFwEIovCwC9Ap4dC7D+KZ6zeYrtoXG1lEIEvBjxQFZc0jV/3D6vmByfvEYzcJOWlg0sCkgUkDkwbesxoAJAJFIBpWls+Avd0fDqDNmrfjcNwvD/CD4u2eE+fIUbEJnIkPgpODh0Ri6jk3DkuOHy4b1hoKAUaw2h33u8XsEhiBVcorES66C2zZbzerNT6BU1DUnmdltt1uV8fFegOMsPfollv4arZcXvDCxO+hdgEP7fwXYeY8SnMNuW7glvlqzSkE54BFMItOSQjEEClwJAACADrOtwsenYWRXWV0Mwhqvzhu8a2PYgkxYY6dKg3KY8JdFG7xKsFDFGCUu7hu4kEK4tMRuBUMxosUP01cOnpEHE1oQGEZgRxFRSg4KbpNwySFQTsPohvun/wXLkknyoEuei2UFRj2mBQUGj0M1UFU+QDTbLhWVcpJcOpClZKxx5BOadLApIFJA5MGJg28jDSw2oILrKeZGBafh4e5ZCALjrDrTCBD7spKB1yYuAFnQB9mPzMvcUeY/wEejss1GBCTz86z/iDIHoyIjea4yVQUo4u3RqZ3+Hb4nQCXGWQVzw5CTpMVSMQxI8UqBxXtM/2jrakyjPZq49AxnbTk6tHfkf9GXtfy82N8SDHxprGIVZBGcQGXfGjgxhaC4rIiXdxYZv/SFxR1x5Di34nPKFNchJwft8vlaqtzLZUrK5xS91BP9Rr4AsUFb5FQPYXoPeMswTEtlBMwcdKi8WSIBpxmBYBKzwVAgpdMsDVoC8dKjUXS6oRRHpc/tei/+ezkdFMqrWs+E2NIeTKnquG+S6frpIFJA5MGJg1MGnhZa8D8S9ALi33wX5thNtPNwVxSJnd2ux3kUvEh3CDHw2rFB7LYAxiFFOazLewBLYjnYfnhFGYdSop3aLNcXB7hjQMOATGFkIJVwIP9dUKO1pkgE7wTn9P19TUB7jxxR0xPpp8QHWcXS90t9rutiTj4Z3CpLOb8TwnGIXkCh9j1XZqAOvDGAbja7GbXwVvBGpnhgkI6D4rxVfEL8SytDwEGnFT6fdaQDkAeD0/ib4xYG4hliUoF340xhryRTToiPy9ZkFEgRfw3iT0K6MsYB6QCNBWygAk9SKlyG/qU1l0wSrmCApR0k/JTbQj1mKbdC5mKnYJHp5Dh54r00SQvrPTtdfTCuExUkwYmDUwamDQwaeC9UgOibICbuDti7mvahs0NCMgcWIyg2a8Y7Lht2paDQWVhIQouISTuQIpheAlNZoLjiAGLBPLCDxV4e+C8gZTi4OCR2a+X6/32WrPMkyWgeS4EabPcXO1314Ka+VFALLHK4oqAseNuSyCOH8LNBAMJMwZ3EsOTGTUsk1KKeZoG1cXfI0WOMySgEBezYOpNhwFqW/NY0EKDq8zmGdURGirssbiTeOYMJpAl4Us6zLRcwphNH1YQslDxEiAXPYYGi1Ja0E0m0QQRlTAqAwkL0hTTdrAl6z8pn0PFTo8czzMFv84Lnjcf3NSpxEzW4ILAztJ4WyM8q5iykwYmDUwamDQwaeDlq4GVIBqByKy61V2JJN4LiLGQanY1T7hMJo/8Lx0SayOMZXl9LWpa8LPElF+Idqk4mcw+xatiQVdmiiS3IMlcPHNKYIVAH36QQKAZmAMRFDzg4uFYisvEUrJtYoIWa1NtQRLik2Kwj+sLHp2KQ0qciiht0EO0zV680nZt9k6H/gWZQTLkP86uTVEp0irCBHsUTQX3XAM0xnMwdCMHZyxmE7MdyWE+s2I8ORX5E3bWgMWfk5k9qCWQwtip4iLCZ9UXJZW7iARCqciVCCSd0Voyy3jACt+AmYGQXEYRaRAq2KrzQT8ZbFBocYgnLY21KWxUXMshpCAOplRI6ezxSfNb6dR/io11FIZEEwa6pavpdtLApIFJA5MGXq4aMAUlEphhz6r1TFztd7vD3mqug+jeQJTFOs6fmSXoPB6HFR8LiCLIBTgx/zNbJoZaGLFJontZJg4DJKo4Rn0pnHgtWDp2PJNRwUlmq6y0D1CBZDar5Xa/ZeQzs2aR1PoSAFod17wzZsTYdbNt6La7vVkuVJBBgnsW8y2Est+tDjPr08RFZwouM3K8SFaei9nhbdke3Mavk/Bo4xPdFMgQBLK4NhaL1A8zfYN1xn693K/jPDJkvIIQ4sMCDYJxAJ40rQ6yYouOaCydpgMAJQBIMjop03/BNNjRT9aVgSBUWmBJfXATgnCEnWAp+ijEU4gouqpHjaApqEtddVUV3bay8Rg1zdltZx971etj6x6sOO/owZrpbtLApIFJA5MGJg28z2tgJfbFeqYdgFHLssTLXHByMObX28XmEgzgvlnZyee4v9qJ0Vmy6YKW7zLHT2TmK2AiYcz7tXAdK6Pcl8/BzBVks726t9zaYcfuOlvzOhaHH46mxGJbV5e77XZvnms9Ewh0AFh2+2tIYLu4D/LAIAKiQQTrvQIUMh8FU8ARQWvxOXHTbGb3kSU6xnY9e5vm7A735hvwQkebDfQBuaws5jeBZuEY304CgYCr5fXcBBzYkVjveFsWi+us7TcZJx4pZBkmaHPcgFfX5sDi24FMtiTM3jw2NIKvtMV4xx20AmvgJuBnvrtvTVxYEwOkJFrQzUVP1cFM8bXhFRRSLiJ/5fMPXT9NhVECr9wSOnU1ixZvTSBXApEKYMFS+knKZGPwWvlwRj7hQRUNthKA3SmwKSkNC+t18c18Weqqvisec8XvgSaPIZuKJw1MGpg0MGlg0sB7owZsQ2PWxnKo2MMgh3KUuFnZGJCR5uhIYvGtPN8vBPkuLirGWVROQaGltecVnAOqhPK6FoBz8mjourp7J/NQGLDaQAXvEDCRHYC4SLib9msYB20hG5YZaBE2ZDk74GCPovQel0l5amJyAwKCHGonwZoZMgOmSe2mU4AgVjnumQCPWPrgIeigbsRaJ5aJKyvSBPTgv9tnRZe5qsYaIrK10tj/iutOp4UJlEhiusX0COxO8HUSHxP2aWZ0eiunjpVupIfydJc47ZsUNaRFkFd2bwzLIU7bxxB4GNCSDyS9umn5McjM14lTak+jQ5GbStV4vLspDA0pB/QzlI9/HtlqrJ0ykwYmDUwamDQwaeBlpoHacSe
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAJAAwADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDpgMjNOHXFIo4p4546UmdaY5eOlSAcUwDingf/AK6kq44DGBTx2IpoHPapAKljTHDrTwB1po+6KeOOKllXHU5fSkApwHPWpaHccBipB9aaOntThioaHckAxUyduKiWplBFS0S2TpVhO1VkNWEpWM5FhKmXtUC1OtWkZMmU1IKiWpBWsTNkgp1MFOFaIkcKKQUtUIKKKKYBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAeUqKfjC0gA/wD108cgUmb3FWnimjjingelIq45R7U8Cmin4GBipHccBx7U8cU0ehp4GOnWkykxwz+FOHSk7U4DvU2HccKeDTQDTx2qWh3HjNTL+tQrUq9RU2FcmSrCGq6VOppWIZZWpkqulTrVJGbJlqUVCtSrWiIZIKcKaKcK0RItLSClpiCiiimAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHlwHNOwMjtSgU8DFIu4gApwHegDingUDTAdjTl/wAigdcCnAce9SUmOA5pQOaAP0pQMnpQO44etO6Ug9qcKloakOHtTh9O9NHWndcUrDuSA81InrUannFSIcVNguTrU6nkVXX1qdDRYTZOlTrVdDzU6dqaRDJ1qVahU1KtWiWSCnimCniqRItLSClqhBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHmmOtOUZFAGacF/CgVwAPpTulAFPAOPagLgB/8AWpQOePzoA/Gnhe1Kw7iDkU8DHNIBinAUWKuHU04elGOlGPSlYaYtOHUGm9BTunNKxVx4/UVIpzUQJ6VKvalYdyZKnWq61OnalYLlhOKnWoEqZKdiSdalWoVqZaZJIKcKaKcKoQtOpop1MQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQB5yB0p4HPNKq04Lg0jK4gHFKBTgOtOC0wuIq8+9LjrT9vrTgOtMdxgFLt+angUYxQNMbgkc0vOO1KRR9aLDuN+tKKOetHelYpMcDipVIqEY4qVaTRVydTUyGoFNTJSsO5ZSp0qsnarCdqLCuTqalWoVqZaYEgp4pgp4piHUtNFOFAgoopCQASTgDuaAFoqtBqFldSNHb3lvM69VjkViPwBqzQAUUUUAFFFFABRRRQAUUUUAFFUNW1nTdCsje6rfQWdsGC+ZM4UFj0A9T7e1Wba5gvLaK5tpkmglUPHJGwZXU9CCOooAmooooAKKKKACiis7V9d0nw/bJc6vqNtZQu+xHnkChm9BnrQBo0UyKSOeJJYnWSN1DI6HIYHkEEdRT6ACiiigAooooAKKKZJNHEAZJETPTcwGaAH0UgYMoYEEHkEd6WgAoqsNQsmuPs63luZ848oSLu/LOas0AFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBwQWlCdxTwvX2p4X2pIwGBKeF5p6rTwlUBFtFO21IExTtnNMZFt9aTbU2OKTZTC5CRSEVKVppFA7kZGKM07HNIRRYdwA49qkXrxUYHSpFpWKTJV6ip0OcVAtTJSsVcsIanSq6VOlKw7lhamWoUqVaAJRTxTBTxQIcKUU2nCgAr5o8Wa7r/wAWviNJ4T0W7MGkwSPGAGIjZUOGlfH3ueg9x3JNfS9fMvwFddO+KeqWN4Ql29pNCobqXWRCy/XCsfwoAueIfgDf+HtGk1fQ9ekub2zQzGMQ+SzBRklGDHBHYd/Wu4+C3xCuvFmgXdlq8pl1HTApMx+9NEc4J9WBBBPfjuTXp19dQWNhcXd0yrbwRNJKzdAoGT+lfOH7O0Era/r9woPkpp+xvQMzAr+itQB6db/HLwXcadeXvn3kcdrsBWSDDSM2cKgB5Pyn0HFdHpHjvRNW8Ht4p82Sz0pd26S7TYRtO08AnPPAxnJ4618+/AnwrpHiXxNfvrFol3FZ24eOCQZQszYyR3wM8Hjmuy/aCWPRfBug6LpsCWuntcu/kwrtQbRkDA7Zdjj1oA6/Sfjf4M1jWU02K4uoHkbZFNcQ7I3bsAckjPuBWv4O+JOg+Obq5ttH+1eZbIJJPPiCDBOOOTXOeFPhx4Vu/hJYRz6ZZvNe6ctxLetEDKsjpu3B8ZG0nj6fWuD/AGbCBr+uknA+yp/6FQB6l4o+L/hLwnqL6fd3U9zeRnEsNpHvMZ9GJIGfbORV7wj8SvDPjWV7fSrtxdou821wmyTb6jsfwJxXkcPiXwJYeL7yXwj4H1DxJqDF98jM0qMS2WdAwc8n+LA6+9c74Znul/aCsZpdIGiTS3mXsEGBCHj6dB1Bz0HXoKAPe9V+Jvh3RvF8fhm9e4S+coN/ljyl3DIJYngY61if8L58Df2r9i+1Xfl7tv2v7OfK+vXdj3215d8TtPh1X9oKz065yYLueyglAOCVfap/Qmuj+PHg3w9ovg7T9Q0rSbWxuEvVty1vGE3oUc/NjqcqOTz1oA7L4wN4Su/B1i/iWa9+wy3KtbT6ftZt5RiDzwVK5/Sr9j4n8M+DvhfperW4vBoaRJHBlN0pDE4LDPUnJNeReN5pJ/2dPBjyMWYXITJ9FWUAfkBXr3w902w1b4S+H7XUrK3vLc
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Function to convert a single page PDF page to a JPEG image\n",
"def convert_pdf_page_to_jpg(pdf_path: str, output_path: str, page_number=0):\n",
" if not os.path.exists(pdf_path):\n",
" raise FileNotFoundError(f\"PDF file not found: {pdf_path}\")\n",
" doc = fitz.open(pdf_path)\n",
" page = doc.load_page(page_number) # 0 is the first page\n",
" pix = page.get_pixmap()\n",
" # Save the pixmap as a JPEG\n",
" pix.save(output_path)\n",
"\n",
"\n",
"def display_img_local(image_path: str):\n",
" img = Image.open(image_path)\n",
" display(img)\n",
"\n",
"\n",
"pdf_path = 'data/org-chart-sample.pdf'\n",
"output_path = 'org-chart-sample.jpg'\n",
"\n",
"convert_pdf_page_to_jpg(pdf_path, output_path)\n",
"display_img_local(output_path)"
],
"metadata": {
"collapsed": false,
"ExecuteTime": {
"end_time": "2024-04-10T03:50:54.033536Z",
"start_time": "2024-04-10T03:50:53.929732Z"
}
},
"id": "c3fc6f64b2a2f8e6",
"execution_count": 48
},
{
"cell_type": "markdown",
"source": [
"The organizational chart image has been successfully extracted from the PDF file and displayed. Let's now define a function to analyze the organizational chart image using the new GPT4 Turbo with Vision. The function will extract information about the employees, their roles, and their managers from the image. We will use function/tool calling to specify the input parameters for the organizational structure, such as the employee name, role, and manager's name and role. We will use Pydantic models to define the structure of the data.\n"
],
"metadata": {
"collapsed": false
},
"id": "5f6b4299c99721bc"
},
{
"cell_type": "code",
"outputs": [],
"source": [
"base64_img = encode_image(output_path)\n",
"\n",
"class RoleEnum(str, Enum):\n",
" \"\"\"Defines possible roles within an organization.\"\"\"\n",
" CEO = \"CEO\"\n",
" CTO = \"CTO\"\n",
" CFO = \"CFO\"\n",
" COO = \"COO\"\n",
" EMPLOYEE = \"Employee\"\n",
" MANAGER = \"Manager\"\n",
" INTERN = \"Intern\"\n",
" OTHER = \"Other\"\n",
"\n",
"class Employee(BaseModel):\n",
" \"\"\"Represents an employee, including their name, role, and optional manager information.\"\"\"\n",
" employee_name: str = Field(..., description=\"The name of the employee\")\n",
" role: RoleEnum = Field(..., description=\"The role of the employee\")\n",
" manager_name: Optional[str] = Field(None, description=\"The manager's name, if applicable\")\n",
" manager_role: Optional[RoleEnum] = Field(None, description=\"The manager's role, if applicable\")\n",
"\n",
"\n",
"class EmployeeList(BaseModel):\n",
" \"\"\"A list of employees within the organizational structure.\"\"\"\n",
" employees: List[Employee] = Field(..., description=\"A list of employees\")\n",
"\n",
"def parse_orgchart(base64_img: str) -> EmployeeList:\n",
" response = instructor.from_openai(OpenAI()).chat.completions.create(\n",
" model='gpt-4-turbo',\n",
" response_model=EmployeeList,\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": 'Analyze the given organizational chart and very carefully extract the information.',\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\n",
" \"type\": \"image_url\",\n",
" \"image_url\": {\n",
" \"url\": f\"data:image/jpeg;base64,{base64_img}\"\n",
" }\n",
" },\n",
" ],\n",
" }\n",
" ],\n",
" )\n",
" return response"
],
"metadata": {
"collapsed": false,
"ExecuteTime": {
"end_time": "2024-04-10T03:50:54.043273Z",
"start_time": "2024-04-10T03:50:54.034475Z"
}
},
"id": "b11469486fc95c23",
"execution_count": 49
},
{
"cell_type": "markdown",
"source": [
"Now, we will define a function to parse the response from GPT-4 Turbo with Vision and extract the employee data. We will tabulate the extracted data for easy visualization. Please note that the accuracy of the extracted data may vary based on the complexity and clarity of the input image."
],
"metadata": {
"collapsed": false
},
"id": "73619ee88ed5cd2"
},
{
"cell_type": "code",
"outputs": [
{
"data": {
"text/plain": " employee_name role manager_name manager_role\n0 Juliana Silva CEO None None\n1 Kim Chun Hei CFO Juliana Silva CEO\n2 Chad Gibbons CTO Juliana Silva CEO\n3 Chiaki Sato COO Juliana Silva CEO\n4 Cahaya Dewi Manager Kim Chun Hei CFO\n5 Shawn Garcia Manager Chad Gibbons CTO\n6 Aaron Loeb Manager Chiaki Sato COO\n7 Drew Feig Employee Cahaya Dewi Manager\n8 Richard Sanchez Employee Cahaya Dewi Manager\n9 Sacha Dubois Intern Cahaya Dewi Manager\n10 Olivia Wilson Employee Shawn Garcia Manager\n11 Matt Zhang Intern Shawn Garcia Manager\n12 Avery Davis Employee Aaron Loeb Manager\n13 Harper Russo Employee Aaron Loeb Manager\n14 Taylor Alonso Intern Aaron Loeb Manager",
"text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>employee_name</th>\n <th>role</th>\n <th>manager_name</th>\n <th>manager_role</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>0</th>\n <td>Juliana Silva</td>\n <td>CEO</td>\n <td>None</td>\n <td>None</td>\n </tr>\n <tr>\n <th>1</th>\n <td>Kim Chun Hei</td>\n <td>CFO</td>\n <td>Juliana Silva</td>\n <td>CEO</td>\n </tr>\n <tr>\n <th>2</th>\n <td>Chad Gibbons</td>\n <td>CTO</td>\n <td>Juliana Silva</td>\n <td>CEO</td>\n </tr>\n <tr>\n <th>3</th>\n <td>Chiaki Sato</td>\n <td>COO</td>\n <td>Juliana Silva</td>\n <td>CEO</td>\n </tr>\n <tr>\n <th>4</th>\n <td>Cahaya Dewi</td>\n <td>Manager</td>\n <td>Kim Chun Hei</td>\n <td>CFO</td>\n </tr>\n <tr>\n <th>5</th>\n <td>Shawn Garcia</td>\n <td>Manager</td>\n <td>Chad Gibbons</td>\n <td>CTO</td>\n </tr>\n <tr>\n <th>6</th>\n <td>Aaron Loeb</td>\n <td>Manager</td>\n <td>Chiaki Sato</td>\n <td>COO</td>\n </tr>\n <tr>\n <th>7</th>\n <td>Drew Feig</td>\n <td>Employee</td>\n <td>Cahaya Dewi</td>\n <td>Manager</td>\n </tr>\n <tr>\n <th>8</th>\n <td>Richard Sanchez</td>\n <td>Employee</td>\n <td>Cahaya Dewi</td>\n <td>Manager</td>\n </tr>\n <tr>\n <th>9</th>\n <td>Sacha Dubois</td>\n <td>Intern</td>\n <td>Cahaya Dewi</td>\n <td>Manager</td>\n </tr>\n <tr>\n <th>10</th>\n <td>Olivia Wilson</td>\n <td>Employee</td>\n <td>Shawn Garcia</td>\n <td>Manager</td>\n </tr>\n <tr>\n <th>11</th>\n <td>Matt Zhang</td>\n <td>Intern</td>\n <td>Shawn Garcia</td>\n <td>Manager</td>\n </tr>\n <tr>\n <th>12</th>\n <td>Avery Davis</td>\n <td>Employee</td>\n <td>Aaron Loeb</td>\n <td>Manager</td>\n </tr>\n <tr>\n <th>13</th>\n <td>Harper Russo</td>\n <td>Employee</td>\n <td>Aaron Loeb</td>\n <td>Manager</td>\n </tr>\n <tr>\n <th>14</th>\n <td>Taylor Alonso</td>\n <td>Intern</td>\n <td>Aaron Loeb</td>\n <td>Manager</td>\n </tr>\n </tbody>\n</table>\n</div>"
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# call the functions to analyze the organizational chart and parse the response\n",
"result = parse_orgchart(base64_img)\n",
"\n",
"# tabulate the extracted data\n",
"df = pd.DataFrame([{\n",
" 'employee_name': employee.employee_name,\n",
" 'role': employee.role.value,\n",
" 'manager_name': employee.manager_name,\n",
" 'manager_role': employee.manager_role.value if employee.manager_role else None\n",
"} for employee in result.employees])\n",
"\n",
"display(df)"
],
"metadata": {
"collapsed": false,
"ExecuteTime": {
"end_time": "2024-04-10T03:51:03.255172Z",
"start_time": "2024-04-10T03:50:54.044659Z"
}
},
"id": "962c5ea24b8a9dc9",
"execution_count": 50
},
{
"cell_type": "markdown",
"source": [
"The extracted data from the organizational chart has been successfully parsed and displayed in a DataFrame. This approach allows us to leverage GPT-4 Turbo with Vision capabilities to extract structured information from images, such as organizational charts and diagrams, and process the data for further analysis. By using function calling, we can extend the functionality of multimodal models to perform specific tasks or call external functions."
],
"metadata": {
"collapsed": false
},
"id": "c55b5fbf97ae7ba8"
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue Copy Permalink