"# Orchestrating Agents: Routines and Handoffs\n",
"\n",
"When working with language models, quite often all you need for solid performance is a good prompt and the right tools. However, when dealing with many unique flows, things may get hairy. This cookbook will walk through one way to tackle this.\n",
"\n",
"We'll introduce the notion of **routines** and **handoffs**, then walk through the implementation and show how they can be used to orchestrate multiple agents in a simple, powerful, and controllable way.\n",
"\n",
"Finally, we provide a sample repo, [Swarm](https://github.com/openai/swarm), that implements these ideas along with examples."
"The notion of a \"routine\" is not strictly defined, and instead meant to capture the idea of a set of steps. Conretely, let's define a routine to be a list of instructions in natural langauge (which we'll represent with a system prompt), along with the tools necessary to complete them.\n",
"Let's take a look at an example. Below, we've defined a routine for a customer service agent instructing it to triage the user issue, then either suggest a fix or provide a refund. We've also defined the necessary functions `execute_refund` and `look_up_item`. We can call this a customer service routine, agent, assistant, etc –however the idea itself is the same: a set of steps and the tools to execute them."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"# Customer Service Routine\n",
"\n",
"system_message = (\n",
" \"You are a customer support agent for ACME Inc.\"\n",
" \"Always answer in a sentence or less.\"\n",
" \"Follow the following routine with the user:\"\n",
" \"1. First, ask probing questions and understand the user's problem deeper.\\n\"\n",
" \" - unless the user has already provided a reason.\\n\"\n",
" \"2. Propose a fix (make one up).\\n\"\n",
" \"3. ONLY if not satesfied, offer a refund.\\n\"\n",
" \"4. If accepted, search for the ID and then execute refund.\"\n",
" \"\"\n",
")\n",
"\n",
"def look_up_item(search_query):\n",
" \"\"\"Use to find item ID.\n",
" Search query can be a description or keywords.\"\"\"\n",
"\n",
" # return hard-coded item ID - in reality would be a lookup\n",
"The main power of routines is their simplicity and robustness. Notice that these instructions contain conditionals much like a state machine or branching in code. LLMs can actually handle these cases quite robustly for small and medium sized routine, with the added benefit of having \"soft\" adherance –the LLM can naturally steer the conversation without getting stuck in dead-ends.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Executing Routines\n",
"\n",
"To execute a routine, let's implement a simple loop that:\n",
"As you can see, this currently ignores function calls, so let's add that.\n",
"\n",
"Models require functions to be formatted as a function schema. For convenience, we can define a helper function that turns python functions into the corresponding function schema."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"import inspect\n",
"\n",
"def function_to_schema(func) -> dict:\n",
" type_map = {\n",
" str: \"string\",\n",
" int: \"integer\",\n",
" float: \"number\",\n",
" bool: \"boolean\",\n",
" list: \"array\",\n",
" dict: \"object\",\n",
" type(None): \"null\",\n",
" }\n",
"\n",
" try:\n",
" signature = inspect.signature(func)\n",
" except ValueError as e:\n",
" raise ValueError(\n",
" f\"Failed to get signature for function {func.__name__}: {str(e)}\"\n",
"tool_schemas = [function_to_schema(tool) for tool in tools]\n",
"\n",
"response = client.chat.completions.create(\n",
" model=\"gpt-4o-mini\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Look up the black boot.\"}],\n",
" tools=tool_schemas,\n",
" )\n",
"message = response.choices[0].message\n",
"\n",
"message.tool_calls[0].function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, when the model calls a tool we need to execute the corresponding function and provide the result back to the model.\n",
"\n",
"We can do this by mapping the name of the tool to the python function in a `tool_map`, then looking it up in `execute_tool_call` and calling it. Finally we add the result to the conversation."
" # call corresponding function with provided arguments\n",
" return tools_map[name](**args)\n",
"\n",
"for tool_call in message.tool_calls:\n",
" result = execute_tool_call(tool_call, tools_map)\n",
"\n",
" # add result back to conversation \n",
" result_message = {\n",
" \"role\": \"tool\",\n",
" \"tool_call_id\": tool_call.id,\n",
" \"content\": result,\n",
" }\n",
" messages.append(result_message)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In practice, we'll also want to let the model use the result to produce another response. That response might _also_ contain a tool call, so we can just run this in a loop until there are no more tool calls.\n",
"\n",
"If we put everything together, it will look something like this:"
"Now that we have a routine, let's say we want to add more steps and more tools. We can up to a point, but eventually if we try growing the routine with too many different tasks it may start to struggle. This is where we can leverage the notion of multiple routines – given a user request, we can load the right routine with the appropriate steps and tools to address it.\n",
"\n",
"Dynamically swapping system instructions and tools may seem daunting. However, if we view \"routines\" as \"agents\", then this notion of **handoffs** allow us to represent these swaps simply –as one agent handing off a conversation to another."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Handoffs\n",
"\n",
"Let's define a **handoff** as an agent (or routine) handing off an active conversation to another agent, much like when you get transfered to someone else on a phone call. Except in this case, the agents have complete knowledge of your prior conversation!\n",
"\n",
"To see handoffs in action, let's start by defining a basic class for an Agent."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"class Agent(BaseModel):\n",
" name: str = \"Agent\"\n",
" model: str = \"gpt-4o-mini\"\n",
" instructions: str = \"You are a helpful Agent\"\n",
" tools: list = []"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now to make our code support it, we can change `run_full_turn` take an `Agent` instead of separate `system_message` and `tools`:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"def run_full_turn(agent, messages):\n",
"\n",
" num_init_messages = len(messages)\n",
" messages = messages.copy()\n",
"\n",
" while True:\n",
"\n",
" # turn python functions into tools and save a reverse map\n",
" tool_schemas = [function_to_schema(tool) for tool in agent.tools]\n",
" tools_map = {tool.__name__: tool for tool in agent.tools}\n",
"Great! But we did the handoff manually here –we want the agents themselves to decide when to perform a handoff. A simple, but surprisingly effective way to do this is by giving them a `transfer_to_XXX` function, where `XXX` is some agent. The model is smart enough to know to call this function when it makes sense to make a handoff!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Handoff Functions\n",
"\n",
"Now that agent can express the _intent_ to make a handoff, we must make it actually happen. There's many ways to do this, but there's one particularly clean way.\n",
"\n",
"For the agent functions we've defined so far, like `execute_refund` or `place_order` they return a string, which will be provided to the model. What if instead, we return an `Agent` object to indate which agent we want to transfer to? Like so:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"refund_agent = Agent(\n",
" name=\"Refund Agent\",\n",
" instructions=\"You are a refund agent. Help the user with refunds.\",\n",
" tools=[execute_refund],\n",
")\n",
"\n",
"def transfer_to_refunds():\n",
" return refund_agent\n",
"\n",
"sales_assistant = Agent(\n",
" name=\"Sales Assistant\",\n",
" instructions=\"You are a sales assistant. Sell the user a product.\",\n",
" tools=[place_order],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can then update our code to check the return type of a function response, and if it's an `Agent`, update the agent in use! Additionally, now `run_full_turn` will need to return the latest agent in use in case there are handoffs. (We can do this in a `Response` class to keep things neat.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class Response(BaseModel):\n",
" agent: Optional[Agent]\n",
" messages: list"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now for the updated `run_full_turn`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def run_full_turn(agent, messages):\n",
"\n",
" current_agent = agent\n",
" num_init_messages = len(messages)\n",
" messages = messages.copy()\n",
"\n",
" while True:\n",
"\n",
" # turn python functions into tools and save a reverse map\n",
" tool_schemas = [function_to_schema(tool) for tool in current_agent.tools]\n",
" tools = {tool.__name__: tool for tool in current_agent.tools}\n",
"As a proof of concept, we've packaged these ideas into a sample library called [Swarm](https://github.com/openai/swarm). It is meant as an example only, and should not be directly used in production. However, feel free to take the ideas and code to build your own!"
