Function Calling Tutorial: Connect an LLM to Tools and APIs

Overview

In a function-calling workflow, your application—not the model—owns the actual operation. The model receives a list of available tools and decides whether one is relevant. If it requests a tool, your server validates the request, performs the operation, and sends a structured result back to the model. The model can then explain the result or request another tool.

A typical sequence looks like this:

1. The user sends a request such as, “Check the status of my order.”
2. Your application sends the conversation and tool definitions to the LLM.
3. The model returns a tool call with a name and arguments.
4. Your server checks the tool name, validates the arguments, and enforces permissions.
5. The server calls the relevant internal service or external API.
6. The server returns a structured tool result to the model.
7. The model produces a user-facing answer or requests another approved tool.

This separation is important. The model can propose an action, but it should not receive unrestricted credentials, directly execute arbitrary code, or decide which users may access sensitive data. Treat LLM tool calling as an orchestration layer around ordinary application code, not as a replacement for authentication, authorization, validation, or business logic.

Step-by-step workflow

1. Start with a narrow tool contract

Define one tool for one clear capability. A tool such as get_order_status is easier to secure and test than a general-purpose run_database_query function. Give each tool a short description that explains when it should be used and what it returns.

Define arguments with an explicit schema. Include required fields, accepted types, allowed values, and useful limits. For example, an order-status tool might accept an order identifier but not an arbitrary SQL statement:

{
  "name": "get_order_status",
  "description": "Look up the current status of an order the authenticated user can access.",
  "parameters": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The order identifier supplied by the user"
      }
    },
    "required": ["order_id"],
    "additionalProperties": false
  }
}

The exact schema format varies by provider, but the design principle is stable: make the contract explicit and keep it smaller than the underlying API whenever possible.

2. Separate model-visible data from server-only data

Tool arguments should contain only information the model needs to select and parameterize the action. Add server-controlled values after the tool call is received. For example, do not ask the model to provide a customer ID when your authenticated session already identifies the customer. Your server should attach that identity while executing the function.

The same rule applies to credentials, tenant identifiers, permission scopes, internal URLs, and audit metadata. Keep secrets in the server environment or a secret manager. The model should never be the authority for access control.

3. Implement a controlled execution loop

Your application needs a loop that can handle either a normal response or one or more tool calls. A provider-neutral outline looks like this:

messages = [system_message, user_message]

for step in range(MAX_STEPS):
    response = call_model(messages, tools=approved_tools)

    if response.is_final_text:
        return response.text

    for requested_call in response.tool_calls:
        validate_tool_name(requested_call.name)
        arguments = parse_json(requested_call.arguments)
        validate_arguments(requested_call.name, arguments)
        authorize(requested_call.name, user_context, arguments)

        result = execute_tool(requested_call.name, arguments, user_context)
        messages.append(response.as_assistant_message())
        messages.append(make_tool_result(requested_call.id, result))

raise WorkflowLimitError('Maximum tool steps exceeded')

Use a maximum step count even for apparently simple workflows. Without a limit, a confused model, malformed result, or repeated failure can create an expensive loop. Record the tool-call ID, tool name, validation outcome, latency, and final status for troubleshooting.

4. Validate arguments twice

Schema guidance helps the model format its request, but it is not a security boundary. Parse the arguments as data, reject malformed JSON, enforce types, and apply business rules in application code. A valid string may still be too long, refer to another tenant, or contain an identifier that does not exist.

Return errors in a predictable structure. A useful tool error can include a safe error code, a short explanation, and whether retrying is appropriate:

{
  "ok": false,
  "error": {
    "code": "ORDER_NOT_FOUND",
    "message": "No accessible order matched that identifier.",
    "retryable": false
  }
}

Do not pass stack traces, access tokens, raw database errors, or confidential records into the conversation. Log sensitive diagnostic detail on the server with appropriate controls instead.

5. Add retries deliberately

Retries are useful for transient network failures, rate limits, and temporary upstream errors. They are dangerous when applied to non-idempotent actions such as sending an email, creating a payment, or placing an order. Classify operations before adding retry behavior.

For retryable operations, use a small bounded retry policy with backoff. For actions that can produce duplicates, use an idempotency key or a server-side operation record. The model should not decide whether an operation is safe to repeat. That decision belongs to the tool implementation.

6. Handle multi-step plans conservatively

A multi-step workflow might retrieve an account, check eligibility, and then create a draft. Keep each step observable and enforce a clear transition policy. A tool that changes state should generally require stronger checks than a read-only tool. Consider requiring confirmation before irreversible actions, especially when the request is ambiguous.

For example, an assistant may be allowed to call create_draft_email automatically but should ask for confirmation before send_email. This human-in-the-loop pattern is covered in more detail in our guide to adding human review to AI workflows.

Tools and handoffs

A reliable implementation usually has four distinct layers:

• Model adapter: Converts your internal message and tool representation into the selected provider’s format and normalizes the response.
• Tool registry: Stores tool names, schemas, descriptions, version information, and execution handlers.
• Policy layer: Checks identity, permissions, rate limits, approval requirements, and environment restrictions before execution.
• Tool handlers: Call APIs or internal services and return a deliberately small, structured result.

This separation makes provider changes less disruptive and keeps business logic out of prompts. It also makes it easier to disable a tool without removing unrelated application code.

Keep tool outputs concise. Return the fields needed for the next decision rather than dumping an entire API response into the context. For a product search, that might mean product ID, name, availability, and price—not every internal field. Smaller outputs reduce ambiguity and make evaluations more stable.

When a tool depends on retrieved documents, treat retrieval as its own component. A RAG workflow can expose a search tool that returns document identifiers, excerpts, and source metadata before a later answer is generated. See the RAG tutorial for a broader retrieval workflow and the RAG reliability checklist for evaluation considerations.

Quality checks

Test the workflow at the tool boundary, not only through conversational examples. A useful test set should include:

• Clear requests that should produce one known tool call.
• Requests that should produce no tool call because the capability is unavailable.
• Missing, extra, malformed, or incorrectly typed arguments.
• Attempts to access another user’s data.
• Transient API failures and permanent validation errors.
• Repeated tool calls and maximum-step behavior.
• Ambiguous requests that should trigger clarification or confirmation.
• Large, empty, stale, or unexpected tool results.

Measure more than whether the final answer sounds helpful. Track tool-selection accuracy, argument validity, authorization failures, execution success, retry counts, latency, and whether the final answer accurately reflects the tool result. For actions that change state, verify the real system state in a separate check.

Keep a versioned set of representative conversations. When you change a schema, system prompt, model, or handler, replay the set and compare outcomes. The prompt testing framework guide provides a practical structure for organizing these evaluations.

Also test prompt-injection resistance. Treat text returned by a webpage, document, email, or API as untrusted data. A retrieved document may contain instructions that conflict with your system rules; it should not be allowed to redefine available tools or permissions. Use allowlists, strict tool schemas, and server-side authorization to limit the impact.

When to revisit

Revisit this workflow whenever the model provider changes its tool-calling format, structured-output behavior, authentication method, or error responses. Provider documentation and SDK abstractions can evolve, so keep the model adapter isolated and add a compatibility test before upgrading dependencies.

Update the implementation when you add a tool, change an argument, alter a permission rule, introduce a state-changing action, or connect a new external API. Treat schema changes as interface changes: version them where practical, update fixtures, and test old conversations against the new behavior.

Review the workflow after incidents involving duplicate actions, unauthorized data, unexpected costs, long-running loops, or misleading tool results. These events often indicate a missing control in validation, idempotency, authorization, or observability rather than a prompt problem alone.

As a practical maintenance routine, keep a small checklist with each release:

1. Confirm every exposed tool has an owner, schema, permission rule, and failure policy.
2. Run regression conversations for normal, ambiguous, invalid, and adversarial inputs.
3. Verify retry and idempotency behavior for every state-changing operation.
4. Inspect logs for tool latency, failure rates, repeated calls, and unexpected arguments.
5. Review whether tool outputs contain more sensitive or irrelevant data than the model needs.
6. Record the provider, model, prompt, schema, and application versions used by each evaluation.

Start with one read-only tool, a strict schema, and a bounded execution loop. Once that path is observable and tested, add write actions behind explicit authorization and confirmation. This incremental approach produces a more dependable AI API integration than exposing a large set of powerful functions at the beginning.