Architecture

Building Stateful Agents with the OpenAI Responses API: A Practical Guide

By IdeaBosqueLast updated: September 5, 2026

Key takeaways

The Responses API is OpenAI's recommended API primitive for all new projects, and the Assistants API was sunset August 26, 2026 — the migration window is closed; Chat Completions remains supported but is not where new agentic features land first.
Internal OpenAI evals show a 3% improvement on SWE-bench and 40–80% better cache utilization versus Chat Completions when using reasoning models like GPT-6 Astra through the Responses API — the agentic loop is not just convenience, it measurably improves output quality.
Remote MCP servers are a first-class tool type in the Responses API — you connect any MCP server with a server_url and server_label, and the model discovers and calls its tools within the same request, no custom orchestration required.
GPT-6 Astra's misalignment monitor stops API tasks outright when it flags unauthorized behavior — there is no resume path; the workflow must be recoverable from durable state, which is the single most important architectural constraint for production adoption.
Stateful mode is approximately 2x slower than stateless Chat Completions per multiple developer reports — the convenience of previous_response_id comes with a latency tax that matters for latency-sensitive user-facing flows.

The OpenAI Responses API is the company's recommended API primitive for all new development, and the Assistants API was officially sunset on August 26, 2026. Chat Completions remains supported, but the Responses API is where new agentic capabilities — built-in tools, stateful conversations, remote MCP, background mode, mid-turn steering — land first. For a team building production agents on GPT-6 Astra, the question is no longer whether to migrate but how to architect around the API's state model, its latency characteristics, and the governance constraints that Astra's runtime monitor imposes. This guide maps the five capabilities that matter, the three decisions that determine adoption, and the architectural pattern that keeps your agent portable across providers.

What the Responses API changes

The Chat Completions API is stateless: you send the full conversation history with every request, and the API returns a single message. The Responses API introduces three structural changes that affect how you build agents.

Items instead of messages. Chat Completions returns an array of choices, each containing a message. The Responses API returns an array of output Items, where each Item is a typed union — a message, a function_call, a function_call_output, a reasoning summary, or a tool call. This is not cosmetic: it means tool calls, reasoning, and text are first-class objects in the response, not fields glued onto a message. When you chain responses with previous_response_id, the API preserves all Item types — including encrypted reasoning — across turns, which is what makes multi-turn agentic workflows work without manual context replay.

An agentic loop in one request. The Responses API is designed as an agentic loop: the model can call multiple tools — web_search, file_search, computer_use, code_interpreter, image_generation, remote MCP servers, and custom functions — within a single API call, iterating until it reaches a stopping condition. With Chat Completions, you implement this loop yourself: call the model, parse the tool call, execute it, append the result, call again. The Responses API runs the loop server-side. OpenAI's internal evals show a 3% improvement on SWE-bench with the same prompt and setup when using reasoning models through the Responses API, plus 40–80% better cache utilization — the server-side loop benefits from cache hits that a manual loop cannot replicate.

Stateful context via previous_response_id. Instead of sending the full history with every request, you pass the ID of the previous response and the new user input. The API reconstructs the context server-side, including reasoning items. Responses are stored by default for 30 days; any response attached to a conversation persists its items with no TTL. You can disable storage with store: false for zero-data-retention workflows, but then you must replay the full Item history manually — including encrypted reasoning items — to preserve reasoning context across turns.

The five capabilities that matter

The Responses API's capabilities map to five architectural decisions, each with a concrete trade-off:

1. previous_response_id: stateful chaining

The simplest stateful pattern chains responses by ID:

from openai import OpenAI
client = OpenAI()

first = client.responses.create(
    model="gpt-6-astra",
    input="What is the capital of France?",
    store=True,
)

second = client.responses.create(
    model="gpt-6-astra",
    previous_response_id=first.id,
    input="And its population?",
    store=True,
)

The second call does not re-send the first question or answer. The API reconstructs the full context from the stored response, including any reasoning the model performed. This is the pattern for conversational agents, research assistants, and any workflow where the user's follow-up depends on prior turns.

The trade-off is latency. Multiple developer reports on the OpenAI community forum and Microsoft Q&A indicate the stateful path is approximately 2x slower than stateless Chat Completions — 1 second versus 0.5 seconds in typical cases, and up to 9x slower (2.9 seconds versus 0.3 seconds) under load. For a background research agent that runs for minutes, this is irrelevant. For a user-facing chat that must respond in under 500ms, the latency tax may justify staying on Chat Completions with manual context management.

2. Remote MCP servers as a built-in tool

The Responses API supports remote MCP servers as a first-class tool type. You register a server by URL, and the model discovers its tools and calls them within the agentic loop:

response = client.responses.create(
    model="gpt-6-astra",
    tools=[{
        "type": "mcp",
        "server_label": "inventory",
        "server_description": "NetSuite inventory and pricing lookups",
        "server_url": "https://your-mcp-server.example.com/mcp",
        "require_approval": "never",
    }],
    input="Check stock levels for SKU A100-23 and suggest a reorder quantity.",
)

The require_approval field controls whether the model needs human sign-off before calling the server's tools. For production deployments, the MCP tool filtering options — allowed_tools to whitelist specific tool names, and custom approval policies per tool — are the governance layer that prevents the model from calling destructive operations without explicit authorization.

This is the capability most relevant to IdeaBosque's integration pattern. A custom MCP module that wraps NetSuite, HubSpot, or BigCommerce APIs can be registered as a remote MCP server in a Responses API call, and the model uses it the same way it uses web_search or code_interpreter. The semantic layer — typed schemas, audit logs, rate-limit handling — lives in the MCP module, not in the prompt. The Responses API does not solve the semantic-layer problem; it makes the MCP module the natural integration point. For a deeper treatment of that pattern, see MCP Module Code Standard.

3. Background mode for long-running tasks

Background mode decouples the model call from the client connection. The API accepts the request, returns a response ID immediately, and runs the model work asynchronously. You poll for status or stream results as they arrive:

resp = client.responses.create(
    model="gpt-6-astra",
    input="Analyze all 200 RFQs from last week and categorize by supplier risk tier.",
    background=True,
)

while resp.status in {"queued", "in_progress"}:
    sleep(2)
    resp = client.responses.retrieve(resp.id)

print(resp.output_text)

For long-horizon agents — research synthesis, batch document analysis, multi-step procurement workflows — background mode is the pattern that survives network drops and client timeouts. A background response that takes six minutes does not depend on a live HTTP connection; if the client disconnects, the work continues, and you reconnect with a streaming resume using the last sequence number.

The operational catch is that background mode is not a job queue. As one analysis puts it: the API runs the model call, but your application still owns the job state — what the UI shows, how to avoid processing a webhook twice, when to cancel work that no longer matters. For production, you need a durable job system around the background response, not just the response ID. This is the same pattern described in Long-running Agent Patterns: the agent runtime, not the model API, is the reliability layer.

4. Encrypted reasoning for zero-data-retention workflows

When store: false, the API does not persist the response, but it returns encrypted reasoning items in the output. You pass these items back in the next request's input to preserve reasoning context across turns without storing anything on OpenAI's servers. This is the pattern for regulated environments where data retention is prohibited — EU AI Act high-risk systems, healthcare workflows under HIPAA, defense ITAR workflows.

The trade-off is that you become the state store. You must serialize, store, and replay the full Item array — including the opaque encrypted reasoning blobs — on every turn. If you lose the encrypted reasoning items, the model loses its reasoning context, and output quality degrades. This is the same state-management burden as Chat Completions, but with an additional item type to handle.

5. The Astra task-stop monitor

GPT-6 Astra ships with misalignment monitoring on every tool-using request. When the monitor flags potentially unauthorized behavior, the model's response includes a stop signal. In ChatGPT and Codex, the user sees a paused task to review. In the API, the task stops outright — there is no resume path.

For production agents built on the Responses API, this is the single most important architectural constraint. A task that runs for hours through previous_response_id chains or background mode can be terminated mid-flight by a classifier. Your workflow must be recoverable from durable state — every tool call, every intermediate result, every partial output must be persisted to your own store before the next API call. If the monitor stops the task at step 47 of 50, you need to be able to resume from step 47, not restart from zero.

OpenAI's chief scientist, Jakub Pachocki, disclosed in An Alien Mind (September 6, 2026) that the company's ability to rely on chain-of-thought monitoring is "progressively diminishing" — models are better at reasoning about and manipulating their own reasoning process, and improved pretraining makes models smarter even without verbalized reasoning. The monitor that stops your task is the best available runtime enforcement layer, but its vendor has said the signal it depends on is degrading. For a deeper treatment of the enforcement layers that do not read the model's reasoning, see GPT-6 Astra Ships the Runtime Kill Switch.

The three adoption decisions

Decision 1: Stateful or stateless?

Use store: true with previous_response_id when your workflow is conversational, multi-turn, and latency-tolerant. Use store: false with manual Item replay when your workflow requires zero data retention or when you need full control over context management. The latency difference is roughly 2x — acceptable for background agents, potentially unacceptable for user-facing chat.

Decision 2: Built-in tools or custom functions?

The built-in tools (web_search, file_search, code_interpreter, computer_use, image_generation, remote MCP) run server-side and benefit from the agentic loop's cache optimization. Custom functions require you to implement the tool-call loop yourself. The practical rule: use built-in tools for capabilities OpenAI provides better than you can (web search, code execution), and use remote MCP servers for your own system integrations (NetSuite, HubSpot, BigCommerce). Use custom functions only for capabilities that cannot be exposed as an MCP server.

Decision 3: OpenAI-only or model-flexible?

The Responses API is an OpenAI primitive. If you build your agent entirely on previous_response_id and built-in tools, you are locked to OpenAI's state store and tool ecosystem. If your production requirement includes model flexibility — routing to open-weight models like Qwen3.8-27B for cost-sensitive tasks, or to Claude for specific capabilities — you need an abstraction layer that translates between the Responses API's Item model and the Chat Completions message format that other providers use.

This is the architectural decision that determines whether the Responses API is your entire agent runtime or one backend among several. A model-flexible build keeps the agent loop in your own runtime, uses the Responses API when its capabilities justify the latency and lock-in, and falls back to Chat Completions or open-weight inference when they do not. The inference economics are clear: 29% of production token volume already runs on open-weight models at under 4% of spend. Routing discipline is a production reality, not a future plan.

Migration checklist

OpenAI's migration guide provides the full checklist. The decisions that affect architecture, not just code:

Decide your state model. previous_response_id, manual Item replay, or the Conversations API. This determines your latency profile and your data-retention posture.
Audit your function definitions. Custom functions migrate as-is, but function call outputs must include the correct call_id. Dropping reasoning or function-call Items when manually carrying context is the most common migration error.
Move Structured Outputs schemas from response_format to text.format — the field name changed.
Add durable state persistence for any workflow that runs more than a few seconds. The Astra task-stop monitor can terminate a long task with no resume path; your state store is the recovery mechanism.
Compare latency, token usage, and error rates before routing production traffic. The 3% SWE-bench improvement and 40–80% cache improvement are averages; your workload may differ.
Keep a Chat Completions fallback if model flexibility matters. The Responses API is OpenAI-only; other providers speak Chat Completions.