Designing a Reliable AI Agent Platform: A Reference Architecture

Building an AI agent that answers a prompt is relatively easy.

Building an AI agent platform that can execute real workloads reliably, recover from failure, call tools safely, preserve state, and scale across multiple workers is a different engineering problem.

This reference architecture explores how TypeScript, Python, Redis, PostgreSQL, Docker, and Kubernetes can support reliable agent workflows.

The design target is not another chatbot. It is a reusable platform for running intelligent, tool-using workflows inside a reliable backend system.

The problem#

A simple AI integration often starts with a single API request:

const response = await openai.responses.create({
  model: "gpt-5",
  input: "Analyze this customer request.",
})

This approach works for demonstrations, but production systems quickly introduce more demanding requirements.

A robust agent platform may need to support:

• Multiple specialized agents.
• Long-running workflows.
• Structured tool calls.
• Human approval steps.
• Retryable background jobs.
• Persistent conversation and workflow state.
• Local and cloud-hosted language models.
• Horizontal worker scaling.
• Per-user permissions and usage limits.
• Detailed execution logs.
• Reliable recovery after process or server failures.

The central challenge is connecting probabilistic AI reasoning with deterministic backend infrastructure.

Language models are non-deterministic, but production systems cannot be unpredictable about security, state transitions, billing, retries, or data integrity.

The reference architecture therefore isolates AI reasoning from the parts of the system that require strict guarantees.

Design goals#

The reference design starts with the following goals.

Reliability#

A failed model request, worker restart, network interruption, or tool timeout should not corrupt the workflow.

Observability#

Every workflow step should produce enough information to answer:

• What happened?
• Which model was used?
• Which tool was called?
• How long did it take?
• Why did it fail?
• Can it be retried safely?

Scalability#

API servers, agent workers, and tool workers should scale independently.

Security#

The model should never receive unrestricted access to internal services, databases, files, or user credentials.

Extensibility#

New models, tools, agents, and workflow types should be added without rewriting the entire platform.

Model independence#

The application should support cloud providers and local models through a common internal abstraction.

High-level architecture#

The reference architecture is divided into several clearly defined layers.

This separation prevents the language model from becoming the center of the entire application.

The model participates in a workflow, but it does not own the workflow.

Request lifecycle#

A typical request moves through the system in the following order:

1. A client creates an agent run through the API.
2. The API validates the request and checks permissions.
3. A durable workflow record is created in PostgreSQL.
4. A job is added to Redis.
5. An available agent worker claims the job.
6. The worker loads the workflow context.
7. The model decides whether it needs to answer, call a tool, or request approval.
8. Tool calls are validated and dispatched to isolated workers.
9. Results are written back to PostgreSQL.
10. The next workflow step is scheduled.
11. The client receives updates through polling, Server-Sent Events, or WebSockets.

This model makes the API request short-lived while the actual agent workflow can continue for seconds or minutes.

Why separate the API from agent execution#

Running agent workflows directly inside an HTTP request creates several problems:

• Requests can exceed platform timeouts.
• A process restart destroys in-memory state.
• Expensive AI work competes with normal API traffic.
• Retrying the HTTP request may repeat side effects.
• Scaling the API also scales expensive agent resources unnecessarily.

In this design, the API creates a durable run and schedules asynchronous work.

type CreateAgentRunInput = {
  userId: string
  agentType: string
  message: string
}

export async function createAgentRun(
  input: CreateAgentRunInput
) {
  const run = await database.transaction(async (transaction) => {
    const createdRun = await transaction.agentRun.create({
      data: {
        userId: input.userId,
        agentType: input.agentType,
        status: "queued",
        input: {
          message: input.message,
        },
      },
    })

    await transaction.outboxEvent.create({
      data: {
        type: "agent.run.created",
        aggregateId: createdRun.id,
        payload: {
          runId: createdRun.id,
        },
      },
    })

    return createdRun
  })

  return run
}

In this example, the API does not publish directly to Redis inside the transaction. Instead, it stores an outbox event.

That distinction matters.

The transactional outbox#

A common failure scenario looks like this:

1. PostgreSQL successfully creates a workflow.
2. The application tries to publish a queue message.
3. Redis is temporarily unavailable.
4. The database contains a queued workflow, but no worker knows about it.

In this design, a transactional outbox addresses the gap by writing the workflow and the event in the same PostgreSQL transaction.

A separate publisher could read unprocessed outbox events and publish them to Redis.

export async function publishOutboxBatch() {
  const events = await database.outboxEvent.findMany({
    where: {
      processedAt: null,
    },
    orderBy: {
      createdAt: "asc",
    },
    take: 100,
  })

  for (const event of events) {
    await queue.publish(event.type, event.payload)

    await database.outboxEvent.update({
      where: {
        id: event.id,
      },
      data: {
        processedAt: new Date(),
      },
    })
  }
}

Durable workflow state#

Redis is useful for queues, locks, short-lived caches, and coordination.

It is not the primary source of truth for agent execution.

In this reference design, PostgreSQL stores the durable state of every run.

CREATE TABLE agent_runs (
  id UUID PRIMARY KEY,
  user_id UUID NOT NULL,
  agent_type TEXT NOT NULL,
  status TEXT NOT NULL,
  input JSONB NOT NULL,
  output JSONB,
  current_step INTEGER NOT NULL DEFAULT 0,
  version INTEGER NOT NULL DEFAULT 1,
  started_at TIMESTAMPTZ,
  completed_at TIMESTAMPTZ,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Each individual step is stored separately:

CREATE TABLE agent_run_steps (
  id UUID PRIMARY KEY,
  run_id UUID NOT NULL REFERENCES agent_runs(id),
  step_number INTEGER NOT NULL,
  step_type TEXT NOT NULL,
  status TEXT NOT NULL,
  model TEXT,
  tool_name TEXT,
  input JSONB,
  output JSONB,
  error JSONB,
  started_at TIMESTAMPTZ,
  completed_at TIMESTAMPTZ,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

  UNIQUE (run_id, step_number)
);

This schema provides the basis for an auditable execution history.

With corresponding recovery logic and tests, a worker could restart and continue from the last completed step instead of starting the entire workflow again.

Agent worker design#

The illustrative Python worker is responsible for model interaction, structured output, and reasoning-related operations.

It is not responsible for unrestricted infrastructure access.

from dataclasses import dataclass
from typing import Any

@dataclass
class AgentJob:
    run_id: str
    agent_type: str
    input: dict[str, Any]

async def process_agent_job(job: AgentJob) -> None:
    run = await workflow_repository.get_run(job.run_id)

    if run.status in {"completed", "cancelled"}:
        return

    await workflow_repository.mark_running(job.run_id)

    try:
        context = await context_builder.build(run)
        decision = await agent_runtime.execute(context)

        await workflow_engine.apply_decision(
            run_id=job.run_id,
            decision=decision,
        )

    except Exception as error:
        await workflow_repository.record_failure(
            run_id=job.run_id,
            error=str(error),
        )

        raise

In this design, the worker receives a job identifier and reloads the current state from PostgreSQL.

This avoids passing large or stale workflow state through the queue.

Structured model output#

Free-form model responses are difficult to process safely.

The agent therefore produces one of a small number of structured decisions.

import { z } from "zod"

export const agentDecisionSchema = z.discriminatedUnion("type", [
  z.object({
    type: z.literal("final_answer"),
    content: z.string().min(1),
  }),

  z.object({
    type: z.literal("tool_call"),
    tool: z.string().min(1),
    arguments: z.record(z.unknown()),
  }),

  z.object({
    type: z.literal("request_approval"),
    reason: z.string().min(1),
    proposedAction: z.record(z.unknown()),
  }),

  z.object({
    type: z.literal("continue"),
    reason: z.string().min(1),
  }),
])

export type AgentDecision = z.infer<
  typeof agentDecisionSchema
>

This does not make the model deterministic, but it gives the backend a controlled contract.

The intended boundary rejects malformed output rather than guessing what the model intended.

Tool execution boundaries#

Tools are among the most powerful and dangerous parts of an agent platform.

A tool can:

• Query internal databases.
• Send an email.
• Modify a calendar.
• Create a support ticket.
• Execute code.
• Read a document.
• Call an external API.

The model should never receive direct credentials.

Instead, it requests a named tool with structured arguments.

{
  "type": "tool_call",
  "tool": "create_support_ticket",
  "arguments": {
    "customerId": "cus_123",
    "subject": "Payment issue",
    "priority": "high"
  }
}

A production backend should then perform several checks:

1. Is the tool registered?
2. Is this agent allowed to use it?
3. Is this user allowed to trigger it?
4. Do the arguments match the schema?
5. Does the action require approval?
6. Has the same side effect already been executed?

import { z } from "zod"

const createTicketInputSchema = z.object({
  customerId: z.string().min(1),
  subject: z.string().min(3),
  priority: z.enum(["low", "normal", "high"]),
})

export const toolRegistry = {
  create_support_ticket: {
    inputSchema: createTicketInputSchema,
    requiresApproval: true,

    execute: async (input: unknown) => {
      const validatedInput =
        createTicketInputSchema.parse(input)

      return supportClient.createTicket(validatedInput)
    },
  },
}

Idempotency and side effects#

Retries are unavoidable in distributed systems.

A worker may complete a tool call and crash before acknowledging the queue message. The queue can then deliver the same job again.

Without idempotency, the platform might send the same email twice or create duplicate tickets.

Each side-effecting operation should receive an idempotency key:

type ExecuteToolInput = {
  runId: string
  stepId: string
  toolName: string
  arguments: unknown
}

export async function executeTool(
  input: ExecuteToolInput
) {
  const idempotencyKey =
    `${input.runId}:${input.stepId}:${input.toolName}`

  const existingExecution =
    await database.toolExecution.findUnique({
      where: {
        idempotencyKey,
      },
    })

  if (existingExecution?.status === "completed") {
    return existingExecution.result
  }

  return toolExecutionService.execute({
    ...input,
    idempotencyKey,
  })
}

With a uniqueness constraint and a transactional execution record, this pattern can support safe retries. The illustrative lookup alone is not sufficient proof of that guarantee.

Redis responsibilities#

In this reference design, Redis is used for short-lived operational concerns:

Redis should not be used as the only storage location for business-critical workflow state.