Scaling Managed Agents: Multi-Agent Orchestration & Swarm Architecture

When building autonomous AI systems, the earliest instinct is often to build an agent loop that runs inside a virtual machine or Docker container, giving the model a shell, a file system, and an execution environment.

In simple demos, this works. But as workflows scale to long-horizon enterprise tasks, this coupled pattern collapses. As Anthropic’s systems research on Managed Agents reveals, the fundamental challenge of scaling autonomous agents is architectural: how to design a system for “programs as yet unthought of” while decoupling the reasoning engine from the execution hands.

1. The Bitter Lesson in Agent Harnesses: Don’t Adopt a Pet

A recurring lesson across frontier AI engineering—echoing Rich Sutton’s Bitter Lesson—is that hardcoded scaffolding assumptions become technical debt when models improve.

In early agent harnesses, engineers observed behaviors like “context anxiety,” where Claude Sonnet 4.5 wrapped up tasks prematurely as the token limit approached. The harness team built aggressive context-reset mechanisms into the loop. Yet when that same harness ran on Claude Opus 4.5, the behavioral quirk was gone; the scaffold resets had become dead weight.

The Single-Container “Pet” Anti-Pattern

Early managed agent architectures placed the session log, the agent harness, and the sandbox inside a single container:

┌──────────────────────────────────────────────────────────┐
│                   COUPLED CONTAINER (A "PET")            │
│                                                          │
│  ┌────────────────┐   Direct Syscalls   ┌─────────────┐  │
│  │ Agent Harness  │ ──────────────────► │  Sandbox /  │  │
│  │ (Claude Loop)  │                     │ Credentials │  │
│  └───────┬────────┘                     └─────────────┘  │
│          │ In-Memory Log                                 │
│          ▼                                               │
│  ┌────────────────┐                                      │
│  │  Session Data  │                                      │
│  └────────────────┘                                      │
└──────────────────────────────────────────────────────────┘

While placing everything into one container avoids designing network boundaries, it creates catastrophic operational fragility:

1. The Container Becomes a “Pet”: In cloud infrastructure, pets are hand-nursed individuals you cannot afford to lose, while cattle are interchangeable. When a container crashes, the entire session is destroyed.
2. Opacity and Debugging Paralysis: When a long-running session freezes, an external WebSocket cannot distinguish between an internal harness deadlock, a dropped network packet, or an out-of-memory container crash. Opening a shell inside the container to debug risks exposing proprietary customer data.
3. Rigid Network Coupling: The harness assumes that whatever code or resource the model touches lives inside the container next to it. Connecting to customer Virtual Private Clouds (VPCs) requires complex network peering or running private agent harnesses.

2. Decoupling the Triad: Brain, Hands, and Session

Decades ago, operating systems solved hardware heterogeneity by virtualizing devices into high-level abstractions—processes and files. A POSIX read() system call operates identically whether accessing a 1970s magnetic tape drive or a modern PCIe Gen 5 NVMe SSD.

Managed Agents apply the identical principle to autonomous AI systems by decoupling the architecture into three independent primitives:

               ┌─────────────────────────────────┐
               │         THE BRAIN               │
               │   Stateless Agent Harness       │
               │   (Claude / Foundation Model)   │
               └────────┬───────────────┬────────┘
                        │               │
     getSession() /     │               │ execute(name, input)
     emitEvent()        │               │
                        ▼               ▼
      ┌────────────────────┐    ┌──────────────────────┐
      │    THE SESSION     │    │      THE HANDS       │
      │  Append-Only Log   │    │  Isolated Sandboxes  │
      │  (Durable Storage) │    │  Containers / Tools  │
      └────────────────────┘    └──────────────────────┘

The Harness Leaves the Container

When the harness is decoupled from the execution container:

• Sandboxes Become Cattle: The harness communicates with the sandbox using a standard tool invocation interface: execute(name, input) -> string. If a container dies, the harness catches the error as a standard tool exception and passes it back to the model. The model can then trigger a fresh container provisioning step via provision({resources}).
• Harnesses Become Cattle: Because the session log lives in durable external storage, the harness stores no state in memory. If a harness host crashes, a replacement worker instance boots via wake(sessionId), retrieves the complete event stream with getSession(id), and resumes execution seamlessly from the last event.

// Core Meta-Harness Interfaces
export interface AgentBrain {
  wake(sessionId: string): Promise<void>;
  processTurn(sessionId: string): Promise<TurnResult>;
}

export interface SessionStore {
  getSession(sessionId: string): Promise<SessionMetadata>;
  emitEvent(sessionId: string, event: AgentEvent): Promise<void>;
  getEvents(sessionId: string, slice?: { start?: number; limit?: number }): Promise<AgentEvent[]>;
}

export interface ExecutionHand {
  provision(resources: { cpu: number; memoryMb: number; image: string }): Promise<string>;
  execute(handId: string, toolName: string, input: Record<string, unknown>): Promise<string>;
  destroy(handId: string): Promise<void>;
}

3. Interactive Managed Agent Lifecycle

Explore an interactive simulation demonstrating how decoupled supervisor handoffs, evaluation gates, and isolated tool execution behave in a structured lifecycle:

4. Zero-Trust Security: Decoupling Credentials from Execution

In coupled agent architectures, untrusted code generated by the LLM runs in the same environment where API credentials and tokens reside. A prompt injection attack only needs to persuade the agent to inspect its own environment variables or local disk to exfiltrate full-privilege tokens.

Narrow token scoping is insufficient: as models grow more capable, arbitrary code execution inside an environment holding even restricted credentials creates an untenable blast radius.

The structural remedy is absolute credential isolation: tokens are never placed inside the sandbox where generated code executes.

┌────────────────────────────────────────────────────────┐
│                   MANAGED AGENT CORE                   │
│                                                        │
│  ┌─────────────────┐       Token-Free      ┌────────┐  │
│  │  Agent Harness  │ ────────────────────► │ Sandbox│  │
│  └────────┬────────┘       Tool Call       └────────┘  │
│           │ Session                                    │
│           │ Token                                      │
│           ▼                                            │
│  ┌─────────────────┐    Fetch Secrets     ┌─────────┐  │
│  │ Dedicated Proxy │ ───────────────────► │ Secret  │  │
│  │  (MCP Gateway)  │                      │  Vault  │  │
│  └────────┬────────┘                      └─────────┘  │
│           │ External Request With Token                │
│           ▼                                            │
│  ┌─────────────────┐                                   │
│  │  External API   │                                   │
│  └─────────────────┘                                   │
└────────────────────────────────────────────────────────┘

Anthropic’s Managed Agents architecture implements this via two isolated patterns:

1. Pre-Injected Git Remotes: For repository operations, the host uses access tokens during sandbox initialization to clone the repository and configure the remote URL with authenticated credentials. Inside the sandbox, the model issues standard git pull and git push commands without the agent ever touching or inspecting the secret.
2. MCP Vault Proxies: Custom tools and Model Context Protocol (MCP) servers store OAuth credentials inside a secure vault outside the sandbox. When the agent calls an external tool, it invokes a dedicated gateway proxy using its session identifier. The proxy fetches the credential from the vault and executes the upstream request. The harness and sandbox never hold the credentials.

5. The Session is Not the Context Window

A critical misconception in long-running agent systems is equating the agent’s active context window with the entire session history.

Standard context compaction techniques—such as trimming conversation turns, summarizing past steps, or maintaining temporary scratchpad files—all involve irreversible decisions about what information to discard. If an agent summarizes turn 12 and discards the raw tool output, a downstream step that needs a subtle detail from turn 12 will inevitably fail.

Storing Context as an External Interrogable Object

In Managed Agents, the session log serves as a permanent, durable context object that lives outside the model’s active context window.

Rather than keeping everything in prompt memory, the harness uses getEvents() to interrogate positional slices of the event history:

• Resuming from Interruption: Fetch events from the last recorded checkpoint.
• Rewinding for Failure Analysis: Fetch the window of 5–10 events immediately preceding a tool error to diagnose subtle parameter drift.
• Selective Inspection: Query specific past actions and decisions without carrying thousands of tokens of historical context on every turn.

export interface ContextTransformationPipeline {
  /**
   * Transforms raw session events into an optimized prompt window.
   * Maximizes prompt caching while keeping working tokens minimal.
   */
  prepareContext(events: AgentEvent[], budgetTokens: number): Promise<ModelPrompt>;
}

export class ProductionHarnessTransformer implements ContextTransformationPipeline {
  async prepareContext(events: AgentEvent[], budgetTokens: number): Promise<ModelPrompt> {
    // 1. Maintain static prompt prefixes to maximize 90%+ prompt cache hit rate
    const systemPrompt = this.getStaticSystemManifest();
    
    // 2. Extract recent active turns
    const recentWindow = events.slice(-6);
    
    // 3. For older turns, format high-density semantic event summaries
    const historicalSummaries = this.summarizeOlderTurns(events.slice(0, -6));
    
    return {
      system: systemPrompt,
      messages: [...historicalSummaries, ...recentWindow],
    };
  }
  
  private getStaticSystemManifest(): string {
    return "STABLE_SYSTEM_PROMPT_OPTIMIZED_FOR_PROMPT_CACHING";
  }
  
  private summarizeOlderTurns(events: AgentEvent[]): FormattedTurn[] {
    // Structured, lossless turn digests that preserve artifact IDs and tool statuses
    return events.map(e => ({
      role: 'assistant',
      content: `[EVENT ${e.index}: ${e.type} -> ${e.summary}]`
    }));
  }
}

By separating durable context storage in the session from transient context engineering in the harness, the architecture avoids permanent data loss while enabling high prompt cache hit rates (>90%).

6. Many Brains, Many Hands: High-Concurrency Scaling

When the brain and the hands are coupled inside a single container, concurrency suffers from severe latency overhead:

Every single session—even an exploratory query that will never touch the file system—must wait for a container to provision, clone the git repository, boot dependencies, and connect network sockets before a single token can be generated. This delay is directly reflected in Time-To-First-Token (TTFT).

Zero-Wait Inference (Many Brains)

Decoupling the harness enables lazy, on-demand execution:

• Immediate Inference: The orchestration layer accepts a user prompt, pulls events from the session log, and immediately begins model inference.
• Just-in-Time Provisioning: Only when the model decides to invoke a tool requiring a sandbox does the runtime call provision({resources}).

Anthropic reports that this architectural transition reduced p50 TTFT by roughly 60% and p95 TTFT by over 90%.

Heterogeneous Execution Environments (Many Hands)

Furthermore, decoupling allows one brain to control many hands simultaneously:

• A single orchestration brain can dispatch bash commands to a Linux container, UI interactions to a mobile device emulator, web scraping to a headless browser island, and enterprise queries to remote MCP servers.
• Because hands are completely decoupled, brains can hand off sandboxes to other brains: a planning agent can initialize a repository sandbox, perform architectural analysis, and hand the exact live sandbox reference to a coding specialist agent without serialization overhead.

                               ┌───────────────────────────┐
                               │  Supervisor Orchestrator  │
                               │        (The Brain)        │
                               └─────────────┬─────────────┘
                                             │
                      ┌──────────────────────┼──────────────────────┐
                      ▼                      ▼                      ▼
            ┌──────────────────┐   ┌──────────────────┐   ┌──────────────────┐
            │  Linux Container │   │  Headless Chrome │   │ Remote MCP Tool  │
            │  (Hand: Code/FS) │   │  (Hand: Browser) │   │  (Hand: DB/API)  │
            └──────────────────┘   └──────────────────┘   └──────────────────┘

7. Architectural Checklist for Production Managed Agents

When engineering your next-generation autonomous agent platform, benchmark your architecture against this checklist:

• Decoupled Triad: Ensure Session, Harness, and Sandbox operate as distinct services with strict interface boundaries.
• Cattle Sandboxes: Treat execution containers as disposable. Sandboxes must be re-provisionable in seconds via declarative recipes.
• Stateless Harnesses: Verify that any harness instance can crash without data loss, resuming immediately via wake(sessionId).
• Zero-Trust Credential Vaults: Never pass raw API tokens or SSH keys into execution containers. Use authenticated remote proxies.
• Separation of Session & Context: Store complete event streams externally in append-only storage; perform prompt caching and context compaction transiently in the harness.
• Lazy JIT Provisioning: Never block model inference on container startup. Provision sandboxes only when tool calls are generated.