2.2 Message Batches API & Asynchronous Workloads

2.2 Message Batches API & Asynchronous Workloads

Core Concept: The Anthropic Message Batches API provides an asynchronous, high-throughput execution channel engineered for bulk, non-interactive workloads. By decoupling requests from real-time synchronous serving constraints, the Batches API provides a flat 50% discount across all token modalities (input, output, cache creation, and cache read) with most batches finishing in under 1 hour and a hard 24-hour expiration.

---

Architectural Purpose of the Message Batches API

Modern enterprise AI architectures frequently require processing massive volumes of data where immediate sub-second responses are unnecessary. Running millions of tokens through standard synchronous endpoints (POST /v1/messages) introduces significant architectural friction:

1. Connection Fragility: Synchronous HTTP connections held open across long generation windows risk timeout errors, dropped sockets, and transient network disconnects.
2. Rate Limit Contention: High-volume batch jobs (such as re-indexing document embeddings, evaluating historical conversations, or running overnight classification pipelines) compete directly with user-facing interactive traffic for requests-per-minute (RPM) and tokens-per-minute (TPM) quotas.
3. Economic Inefficiency: Paying standard retail token pricing for background tasks that could easily execute during off-peak inference windows wastes organizational budget.

The Message Batches API solves these challenges by providing a dedicated asynchronous ingestion and execution pipeline. In exchange for relaxing the latency guarantee - most batches finish within an hour, and any request not sent to the model within 24 hours expires - Anthropic cuts all token costs in half—including inputs, outputs, prompt cache writes, and prompt cache reads. Furthermore, batches execute against dedicated asynchronous capacity pools, shielding interactive production services from rate limit exhaustion.

---

Synchronous vs. Asynchronous Decision Matrix

Selecting between the standard Messages API (with or without streaming) and the Message Batches API is a foundational architectural decision:

---

Batch Request Lifecycle & Operations

The Message Batches API follows a decoupled lifecycle consisting of submission, background processing, monitoring, and result retrieval.

1. Creating a Batch (POST /v1/messages/batches)

Batches are initialized by posting a collection of message requests. Each individual item in the requests list requires two primary attributes:

• custom_id: A developer-assigned unique string (1 to 64 characters; alphanumeric, underscores, and hyphens) used to correlate responses back to original records.
• params: A standard Messages API payload object specifying model, max_tokens, messages, and optional configurations like system, tools, temperature, or cache_control.

import anthropic

client = anthropic.Anthropic()

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"doc-classification-{idx}",
            "params": {
                "model": "claude-sonnet-5",
                "max_tokens": 256,
                "system": [
                    {
                        "type": "text",
                        "text": "Classify the sentiment and category of the provided legal filing.",
                        "cache_control": {"type": "ephemeral"}  # Caching works in batches!
                    }
                ],
                "messages": [
                    {"role": "user", "content": document_text}
                ]
            }
        }
        for idx, document_text in enumerate(corpus_documents)
    ]
)

print(f"Batch created with ID: {batch.id}, Status: {batch.processing_status}")

2. Batch Lifecycle States

A batch transitions through distinct states recorded in processing_status:

• in_progress: The batch has been validated and queued. Requests are executing concurrently across worker pools.
• canceling: A cancellation command has been submitted via POST /v1/messages/batches/{batch_id}/cancel. Workers are terminating un-started tasks while allowing running tasks to finish.
• ended: Terminal state. All requests in the batch have reached completion (succeeded, errored, canceled, or expired).

3. Monitoring Batch Progress

Clients query batch state via GET /v1/messages/batches/{batch_id}. The response provides fine-grained counters in the request_counts object:

{
  "id": "msgbatch_01ABCxyz...",
  "type": "message_batch",
  "processing_status": "ended",
  "request_counts": {
    "processing": 0,
    "succeeded": 9982,
    "errored": 18,
    "canceled": 0,
    "expired": 0
  },
  "created_at": "2026-09-10T02:00:00Z",
  "ended_at": "2026-09-10T02:45:12Z",
  "results_url": "https://anthropic-batches.s3.amazonaws.com/results_01ABCxyz.jsonl?AWSAccessKeyId=..."
}

---

Result Processing & Result Correlation via custom_id

Once a batch enters the ended state, results are available for download as a newline-delimited JSON (JSONL) file.

The Non-Deterministic Ordering Rule

A fundamental architectural characteristic of the Batches API is that results in the JSONL output file are NOT emitted in the original submission order. Because Anthropic's distributed scheduler processes requests asynchronously across thousands of parallel worker nodes, shorter prompts or faster completions may finish ahead of earlier submissions.

Developers must correlate results using custom_id rather than array indexing.

JSONL Schema Structure

Each line in the downloaded JSONL file contains an independent JSON object encapsulating the result of a single custom_id:

{"custom_id": "doc-classification-0", "result": {"type": "succeeded", "message": {"id": "msg_01...", "type": "message", "role": "assistant", "content": [{"type": "text", "text": "Category: Regulatory Compliance. Sentiment: Neutral."}], "model": "claude-sonnet-5", "stop_reason": "end_turn", "usage": {"input_tokens": 1520, "output_tokens": 18, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 1450}}}}
{"custom_id": "doc-classification-1", "result": {"type": "errored", "error": {"type": "invalid_request_error", "message": "max_tokens: 256 is smaller than minimum required"}}}
{"custom_id": "doc-classification-2", "result": {"type": "canceled"}}
{"custom_id": "doc-classification-3", "result": {"type": "expired"}}

Result Status Types & Error Boundaries

The result.type field reflects one of four outcomes:

1. "succeeded": The request completed normally. The object contains the standard Anthropic message object with full content, usage, and token metrics.
2. "errored": The individual request failed (e.g., malformed payload, invalid schema, or content filter trigger). It contains an error object with standard error codes.
3. "canceled": The batch was canceled by the client before this specific request began processing.
4. "expired": The batch reached its 24-hour expiration before this request was sent to the model. You are billed for neither expired nor errored requests.

Granular Error Isolation

In synchronous API calls, a batch of requests orchestrated via client-side threads can fail unpredictably if connection drops or rate limits hit. In the Batches API, error boundaries are isolated to the individual request. If 18 out of 10,000 requests in a batch contain an invalid parameter, those 18 requests emit "errored" lines in the JSONL output, while the remaining 9,982 requests process to "succeeded" status without interruption.

---

Quotas, Limits, and Cancellation Semantics

Architectural Limits

• Maximum Requests per Batch: 100,000 Message requests.
• Maximum Batch Payload Size: 256 MB. A batch is capped by whichever of the two ceilings it reaches first, so work beyond either bound must be partitioned across batches.
• Result Retention Window: Results files remain accessible via results_url for 29 days after the batch enters the ended state. After 29 days, the results file is purged.
• Processing Window: Most batches complete in under 1 hour. Results become available once every request finishes or after 24 hours, whichever comes first; any request still unprocessed at 24 hours expires and is not billed. Actual speed depends on cluster load and queue depth.

Cancellation Semantics

A batch in the in_progress state can be canceled at any time by issuing:

POST /v1/messages/batches/{batch_id}/cancel

Understanding cancellation behavior is a common exam topic:

• The batch immediately transitions from in_progress to canceling.
• Any individual requests that are currently executing in worker memory will run to completion and be recorded as "succeeded" (or "errored").
• Requests that are still queued and have not yet started execution are halted and recorded as "canceled".
• Once all active requests finalize, the batch transitions to ended.
• Developers can still download the partial results file via results_url to retrieve all requests that succeeded before the cancellation completed.

---

Combining Prompt Caching with Message Batches

A powerful production optimization pattern is compounding Prompt Caching with the Message Batches API. Because prompt caching operates inside batch processing:

1. The 50% batch discount applies to all token types, including cache_creation_input_tokens and cache_read_input_tokens.
2. For Claude Sonnet 5:
   • Synchronous base input: $2.00 / MTok
   • Batch base input (50% off): $1.00 / MTok
   • Synchronous cache read (0.1x): $0.20 / MTok
   • Batch cache read (50% of 0.1x): $0.10 / MTok!

When evaluating large document corpuses against a static instruction set using batching and prompt caching together, developers achieve an astonishing 95% total cost reduction compared to standard synchronous un-cached execution ($0.10 vs $2.00 per MTok).