As checked 2026-10-25: Batch inference separates jobs that must reply to a user immediately from jobs that can wait. This is a dated economics guide, not a permanent price guarantee or a personal hands-on benchmark.
Scope: global English reference; USD list prices where shown; local eligibility, currency, taxes, negotiated terms and model release dates require separate confirmation. Dynamic source-page prices have a check date, not a verified initial effective date.
A cheaper job may finish later
Batch inference separates jobs that must reply to a user immediately from jobs that can wait. OpenAI describes a Batch API with discounted processing and an advertised 24-hour completion window. Anthropic’s dated May 27, 2026 tariff lists distinct batch token prices for individual Claude models. Google’s Gemini Developer API shows batch-specific tariffs that must be read separately from standard, flex and priority processing. None of these facts makes batch suitable for an interactive chat reply that must arrive within seconds.
Model the correct workload
For a back-catalog audit, count documents, average input tokens, output tokens, failed attempts and expected completion time. Multiply measured tokens by the specific batch rates from the same SKU and currency. If 40,000 records each have an average of 2,000 input tokens, that is 80 million input tokens, before output or validation. Make the batch split and retry policy explicit: a partial failure may require submitting a subset again, and a lower per-token rate does not remove application engineering or storage costs.
Cost is only one admission rule
Check that the endpoint and tool combination is actually supported in batch mode. Rate limits, file sizes, cancellation behavior, result-order handling, and maximum turnaround differ by service. A request requiring a human’s immediate approval, a fresh inventory lookup or a tight sales-service-level objective should not be routed into a long asynchronous queue simply to reduce token spend. Consider whether cheap overnight processing introduces a larger operational bill if results arrive after their business deadline.
Evaluation and traceability
Keep stable request IDs and hashes so results are correlated with their source records regardless of completion order. Reject schema-invalid outputs and repeat only rejected items. Track queue wait, total completion latency, error class, billable tokens and acceptance rate per batch. Treat vendor-published discount percentages as commercial terms, not as empirical savings from a workload you have not tested.
Practical alternatives
Use synchronous standard processing for real-time support and user-facing answers. Use caches for recurring common context, smaller specialized models for straightforward classification when quality is acceptable, and deterministic code for operations that need no model. Before scheduling a batch, confirm the dated price IDs, region and model version; price table entries can disappear when models are retired.
Design the job file for recovery
Before dispatch, generate a manifest with canonical task IDs, prompt template versions and expected output schema. Split workloads into independently retryable units rather than one opaque mega-job. Save the returned request ID for every record and match results on that ID instead of result line order. If 5 percent of outputs fail validation in a hypothetical test, rerun that subset only and account for its additional usage. The pipeline should report both cost per attempted record and cost per validated record; discounts on calls do not make invalid outputs useful.
Know when not to batch
Time-sensitive fact checks should retrieve and validate the current official source before an editor approves the article. A job that takes hours to return may have been accurate when submitted yet outdated at publication. For dynamic pricing, prefer retrieval and validation close to publishing, while batching stable extraction, classification and formatting work. If the provider’s batch endpoint does not support a required web tool or model version, route that stage separately. Record the queue’s failure policy and expiration rules from current documentation rather than importing assumptions from another vendor.
Decision checklist
- Identify the specific plan or API endpoint, model version, region and currency before pricing a job.
- Measure input, output, caching, generated media and failed-job units separately.
- Check the official pricing page again before procurement, especially if the SKU has no effective-date notice.
- Keep source evidence and a timestamped price ID; do not treat unknowns as zero.
Pricing evidence boundary
This cost discussion concerns Multi-vendor. Reproduce a quote only with its exact model or plan identifier, chargeable unit, billing region, processing mode and timestamp; a changing vendor rate card is not a permanent price guarantee. If any one of these dimensions is missing, keep the associated rate and worked scenario under editorial review instead of filling the gap with an estimate.
Sources & Further Reading
- Batch API asynchronously handles eligible requests with stated cost reduction and turnaround — official first-party source, accessed 2026-10-25; original effective date not specified in source.
- Official list-price PDF with effective date 2026-05-27 and scoped Claude API rate rows — official first-party source, accessed 2026-10-25; official price-effective date 2026-05-27.
- Gemini Developer API shows tier/model/context/modality-specific rates and cache storage — official first-party source, accessed 2026-10-25; original effective date not specified in source.