Batch Inference: Same Model, Half the Price, and Production Never Notices

2026-09-24 · 8 min read · view as markdown

Migrating thousands of candidates through a language model costs what it costs. Or half. At Selenios we cut up to 50% of the token bill every time we process data in bulk, with the same model, the same prompts and the same results. The technique is called batch inference, it is on the price list of every major provider, and most teams I talk to still are not using it for the one kind of workload where it is free money. This is what it is, why it is half price, what it charges you in exchange, and the checklist we wrote after learning it the hard way.

What batch inference is

In real time, every request waits for its response. In batch there is no conversation. You write a file with one JSON object per line, thousands of prompts, and upload it to a bucket. You tell the provider "process this whenever you can". Some time later another file appears with all the answers. No servers of yours waiting on sockets, no retry logic, no queue you have to operate.

On Amazon Bedrock, which is where we run it, the input is a JSONL file in S3 where each line carries an identifier and the same body you would send to InvokeModel:

{"recordId":"a1f3c9e2","modelInput":{ …the InvokeModel body for cv 1… }}
{"recordId":"9c02b7d4","modelInput":{ …the InvokeModel body for cv 2… }}
…
{"recordId":"0d41f8a7","modelInput":{ …the InvokeModel body for cv N… }}

Then one API call creates the job:

import { BedrockClient, CreateModelInvocationJobCommand } from '@aws-sdk/client-bedrock';

const { jobArn } = await bedrock.send(new CreateModelInvocationJobCommand({
  jobName: `cv-parse-${runId}-${shard}`,
  modelId,
  roleArn,
  inputDataConfig: { s3InputDataConfig: { s3Uri: `s3://${bucket}/input/${runId}/${shard}.jsonl` } },
  outputDataConfig: { s3OutputDataConfig: { s3Uri: `s3://${bucket}/output/${runId}/` } },
}));

From there you poll the job status, and when it finishes you read the output files back, match each line to its recordId, and continue your pipeline as if the calls had been made one by one.

Why it costs half

Because you are selling the provider the one thing it has too much of: idle capacity. Inference fleets are sized for peak interactive traffic, and outside the peak the GPUs sit partly empty. When you give up latency, they get work they can schedule into those gaps, and the token price drops by 50%.

This is not a promotion or a negotiated discount. It is the list price of the batch mode on Amazon Bedrock, Anthropic, OpenAI and Gemini, all four at half of their on-demand rate. Same model, same weights, same output. The only thing you are paying less for is the promise of an immediate answer.

A real case: a migration with years of history

We recently migrated a customer from another recruiting system. That means tens of thousands of different CVs, and with them the job openings, the recruiters' notes, the moves between pipeline stages and the evaluations from years of history. Every CV went through a language model on Bedrock to extract its structure into the shape our product expects.

The on-demand bill would have been one number. The batch bill was exactly half, with the same model and the same results. I am not going to put absolute figures here, and it does not matter: the ratio is the point, and it holds at any size. Scale it to thousands of candidates with their whole history and the math stays the same. Half.

There is a second saving that has nothing to do with batch and that you should do anyway: before sending anything, we hash every file. In a recruiting system the same CV shows up attached to many applications. A candidate who applied to five openings has five copies of the same PDF. With an MD5 of the file as the key, the same CV is parsed once and the result is fanned out to every application that references it. The cheapest token is the one you never send.

The advantage nobody talks about: batch does not compete with production

Price is the headline, but it is not the best reason. On-demand traffic comes out of a shared pool, and when that pool fills up the provider answers with a 503 and an "insufficient capacity" message. We have lived it. Now imagine that on top of your normal traffic you fire thousands of calls from a migration script through the same channel, with the same model and the same account. Your users pay for it, in latency and in errors, on the screens they actually use.

Batch is a different queue with a different quota. The migration runs overnight and the platform never notices. Bedrock's own guidance for capacity errors on on-demand says the same thing: move offline workloads to batch. This is the argument that convinced me more than the discount. Half the cost is nice. Bulk work that never touches the experience of the people using your product is the actual win.

What it charges you in exchange

Nothing is free, so here is the bill, with the Bedrock specifics because that is where we run it:

Latency. Minutes or hours, with a commitment to finish within 24 hours. You do not get to choose when.
Fewer features. On Bedrock batch there is no tool calling and no structured output. You ask for JSON in the prompt and parse it yourself, with the same validation you would use for any untrusted model output.
No OCR path on several models. If part of your input is scanned images, those need an on-demand route of their own.
Output order is not guaranteed. Line 1 of the input is not line 1 of the output. The recordId is the only thing that ties them together.
A minimum per job. By default, 100 records. Fewer than that and the job is rejected.

If your use case is interactive, none of this works. If nobody is waiting for the answer, all of it is acceptable.

The checklist we learned the hard way

None of these are in the getting-started guide. All of them cost us something the first time.

Deduplicate before sending. Hash the file, not the application. One parse per CV, not one per application.
Respect the quotas. At least 100 records per job, 1 GB per input file, 5 GB per job. We shard at 50,000 records.
Merge the last shard if it comes out short. 50,050 records split naively into shards of 50,000 gives you a final shard of 50, and that job is rejected. The fix is three lines, and it is the kind of bug you only find with real data:
```
function shard<T>(items: T[], size = 50_000, min = 100): T[][] {
  const shards: T[][] = [];
  for (let i = 0; i < items.length; i += size) shards.push(items.slice(i, i + size));
  const last = shards.at(-1);
  if (shards.length > 1 && last && last.length < min) shards.at(-2)!.push(...shards.pop()!);
  return shards;
}
```
The merged shard ends up slightly over the target size, so keep the target comfortably below the real limits.
Persist each job's state as soon as it is accepted. A job is billable from the second the provider accepts it. If your script crashes after creating the job but before saving its ARN, and you relaunch, you pay twice for the same work. Write the job ARN and the shard it covers to durable storage before you do anything else, and make the launcher skip shards that already have a job.
Use short, unique identifiers. The output comes back unordered, so every line needs a recordId that maps back to your own data without ambiguity. A short hash works better than a long composite key you will have to parse later.
Validate with the same parser you use on-demand. The import should not be able to tell which path a result came from. If batch output needs its own parser, you now have two sources of truth and one of them will drift.
Delete every file in the bucket when you are done. Inputs and outputs. These are CVs, which means personal data, and nothing should stay in S3 after the import is verified. Make the cleanup part of the job, not a note in a runbook.

Who supports it

All four major providers offer it at 50% off, with different limits. As of this week:

Provider	How	Limits worth knowing
Amazon Bedrock	JSONL in S3, `CreateModelInvocationJob`	1 GB per file, 5 GB per job, minimum 100 records per job, no tool calling or structured output. Available for Claude, Nova, Llama, Mistral, DeepSeek, Qwen and others.
Anthropic	Message Batches API	Up to 100,000 requests or 256 MB per batch. Most batches finish in under an hour. Results available for 29 days.
OpenAI	Batch API, JSONL file	50,000 requests and 200 MB per file, 24-hour window, rate limits separate from the synchronous API.
Gemini	Batch Mode, inline or file	Files up to 2 GB, 24-hour target, context caching works inside the batch.

If you already use the AI SDK, there is a simpler path for small batches. Vercel's AI Gateway supports batch through experimental_startBatch, experimental_getBatchStatus and experimental_getBatchResults, with no S3 involved: the requests go inline, up to 1,000 per batch and 4.5 MB, and the same code works against OpenAI and Anthropic models.

import {
  experimental_startBatch as startBatch,
  experimental_getBatchStatus as getBatchStatus,
  experimental_getBatchResults as getBatchResults,
} from 'ai';

const batch = await startBatch({
  requests: cvs.map((cv) => ({
    id: cv.hash,
    type: 'text',
    model: modelId, // any OpenAI or Anthropic model on the gateway
    prompt: buildParsePrompt(cv.text),
  })),
});
await saveBatchReference(batch); // { version, id, provider }: persist it before anything else

// later, from a cron or a worker
const { status } = await getBatchStatus({ batch });
if (status === 'completed') {
  for await (const item of getBatchResults({ batch })) {
    if (item.status === 'succeeded') await importParsedCv(item.id, parseCv(item.text));
  }
}

Same shape as the Bedrock version, without owning a bucket, a role and the cleanup. It is ideal for small and medium batches. For tens of thousands of CVs, the file in S3 is still the right tool.

The rule

The whole decision fits in one question: is anyone waiting for this answer? If yes, on-demand. If no, batch. Migrations, mass re-scoring, data enrichment, evals, classifying historical records: none of these have a human on the other side watching a spinner, and all of them are paying double if they run on-demand.

Same model, half the price, and your users never notice. And honestly, the half price is the least of it. What matters is that the bulk work never touches the experience of the people using your product.