Provider API

This page is the full specification of the API your server hosts to receive work from BatchRouter. There are two kinds of work, and each has its own endpoint:

• Batch work arrives through the OpenAI Batch API, hosted by you, with a few BatchRouter extensions. You get the whole batch and its deadline, so you can hold the jobs and run them in spare capacity. This is where most of the volume is.
• Direct requests arrive through POST /v1/chat/completions. BatchRouter calls it only when a customer sends a single request and wants the answer right away, never for batch work.

Which of the two a lane receives depends on its integration:

A chat-only lane gets no batch work. To receive batch work, host the batch endpoints below and switch the lane to openai_batch.

EndpointsCopy Anchor Link

Host these endpoints at your base URL. The base URL is the https:// URL you enter in the portal, without a trailing /v1.

A chat-only (openai_compat) lane needs GET /v1/models and POST /v1/chat/completions only.

AuthenticationCopy Anchor Link

BatchRouter sends Authorization: Bearer <your API key> on every call. It's the key you store in the provider portal, where it's encrypted. Use a dedicated key you can revoke without affecting your other customers.

Model IDsCopy Anchor Link

The model BatchRouter sends is your Model ID for that model, exactly as you entered it in the portal and exactly as GET /v1/models lists it. See Add your models.

Batch workCopy Anchor Link

A batch takes four steps: BatchRouter uploads the input file, creates the batch, polls it and reads the results. The shapes are the OpenAI Batch API's, so a server that already speaks it needs only the extensions on this page.

Input fileCopy Anchor Link

BatchRouter uploads one JSONL file per batch with POST /v1/files (purpose=batch, multipart). Each line is one request. All lines in a file are for one model and one endpoint.

{"custom_id":"item_7d2f8a1c4b9e40d3a6c5e8f1b2d4a090","method":"POST","url":"/v1/chat/completions","body":{"model":"llama-3.1-8b-instruct","messages":[{"role":"user","content":"Classify this ticket: refund not received"}],"max_tokens":512}}

• url is /v1/chat/completions or /v1/embeddings.
• body is a standard request body for that endpoint. It never contains stream.
• custom_id is BatchRouter's item ID. It is opaque, at most 64 characters and unique within the batch. Echo it exactly in the result for that line — a result whose custom_id is anything else is ignored and flags the lane. See Identity rules.

Create the batchCopy Anchor Link

BatchRouter creates the batch with POST /v1/batches:

POST /v1/batches
Authorization: Bearer <your API key>
Content-Type: application/json
Idempotency-Key: wu_3e6c9b2d5a8f41e0b7d4c1a9f0e2b385:9f2c4e6a8b1d3f50

{
  "input_file_id": "file-abc123",
  "endpoint": "/v1/chat/completions",
  "completion_window": "24h",
  "metadata": {
    "batchrouter_work_unit_id": "wu_3e6c9b2d5a8f41e0b7d4c1a9f0e2b385",
    "batchrouter_deadline_at": "2026-10-08T14:30:00Z",
    "batchrouter_window_seconds": "7200",
    "batchrouter_tier": "standard",
    "batchrouter_role": "standard",
    "batchrouter_pay_input_per_1m": "0.0400",
    "batchrouter_pay_output_per_1m": "0.0800",
    "batchrouter_pay_cached_input_per_1m": "0.0200"
  }
}

completion_window is always "24h", for compatibility. The real deadline is in metadata.

Idempotency-Key. The key is the work unit ID plus a short hash of the exact set of items being sent, so a retry of the same create (for example after a network timeout) carries the same key, while a later dispatch of a different item subset gets a new one. Treat it as opaque. If you receive a key you have already seen, return the batch you already created for it. Never create a second batch for the same key.

Metadata keys. The extensions are strings in metadata, so a stock OpenAI-compatible server accepts them unchanged.

Accept or declineCopy Anchor Link

Your response to the create request is your answer. There is no separate quote or accept step.

Decline when you can't finish the batch by batchrouter_deadline_at. Lines that finish after the deadline are not paid, and their items are reassigned.

StatusCopy Anchor Link

BatchRouter polls GET /v1/batches/{batch_id} and reads status and request_counts.

ResultsCopy Anchor Link

Each finished line has the OpenAI output-file shape, keyed by custom_id:

{"id":"batch_req_7f3a91","custom_id":"item_7d2f8a1c4b9e40d3a6c5e8f1b2d4a090","response":{"status_code":200,"request_id":"req_5c2e","body":{"id":"chatcmpl-91b","object":"chat.completion","created":1791460800,"model":"llama-3.1-8b-instruct","choices":[{"index":0,"message":{"role":"assistant","content":"billing"},"finish_reason":"stop"}],"usage":{"prompt_tokens":18,"completion_tokens":2,"total_tokens":20}}},"error":null}

• A 2xx status_code with a null error is a success.
• A non-2xx status_code or a non-null error fails that item. BatchRouter may retry it on another lane.
• usage in body is required. BatchRouter pays on it. If it is missing, BatchRouter estimates the tokens and flags the lane.

Put successful lines in the output file and failed lines in the error file, as the OpenAI Batch API does. BatchRouter reads at most 20 MB per result file. Lines past the cap are never applied, so keep each file under it.

Partial resultsCopy Anchor Link

Partial results are a BatchRouter extension. They let you hand over lines as soon as they finish, so customers see those items done before the whole batch ends.

GET /v1/batches/{batch_id}/results?after={line_id}&limit={n}
Authorization: Bearer <your API key>

The response is a list of result lines in the same shape as the output and error files:

{
  "object": "list",
  "data": [
    {"id":"batch_req_7f3a91","custom_id":"item_7d2f8a1c4b9e40d3a6c5e8f1b2d4a090","response":{"status_code":200,"request_id":"req_5c2e","body":{"object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":"billing"},"finish_reason":"stop"}],"usage":{"prompt_tokens":18,"completion_tokens":2,"total_tokens":20}}},"error":null},
    {"id":"batch_req_7f3a92","custom_id":"item_7d2f8a1c4b9e40d3a6c5e8f1b2d4a091","response":null,"error":{"code":"context_length_exceeded","message":"Prompt is 9,214 tokens; the limit is 8,192."}}
  ],
  "first_id": "batch_req_7f3a91",
  "last_id": "batch_req_7f3a92",
  "has_more": false
}

Rules:

• Lines are append-only and in the order they finished. Each line has a stable id.
• Successful and failed lines both appear.
• Each custom_id appears at most once.
• A line returned here must be identical to the same line in the final output or error file.
• BatchRouter polls this endpoint while the batch is validating, in_progress or finalizing, and applies each line as it arrives.

Support is set per lane: turn on partial results (partial_results on the lane) in the portal or at registration. If the endpoint answers 404 or 405, BatchRouter stops calling it for that batch and waits for the final files.

Identity rulesCopy Anchor Link

• custom_id echoes the item ID BatchRouter sent, exactly. Any other value — a truncated, re-cased or self-assigned id — is ignored and flags the lane.
• One result per custom_id.
• An exact duplicate of a line already received is ignored.
• A conflicting duplicate (same custom_id, different content) keeps the first result and flags the lane.
• When the batch reaches a terminal status (completed, failed, expired, cancelled), or once batchrouter_deadline_at has passed, every item without a result fails with provider_missing_result and can be reassigned to another lane.

CancelCopy Anchor Link

BatchRouter cancels with POST /v1/batches/{batch_id}/cancel. Lines finished before the cancel are still delivered and paid, so keep them in your results.

Usage and payCopy Anchor Link

BatchRouter pays for each line from its usage, at the rates in that batch's batchrouter_pay_* metadata:

• Chat completions: prompt_tokens and completion_tokens. Report cached input tokens in usage.prompt_tokens_details.cached_tokens.
• Embeddings: prompt_tokens.
• Lines finished after batchrouter_deadline_at are not paid.

Earnings are settled monthly. See Earnings and payouts.

Direct requestsCopy Anchor Link

BatchRouter calls POST /v1/chat/completions only for a customer's direct request: one request whose answer the customer is waiting for. It never sends batch work there.

The body is a standard chat completions body, with your Model ID as model.

• Customer doesn't stream: BatchRouter sends "stream": false and expects a chat.completion object with usage.
• Customer streams: BatchRouter sends "stream": true with "stream_options": {"include_usage": true} and relays your chunks to the customer as they arrive. Answer with Content-Type: text/event-stream: one data: {chat.completion.chunk} event per chunk, a final chunk that carries usage, then data: [DONE].

Streaming is server-sent events over a normal HTTP response.

Turn streaming on for a laneCopy Anchor Link

In the provider portal, open the lane and tick Streams chat completions. Through the API, set chat_streaming: true when you create or update the lane (POST and PATCH on /v1/provider-portal/offerings). It applies to any lane that serves direct requests, whether openai_compat or openai_batch.

When a customer streams, BatchRouter tries the lanes that have it on first. A lane that has not turned it on still takes streamed requests, after those, so nothing breaks if you leave it off. Turn it on once your server answers stream: true as described above. Without it, a streamed request that reaches your lane costs a failover if you answer with a plain JSON body.

What to expect:

• If your endpoint gives a connection error, times out or returns a 5xx before the first byte, BatchRouter fails over to another lane. Once you have started streaming there is no failover, so an error mid-stream fails the customer's request.
• usage is required, as for batch work. If it is missing, BatchRouter estimates the tokens, records them as estimated, and pays and bills on the estimate. A lane that keeps answering without usage can be put under review.

PricesCopy Anchor Link

The price on an openai_batch lane is its batch price, what you earn per 1 million tokens of batch work.

Every openai_batch lane also sets a direct price — direct_input_per_1m, direct_output_per_1m and, optionally, direct_cached_input_per_1m. It is required, and it does two jobs: it is what you earn for direct requests on the lane, and it is the lane's reference price — each batch price must stay below its direct price.

On a chat-only (openai_compat) lane there are no separate direct fields: the lane's regular price is its direct price.

BatchRouter adds its platform fee on top when it prices work for customers, so the price you set is what you earn.

Go-live checksCopy Anchor Link

The portal's endpoint test and Go live call your endpoint with your key. Neither creates work or costs anything.

After you go live, BatchRouter repeats the model check every day. See Test your endpoint for the usual failures and fixes.

Capacity and heartbeatsCopy Anchor Link

Capacity tells BatchRouter how much work a lane can take. You can set it in the portal, or from your own systems through the API:

Authenticate with a BatchRouter API key from the account that owns the provider, and name the provider in a header:

curl -X POST "https://api.batchrouter.com/v1/provider-portal/offerings/$OFFERING_ID/heartbeat" \
  -H "Authorization: Bearer $BATCHROUTER_API_KEY" \
  -H "x-batchrouter-provider-id: your-provider-slug" \
  -H "Content-Type: application/json" \
  -d '{"available_queue_items": 5000, "heartbeat_ttl_seconds": 300}'

Both endpoints take the capacity fields from Publish capacity, such as max_batch_items, max_concurrent_batches, available_queue_items and available_queue_tokens. A heartbeat also takes heartbeat_ttl_seconds, from 30 to 3600. Without it, the lane's configured time-to-live applies.

Running on vLLMCopy Anchor Link

vLLM serves GET /v1/models and POST /v1/chat/completions, so it covers direct requests and the model check as it is. For batch work you run a batch front end in front of it that hosts the batch endpoints on this page. vllm run-batch reads the same JSONL input format and writes the same output line format, which makes it a possible starting point for that front end.

Next stepsCopy Anchor Link