merkiinference

Streaming

Merki supports server-sent-events streaming on chat completion endpoints in both API families.

Quick path

  1. Set stream: true on the request.
  2. Consume data: chunks in order; the stream ends with data: [DONE].
  3. On a mid-stream cut, only delivered tokens are billed. Re-issue the request.

Details

TopicDecision
ShapesOpenAI-compatible SSE on https://api.merki.dev/v1; Anthropic-compatible SSE on https://api.merki.dev/v1/messages.
First tokenTime to first token (TTFT) per model is in the model catalog.
Mid-stream failureThe stream ends without [DONE]. Treat the partial output as delivered and billed; re-issue for the rest. See Errors and Usage.
CachingA streamed response populates the cache the same way a non-streamed one does. See Caching.

Checklist

  • [ ] Client handles a stream that ends without [DONE].
  • [ ] Partial output is usable or explicitly discarded.
  • [ ] Retries create a new request, not a resume.

Next step

Understand what a retry costs: Usage and metering.