Streaming Implementation

Status

Accepted (2024-03)

Context

Streaming responses provide:

• Better UX for long responses.

• Lower time-to-first-token.

• Real-time feedback.

Decision

Use PHP Generators for streaming:

Implementation details:

• Server-Sent Events (SSE) parsing.

• Chunked transfer encoding.

• Memory-efficient iteration.

• Provider-specific adaptations.

Consequences

Positive:

• ●● Memory efficient.

• ● Natural iteration syntax.

• ●● Real-time output.

• ◐ Works with output buffering.

Negative:

• ✕ No response object until complete.

• ◑ Error handling complexity.

• ◑ Connection management.

• ✕ No caching possible.

Net Score: +3.5 (Positive impact - streaming UX benefits outweigh implementation complexity)

Update (2026-07): streaming is in the request lifecycle

The generator mechanism decided here is unchanged, but streamed calls no longer sidestep the cross-cutting concerns the non-streaming path enforces. Originally a streamed call ran no budget pre-flight and produced neither a usage nor a telemetry row — a live budget hole.

Streamed calls now run through a dedicated streaming lifecycle (adr-062): an eager budget pre-flight before the first chunk, pre-first-chunk provider fallback, time-to-first-token measurement, and a finally that records usage (tx_nrllm_service_usage) and telemetry (tx_nrllm_telemetry) on every exit — completion, exception, or an abandoned generator. The lifecycle wraps the generator; the public Generator<int, string, mixed, void> contract is unchanged.

Two of the "Negative" consequences above are now bounded rather than open: "error handling complexity" and "connection management" are handled once, in the lifecycle, instead of at each call site. "No response object until complete" and "no caching possible" remain intrinsic to streaming. Token usage on this path is estimated (providers expose no usage frame mid-stream); see adr-062 for the rationale and the follow-up toward exact figures.