Contact

Features

Streaming & Tool/Function Calling

Incremental SSE-style chat streaming and OpenAI-compatible tool calling behind one provider-agnostic interface, with budget, usage, and telemetry accounting on every path.

Why it exists

Every TYPO3 extension that wanted streamed output or function calling had to implement it per provider — parsing each vendor's SSE frames and each vendor's tool-call payload shape. nr-llm exposes both as single methods (streamChat, chatWithTools) that resolve the configured provider and translate to and from the OpenAI-compatible tool format (ADR-010), so an extension writes the integration once and works across OpenAI, Anthropic, Gemini, Ollama, OpenRouter and the rest.

Streaming used to bypass the middleware pipeline entirely: a streamed chat ran no budget pre-flight, wrote no usage row, and produced no telemetry — a live budget hole where an over-budget user could stream freely and the tokens stayed invisible. ADR-062 closes that with a dedicated streaming lifecycle so streamed calls are accounted for like every other call.

When to use it

  • Use streamChat() when perceived latency matters — long completions render token-by-token instead of after a multi-second wait.
  • Use chatWithTools() when the model needs to fetch data, run an action, or produce structured output; the model decides when to call a tool and you execute it and feed the result back.
  • Do not stream when you need fallback resilience across providers: fallback only works up to the first chunk (once a chunk is delivered a provider swap is impossible), and streamed responses are never cached.
  • Check provider capabilities first — only providers implementing StreamingCapableInterface stream, and only those implementing ToolCapableInterface accept tools.
  • For admin-driven agent runs (the bounded tool loop with per-group toggles and a live trace), drive it through the Playground / ToolLoopService rather than a single chatWithTools() call.

How it works

streamChat() returns a PHP Generator that yields string chunks as the provider produces them. Because a generator is lazy, it cannot be pushed through the response middleware pipeline (every middleware would fire against a not-yet-started stream). Instead a private StreamingDispatcher wraps the provider generator and runs a four-stage lifecycle: eager budget pre-flight before any generator is returned, provider selection with shallow fallback that primes each candidate up to its first yield, lazy re-yielding of each chunk while buffering the completion, and settlement (usage + telemetry) in a finally block so accounting lands on normal completion, mid-stream exception, and abandoned/early-break streams alike. Stream token counts are estimated (~4 chars/token) since the streaming adapters yield only text deltas. Every stream also passes the mandatory secret-redaction guardrail live (ADR-088): a sliding window holds back the last 128 bytes until more text arrives, so the chunks a caller receives do not line up with the provider's deltas, and an end-of-stream audit of the first 50,000 bytes can log a guardrail match but cannot retract text already sent.

Tool calling follows the OpenAI-compatible schema: you pass a list of tool definitions ({type: function, function: {name, description, parameters}}) to chatWithTools(). The response's toolCalls is a nullable list of ToolCall value objects whose arguments are already a JSON-decoded associative array. You echo the assistant turn back with ChatMessage::assistantToolCalls(), execute each requested tool, answer each by id with ChatMessage::toolResult(), and call chat() again so the model can incorporate the results. On the GPT-6 models the OpenAI adapter sends a tool request to the Responses API instead of chat/completions when the endpoint is OpenAI's own host; a compatible gateway stays on chat/completions unless the configuration's options set "openai_tools_transport": "responses" (ADR-203). The provider's own items for that turn come back in $response->metadata['nrllm_provider_items'] and go into assistantToolCalls() as its third argument, so the model keeps its reasoning between steps. Gemini's thought signatures travel in the same slot (ADR-222). nr-llm ships 63 built-in tools in 10 toggleable groups — 45 read-only, plus eighteen writing tools that ship disabled and pause for human approval; a bounded agent loop (ToolLoopService::runLoop) drives multi-round tool execution and is gated by ToolCallPolicy, a fail-closed composite gate — global per-tool state, per-tool group state, admin-only tools, per-configuration allowed groups, the per-run allow-list, and the trust zone of the configuration's provider against the tool's data class (ADR-094) — where each layer can only narrow, never widen.

Reasoning-model output is separated from the answer: AbstractProvider::extractThinkingBlocks() pulls inline <think>…</think> blocks (DeepSeek, Qwen and other models that emit them) out of the content, ClaudeProvider adds Claude's native thinking content blocks, GeminiProvider its thought parts, and OllamaProvider reads Ollama's separate message.thinking field. All of it lands in CompletionResponse::$thinking, available via hasThinking() for debugging without polluting the main content. How much a model reasons is a request option: ChatOptions::withReasoningEffort() takes a ReasoningEffort case (ADR-204); today only the OpenAI GPT-6 models act on it, and the effort actually applied is recorded in the response metadata.

LlmServiceManagerInterface

Public entry point: streamChat() / streamChatWithConfiguration() return a Generator of string chunks; chatWithTools() / chatWithToolsForConfiguration() return a CompletionResponse.

StreamingCapableInterface

Provider contract for streaming — streamChatCompletion(array $messages, array $options = []): Generator and supportsStreaming(): bool.

ToolCapableInterface

Provider contract for tools — chatCompletionWithTools(array $messages, array $tools, array $options = []): CompletionResponse and supportsTools(): bool.

StreamingDispatcher

Private wrapping generator that owns the streaming lifecycle: eager budget pre-flight, pre-first-chunk fallback priming, lazy re-yield, and usage/telemetry settlement in finally (ADR-062).

ToolCall

Readonly value object for one tool call — id, name, and arguments as an already JSON-decoded associative array; carried in CompletionResponse::$toolCalls.

ToolLoopService

Bounded agent loop (runLoop) that runs multiple tool rounds against a configuration, offering only the tools the composite gate ToolCallPolicy admits — the per-run allow-list intersected with the globally enabled set (ToolAvailabilityService) and the remaining gates.

For developers

Both features are single calls on the injected LlmServiceManagerInterface. Streaming yields strings; tool calling returns a CompletionResponse you inspect and answer.

Stream a chat response

streamChat() returns a Generator of string chunks. Echo and flush each chunk as it arrives; budget, usage and telemetry are settled automatically when the stream ends or is abandoned.

php 
$stream = $this->llmManager->streamChat($messages);

foreach ($stream as $chunk) {
    echo $chunk;
    ob_flush();
    flush();
}

Define tools in the OpenAI-compatible format

Each tool is a function entry with a JSON-Schema parameters object. The model decides when to call one based on the conversation.

php 
$tools = [
    [
        'type' => 'function',
        'function' => [
            'name' => 'get_weather',
            'description' => 'Get current weather for a location',
            'parameters' => [
                'type' => 'object',
                'properties' => [
                    'location' => [
                        'type' => 'string',
                        'description' => 'City name',
                    ],
                    'unit' => [
                        'type' => 'string',
                        'enum' => ['celsius', 'fahrenheit'],
                    ],
                ],
                'required' => ['location'],
            ],
        ],
    ],
];

Execute tool calls and feed results back

toolCalls is a list of ToolCall value objects; arguments is already a decoded array. Echo the assistant turn together with the provider's replay items, answer each call by its id, then call chat() again for the final answer.

php 
use Netresearch\NrLlm\Domain\ValueObject\ChatMessage;

$response = $this->llmManager->chatWithTools($messages, $tools);

if ($response->hasToolCalls()) {
    $messages[] = ChatMessage::assistantToolCalls(
        $response->toolCalls,
        $response->content,
        $response->metadata['nrllm_provider_items'] ?? null,
    );

    foreach ($response->toolCalls as $toolCall) {
        $result = match ($toolCall->name) {
            'get_weather' => $this->getWeather($toolCall->arguments['location']),
            default => throw new \RuntimeException("Unknown function: {$toolCall->name}"),
        };

        $messages[] = ChatMessage::toolResult($toolCall->id, json_encode($result, JSON_THROW_ON_ERROR));
    }

    $response = $this->llmManager->chat($messages);
}

Guard on provider capability

Not every provider streams or accepts tools. Check the capability interface before using the feature to fail early rather than at the API call.

php 
$provider = $this->llmManager->getProvider('openai');
if ($provider instanceof StreamingCapableInterface) {
    // Provider supports streaming
}

Gotchas

Fallback stops at the first chunk

The streaming dispatcher walks the configuration's fallback chain and primes candidates only up to their first yield. Once a chunk has been handed to the caller, a provider swap is impossible — so a mid-stream failure cannot be retried against another provider. This asymmetry with the non-streaming pipeline is intrinsic to streaming (ADR-062).

Streamed token counts are estimated, not reported

The streaming adapters yield only text deltas and emit no usage frame at stream end, so the dispatcher estimates tokens with a ~4 chars/token heuristic. An estimate is deliberately better than the previous silent zero for budget enforcement, but it is not the provider's exact figure. Streaming also never caches and always records cache_hit = false.

Tool availability is fail-closed and cascades

A per-run allow-list can only narrow what is globally enabled. A globally disabled tool (ToolStateRepository) or a disabled tool group can never be re-enabled by a skill or playground selection — a per-tool override cannot re-enable a tool inside a disabled group (ADR-039, ADR-043). A tool whose data class lies above the trust zone of the configuration's provider is withheld as well; the extension setting tools.dataClassEnforcement ships as enforce (ADR-094). Third-party ToolInterface implementations must implement getGroup().

No automatic tool execution

chatWithTools() returns the model's requested calls; it never runs your functions. You must dispatch on $toolCall->name, execute, and return each result via ChatMessage::toolResult() before the model can produce a final answer. Treat every tool result and model output as untrusted content.

Go deeper

← All feature deep dives