nr-llm
EN DE

Features

Streaming & Tool/Function Calling

Incremental SSE-style chat streaming and OpenAI-compatible tool calling behind one provider-agnostic interface, with budget, usage, and telemetry accounting on every path.

Why it exists

Every TYPO3 extension that wanted streamed output or function calling had to implement it per provider — parsing each vendor's SSE frames and each vendor's tool-call payload shape. nr-llm exposes both as single methods (streamChat, chatWithTools) that resolve the configured provider and translate to and from the OpenAI-compatible tool format (ADR-010), so an extension writes the integration once and works across OpenAI, Anthropic, Gemini, Ollama, OpenRouter and the rest.

Streaming used to bypass the middleware pipeline entirely: a streamed chat ran no budget pre-flight, wrote no usage row, and produced no telemetry — a live budget hole where an over-budget user could stream freely and the tokens stayed invisible. ADR-062 closes that with a dedicated streaming lifecycle so streamed calls are accounted for like every other call.

When to use it

  • Use streamChat() when perceived latency matters — long completions render token-by-token instead of after a multi-second wait.
  • Use chatWithTools() when the model needs to fetch data, run an action, or produce structured output; the model decides when to call a tool and you execute it and feed the result back.
  • Do not stream when you need fallback resilience across providers: fallback only works up to the first chunk (once a chunk is delivered a provider swap is impossible), and streamed responses are never cached.
  • Check provider capabilities first — only providers implementing StreamingCapableInterface stream, and only those implementing ToolCapableInterface accept tools.
  • For admin-driven agent runs (the bounded tool loop with per-group toggles and a live trace), drive it through the Playground / ToolLoopService rather than a single chatWithTools() call.

How it works

streamChat() returns a PHP Generator that yields string chunks as the provider produces them. Because a generator is lazy, it cannot be pushed through the response middleware pipeline (every middleware would fire against a not-yet-started stream). Instead a private StreamingDispatcher wraps the provider generator and runs a four-stage lifecycle: eager budget pre-flight before any generator is returned, provider selection with shallow fallback that primes each candidate up to its first yield, lazy re-yielding of each chunk while buffering the completion, and settlement (usage + telemetry) in a finally block so accounting lands on normal completion, mid-stream exception, and abandoned/early-break streams alike. Stream token counts are estimated (~4 chars/token) since the streaming adapters yield only text deltas.

Tool calling follows the OpenAI-compatible schema: you pass a list of tool definitions ({type: function, function: {name, description, parameters}}) to chatWithTools(). The response's toolCalls is a nullable list of ToolCall value objects whose arguments are already a JSON-decoded associative array. You echo the assistant turn back with ChatMessage::assistantToolCalls(), execute each requested tool, answer each by id with ChatMessage::toolResult(), and call chat() again so the model can incorporate the results. nr-llm ships 41 built-in read-only tools in 8 toggleable groups; a bounded agent loop (ToolLoopService::runLoop) drives multi-round tool execution and is gated by a fail-closed cascade — global per-tool state, per-tool group state, per-configuration allowed groups, and the per-run allow-list — where each layer can only narrow, never widen.

Reasoning-model output is separated from the answer: AbstractProvider::extractThinkingBlocks() pulls inline <think>…</think> blocks (DeepSeek, Qwen, local models) and Claude's native thinking content blocks into CompletionResponse::$thinking, so thinking is available via hasThinking() for debugging without polluting the main content.

LlmServiceManagerInterface

Public entry point: streamChat() / streamChatWithConfiguration() return a Generator of string chunks; chatWithTools() / chatWithToolsForConfiguration() return a CompletionResponse.

StreamingCapableInterface

Provider contract for streaming — streamChatCompletion(array $messages, array $options = []): Generator and supportsStreaming(): bool.

ToolCapableInterface

Provider contract for tools — chatCompletionWithTools(array $messages, array $tools, array $options = []): CompletionResponse and supportsTools(): bool.

StreamingDispatcher

Private wrapping generator that owns the streaming lifecycle: eager budget pre-flight, pre-first-chunk fallback priming, lazy re-yield, and usage/telemetry settlement in finally (ADR-062).

ToolCall

Readonly value object for one tool call — id, name, and arguments as an already JSON-decoded associative array; carried in CompletionResponse::$toolCalls.

ToolLoopService

Bounded agent loop (runLoop) that runs multiple tool rounds against a configuration, intersecting each per-run allow-list with the globally enabled tool set (ToolAvailabilityService).

For developers

Both features are single calls on the injected LlmServiceManagerInterface. Streaming yields strings; tool calling returns a CompletionResponse you inspect and answer.

Stream a chat response

streamChat() returns a Generator of string chunks. Echo and flush each chunk as it arrives; budget, usage and telemetry are settled automatically when the stream ends or is abandoned.

php 
$stream = $this->llmManager->streamChat($messages);

foreach ($stream as $chunk) {
    echo $chunk;
    ob_flush();
    flush();
}

Define tools in the OpenAI-compatible format

Each tool is a function entry with a JSON-Schema parameters object. The model decides when to call one based on the conversation.

php 
$tools = [
    [
        'type' => 'function',
        'function' => [
            'name' => 'get_weather',
            'description' => 'Get current weather for a location',
            'parameters' => [
                'type' => 'object',
                'properties' => [
                    'location' => [
                        'type' => 'string',
                        'description' => 'City name',
                    ],
                    'unit' => [
                        'type' => 'string',
                        'enum' => ['celsius', 'fahrenheit'],
                    ],
                ],
                'required' => ['location'],
            ],
        ],
    ],
];

Execute tool calls and feed results back

toolCalls is a list of ToolCall value objects; arguments is already a decoded array. Echo the assistant turn, answer each call by its id, then call chat() again for the final answer.

php 
use Netresearch\NrLlm\Domain\ValueObject\ChatMessage;

$response = $this->llmManager->chatWithTools($messages, $tools);

if ($response->hasToolCalls()) {
    $messages[] = ChatMessage::assistantToolCalls($response->toolCalls, $response->content);

    foreach ($response->toolCalls as $toolCall) {
        $result = match ($toolCall->name) {
            'get_weather' => $this->getWeather($toolCall->arguments['location']),
            default => throw new \RuntimeException("Unknown function: {$toolCall->name}"),
        };

        $messages[] = ChatMessage::toolResult($toolCall->id, json_encode($result, JSON_THROW_ON_ERROR));
    }

    $response = $this->llmManager->chat($messages);
}

Guard on provider capability

Not every provider streams or accepts tools. Check the capability interface before using the feature to fail early rather than at the API call.

php 
$provider = $this->llmManager->getProvider('openai');
if ($provider instanceof StreamingCapableInterface) {
    // Provider supports streaming
}

Gotchas

Fallback stops at the first chunk

The streaming dispatcher walks the configuration's fallback chain and primes candidates only up to their first yield. Once a chunk has been handed to the caller, a provider swap is impossible — so a mid-stream failure cannot be retried against another provider. This asymmetry with the non-streaming pipeline is intrinsic to streaming (ADR-062).

Streamed token counts are estimated, not reported

The streaming adapters yield only text deltas and emit no usage frame at stream end, so the dispatcher estimates tokens with a ~4 chars/token heuristic. An estimate is deliberately better than the previous silent zero for budget enforcement, but it is not the provider's exact figure. Streaming also never caches and always records cache_hit = false.

Tool availability is fail-closed and cascades

A per-run allow-list can only narrow what is globally enabled. A globally disabled tool (ToolStateRepository) or a disabled tool group can never be re-enabled by a skill or playground selection — a per-tool override cannot re-enable a tool inside a disabled group (ADR-039, ADR-043). Third-party ToolInterface implementations must implement getGroup().

No automatic tool execution

chatWithTools() returns the model's requested calls; it never runs your functions. You must dispatch on $toolCall->name, execute, and return each result via ChatMessage::toolResult() before the model can produce a final answer. Treat every tool result and model output as untrusted content.

Go deeper

← All feature deep dives