nr-llm
EN DE

Features

Provider Abstraction, Fallback & Encrypted Keys

One interface across seven LLM providers, with configuration-level fallback and API keys stored as nr-vault UUIDs rather than plaintext.

Why it exists

Each LLM provider ships a different endpoint, auth scheme, request/response shape, model naming, and capability set. nr-llm hides those differences behind a single contract (ProviderInterface) so consumers call one API regardless of whether OpenAI, Claude, Gemini, Groq, Mistral, Ollama, or OpenRouter answers. Switching providers is an admin change, not a code change (ADR-001).

Credentials, model catalogs, and use-case prompts change on different schedules and for different reasons, so they are normalized into three tiers: Provider (one API connection = one key = one billing relationship), Model (per-provider capabilities and pricing), and Configuration (system prompt, temperature, and other tunables for a use case). This lets a site keep separate prod/dev keys for the same adapter type and reuse model definitions across configurations (ADR-013).

API keys must be retrievable to authenticate, so they cannot be hashed. The original approach encrypted them at application level with sodium (ADR-012); that ADR is now Superseded — keys are stored as nr-vault identifiers (UUIDs) and resolved, injected, audited, and memory-scrubbed inside the vault, so the plaintext key never surfaces in this extension's code.

When to use it

  • You are adding AI to a TYPO3 extension and want provider selection, key handling, caching, and error handling managed for you — inject LlmServiceManagerInterface and call complete()/chatCompletion().
  • You need the same request to survive a provider outage: list sibling configurations on a configuration's fallback chain to retry on connection errors, HTTP 5xx, or 429 rate limits.
  • You must keep provider credentials out of the database, YAML, and logs — store the nr-vault UUID identifier, never the raw key.
  • You are integrating a new provider: implement ProviderInterface (or extend AbstractProvider), mark it with #[AsLlmProvider], and it auto-registers via ProviderCompilerPass.
  • You run multiple accounts of one provider type (prod/dev/backup) or custom endpoints (Azure OpenAI, Ollama, vLLM) that the three-tier model represents as distinct Provider rows.

How it works

ProviderInterface is the core contract (getName, getIdentifier, configure, chatCompletion, complete, embeddings, testConnection, and capability checks). Optional features are opt-in capability interfaces — VisionCapableInterface, StreamingCapableInterface, ToolCapableInterface, DocumentCapableInterface — detected by instanceof, while embeddings remain a core method. AbstractProvider supplies shared behavior and every shipped adapter extends it.

Registration is attribute-driven: a class marked #[AsLlmProvider(priority: N)] in the Netresearch\NrLlm\ namespace is auto-tagged nr_llm.provider by ProviderCompilerPass at container-compile time (ADR-022). Providers are kept private; nothing resolves them by class name — access is via LlmServiceManager, resolved by the identifier returned from getIdentifier(). Priority is an ordering hint only. Third-party providers outside the namespace keep the legacy Services.yaml tag path.

The three tiers are Extbase entities over tx_nrllm_provider, tx_nrllm_model, and tx_nrllm_configuration. A Configuration references a Model, which references a Provider that owns the connection (endpoint_url, adapter_type, timeout, max_retries, and api_key stored as a vault UUID). Calls run through a middleware pipeline (ADR-026) ordered by tag priority: Guardrail (115), Telemetry (110), Idempotency (105), Cache (100), Budget (75), Fallback (50), Usage (25), CircuitBreaker (20).

FallbackMiddleware runs the primary configuration; on a retryable failure it walks the configuration's FallbackChain of sibling configuration identifiers, skipping not-found and inactive ones, until one succeeds or the chain is exhausted (FallbackChainExhaustedException). Retryable is ProviderConnectionException, a 429 ProviderResponseException, or CircuitOpenException. AbstractProvider::getHttpClient() authenticates through nr-vault's HTTP client (vault->http()->withAuthentication(identifier, placement, options)); api-key-less providers such as Ollama use the raw factory client behind an explicit SSRF host check.

Netresearch\NrLlm\Provider\Contract\ProviderInterface

Core provider contract every adapter implements — name, identifier, configure, chatCompletion, complete, embeddings, testConnection, capability checks.

Netresearch\NrLlm\Provider\AbstractProvider

Shared base for the seven adapters; stores apiKeyIdentifier and builds the vault-authenticated HTTP client in getHttpClient().

Netresearch\NrLlm\Attribute\AsLlmProvider

#[AsLlmProvider(priority)] marker; ProviderCompilerPass auto-tags it nr_llm.provider so no Services.yaml entry is needed.

Netresearch\NrLlm\Domain\DTO\FallbackChain

Immutable, normalized ordered list of LlmConfiguration identifiers to try when the primary fails; shallow (a fallback's own chain is ignored).

Netresearch\NrLlm\Provider\Middleware\FallbackMiddleware

Runtime enforcer (tag priority 50): retries the chain on retryable errors, skips inactive/missing configs, throws FallbackChainExhaustedException when all fail.

Netresearch\NrVault\Service\VaultServiceInterface

Resolves the stored UUID identifier to the real secret and injects it at send time; keeps plaintext keys out of nr-llm's code and database.

For developers

Consume through LlmServiceManagerInterface, add providers via the attribute, keep keys as vault UUIDs, and declare fallbacks on the configuration.

Consume the unified service

Inject LlmServiceManagerInterface and call it. Provider selection, key resolution, caching, and error handling are handled by the pipeline — you never touch an HTTP client or a provider class directly.

php 
use Netresearch\NrLlm\Service\LlmServiceManagerInterface;

class MyController
{
    public function __construct(
        private readonly LlmServiceManagerInterface $llm,
    ) {}

    public function summarizeAction(string $text): string
    {
        return $this->llm->complete("Summarize: {$text}")->content;
    }
}

Add a provider

Implement ProviderInterface (in practice, extend AbstractProvider and add the capability interfaces you support), mark the class with #[AsLlmProvider] so ProviderCompilerPass auto-registers it, return the identifier from getIdentifier(), and add the icon at Resources/Public/Icons/provider-<identifier>.svg.

php 
#[AsLlmProvider(priority: 100)]
final class OpenAiProvider extends AbstractProvider implements
    VisionCapableInterface,
    StreamingCapableInterface,
    ToolCapableInterface
{
    public function getName(): string
    {
        return 'OpenAI';
    }

    public function getIdentifier(): string
    {
        return 'openai';
    }
}

Authenticate through nr-vault

configure() reads apiKeyIdentifier (a vault UUID), not a raw key. isAvailable() is true only when the identifier is set and the vault holds it. getHttpClient() then builds a client via vault->http()->withAuthentication(...), so the secret is injected and audited inside the vault; api-key-less providers (Ollama) take the raw factory client behind an SSRF host check.

php 
protected function getHttpClient(?int $timeout = null): ClientInterface
{
    if ($this->configuredHttpClient !== null) {
        return $this->configuredHttpClient;
    }

    $effective = ($timeout !== null && $timeout > 0) ? $timeout : $this->timeout;

    if ($this->apiKeyIdentifier === '') {
        $this->assertEndpointHostAllowed();
        return $this->httpClientFactory->create($effective > 0 ? $effective : null);
    }

    $client = $this->vault->http()->withAuthentication(
        $this->apiKeyIdentifier,
        $this->getSecretPlacement(),
        $this->getSecretPlacementOptions(),
    );

    return $effective > 0 ? $client->withTimeout($effective) : $client;
}

Declare a fallback chain

A FallbackChain is an ordered, normalized (trimmed + lowercased) list of sibling LlmConfiguration identifiers. Build it immutably with withLink(); FallbackMiddleware walks it on retryable failures (connection error, 5xx, 429, or an open circuit), skipping the primary, and inactive or missing entries.

php 
$chain = (new FallbackChain())
    ->withLink('blog-summarizer-openai')
    ->withLink('blog-summarizer-claude');

// Persisted on the primary configuration; walked by FallbackMiddleware
// (shallow: a fallback configuration's own chain is ignored).

Gotchas

Fallback is shallow

A fallback configuration's own fallback chain is ignored, by design, to prevent recursion and cycles. Only the primary configuration's chain is walked. If the chain contains only the primary identifier, it becomes empty after filtering and the primary's original error is rethrown verbatim rather than a 'chain exhausted' error.

Streaming requests are not routed through fallback

Once chunks have been emitted to the caller a provider cannot be swapped mid-stream, so FallbackMiddleware excludes streaming (Generator-returning) calls. Build the pipeline without FallbackMiddleware for streaming, or short-circuit on ProviderCallContext::operation === Stream.

Keys are vault UUIDs, never plaintext

The api_key column stores an nr-vault identifier, not a secret. ADR-012's application-level sodium encryption is Superseded by the nr-vault integration. Never put a raw key in TCA, YAML, env, or logs; ErrorMessageSanitizerTrait::sanitizeErrorMessage() strips secret-bearing query params before messages are surfaced.

Attribute auto-registration is namespace-scoped

ProviderCompilerPass only reflects service definitions whose class is in the Netresearch\NrLlm\ namespace. Third-party providers outside that namespace must keep the legacy Services.yaml tag (name: nr_llm.provider, priority: N), which is fully supported and takes precedence when both are present. Priority is an ordering hint only — providers are resolved by identifier at runtime.

Go deeper

← All feature deep dives