Provider Resilience — Circuit Breaker, Health, Idempotency

Circuit breaker

A new CircuitBreakerMiddleware tracks consecutive failing calls per provider. After circuitBreaker.failureThreshold consecutive tripping failures the circuit opens: for circuitBreaker.cooldownSeconds further calls to that provider fail fast with a CircuitOpenException instead of waiting on a connection timeout. After the cooldown a single half-open probe is allowed; success closes the circuit, failure re-opens it.

Tripping failures are exactly the set FallbackMiddleware already treats as retryable — ProviderConnectionException (network / timeout / 5xx / retries exhausted) and a 429 ProviderResponseException (rate limit). Client errors (other 4xx), misconfiguration and unsupported-feature errors mean the provider answered; they are not a health signal and neither trip nor reset the circuit.

Pipeline placement — innermost, priority 20. This is the load-bearing design choice, so it is spelled out:

TelemetryMiddleware      110  observes every run
  IdempotencyMiddleware  105  replays a stored result by key
    CacheMiddleware      100  payload cache; short-circuits on hit
      BudgetMiddleware    75  pre-flight budget gate
        FallbackMiddleware 50 swaps configuration on retryable failure
          UsageMiddleware  25 records the served call
            CircuitBreaker 20 guards the actual provider call   (THIS)
              <terminal>

• Inside FallbackMiddleware. An open circuit throws CircuitOpenException, which FallbackMiddleware now treats as retryable. Because the breaker sits inside the fallback loop, that exception is raised from within $next($configuration) on each attempt, so Fallback catches it and advances to the next configuration/provider. A naïve reading puts the breaker "between Budget (75) and Fallback (50)"; that would be wrong — outside Fallback, an open primary circuit would abort the whole call before any fallback ran, the opposite of the intent (skip the sick provider, use a healthy one). The breaker therefore lives below Fallback.

• Inside UsageMiddleware. The breaker wraps only the terminal, so the health signal reflects the pure provider call — usage bookkeeping (success-only, with its own error handling) never contaminates it, and a genuine success closes the circuit before any post-processing.

CircuitOpenException extends ProviderException (so it carries the adr-053 marker interface); ProviderConnectionException is final, so extending that was not an option — instead FallbackMiddleware::isRetryable() gained one explicit instanceof arm.

Circuit state storage: cache, not a table. State (consecutiveFailures + openedAt) lives in the nrllm_circuit cache via CircuitBreakerStoreInterface (no hardcoded backend — the instance's Redis/Valkey is shared across web workers, so one worker tripping a circuit protects them all). Rationale: the state is inherently transient, self-decaying (a forgotten entry reads as closed, the fail-safe default), and needs no schema, no purge command, and no write on every call. A DB table would add write load on the hot path and a maintenance surface for data that is meant to be ephemeral. The half-open single-probe gate is pragmatic (refresh the open window before probing) rather than an atomic compare-and-set, which the cache backend cannot portably offer — a few extra probes after a cooldown is acceptable. The failure counter shares this posture: it is a non-atomic get-modify-set, so under heavy concurrent failure some increments are lost and the breaker trips a little late (the count still climbs about one per cooldown window) rather than never — acceptable for the same reasons.

Provider health scoring

ProviderHealthService reads the existing telemetry log (tx_nrllm_telemetry, adr-058) over a rolling window and computes a per-provider ProviderHealthScore (success rate + mean latency into one comparable 0.0–1.0 number; success rate weighted 4:1 over latency). No new write path and no second source of truth — telemetry already records success/latency per run, so health is a read model over it. Reads go through a dedicated ProviderHealthRepository; the telemetry recorder's own interface stays append/purge-only, as adr-058 intended.

The one built-in consumer is FallbackMiddleware, which asks ProviderHealthService::reorder() to prefer healthier providers among the fallback candidates. It is a hint, opt-in, and minimal-invasive:

• Gated by health.reorderFallback — OFF by default. When off, the chain is returned untouched with no telemetry query, so the configured fallback order stays the default.

• When on, it is a stable sort by descending health: providers of equal (or unknown) health keep their configured order, and no candidate is ever dropped. A provider with no telemetry scores neutral, never unhealthy — an un-exercised provider must not sink just for lack of data.

A pure config-order-primary "tie-break" would be a no-op on a totally-ordered chain (there are no ties to break), so health-primary reordering is offered as the opt-in instead; the default-off flag is what keeps existing behaviour identical.

Idempotency keys

An optional idempotency key on any options object (AbstractOptions::withIdempotencyKey()) makes a repeated request return the stored result instead of calling the provider again. A new IdempotencyMiddleware (priority 105 — just inside Telemetry, outside Cache/Budget, so a replay short-circuits the behavioural stack and is not re-charged, yet is still observed) stores the result under the key and replays it on the next call with the same key. Failed calls and streaming generators are never stored.

Cache, not a table — and not an overload of the existing cache key. The task hypothesis ("make it a thin layer on CacheMiddleware's key") was investigated and rejected: CacheMiddleware is deliberately array-only (it persists array<string, mixed>), so it cannot round-trip the typed responses (CompletionResponse, …) that the chat/completion paths return — which is the case idempotency actually matters for. The dedicated middleware stores over its own nrllm_idempotency VariableFrontend, which serialises any response value, so idempotency works for every operation while staying cache-backed: idempotency results are transient and TTL-bounded by nature, so a table (with its purge/TCA surface) would be the wrong store.

OpenTelemetry — deliberately not built

OTel exporters/collectors are out of scope for a TYPO3 extension. The extension should not own an OTLP exporter, a collector endpoint, or trace sampling configuration — that is the host instance's observability responsibility (the same instance that owns logging, APM, and metrics scraping). What the extension can own — durable per-request metrics — already exists as tx_nrllm_telemetry (adr-058) with a correlation id per run; a host that runs OTel can scrape or forward that. Building an in-extension OTel pipeline would duplicate host infrastructure, couple the extension to an exporter SDK, and add configuration the operator already manages centrally.