AI

AI Gateway is the most under-rated module in CyberOS. The naive read is "it's a thin proxy in front of LiteLLM." The real read is: this is the single most leveraged P0 module, because every other module that touches an LLM (CUO, CHAT, KB, PROJ, every Genie surface) inherits its cost behaviour, compliance posture, and provider resilience. Research review §2.4 explicitly reordered the build sequence to land AI Gateway at P0 · slice 1 — before AUTH — for this reason.

AI Gateway in the runtime — every LLM call flows through here

Auto vs human-in-loop operations matrix

Letting every module embed its own LLM SDK creates three problems at once. (a) Cost-tracking turns into per-module spreadsheets that never agree. (b) PII redaction becomes a decentralized policy that drifts. (c) Provider failover requires per-module change windows whenever Anthropic / OpenAI degrades. The AI Gateway pattern is the standard answer: pay the cost of one integration once, let every other module call a single typed RPC, and centralise the policy.

The bet is the same bet AUTH and BRAIN make: pay the cost once at the substrate. Without AI Gateway, each module re-implements PII redaction (and at least one of them gets it wrong), each module embeds its own SDK (and the OpenAI Python client and the Anthropic Python client disagree on streaming chunks), and the regulator's "show me every prompt that touched personal data" question becomes a forensic project. With AI Gateway, that question is a SQL query.

+ §9.7 + §11.2.1 give the full picture; this table is the working summary.

The naive AI-Gateway design treats cost as a side-effect: count tokens, send a bill. The CyberOS design treats cost as the primary purpose: every call is a fiscal event, attributed to a tenant + persona + module, capped against a budget, and surfaced as an invoice line item the moment it lands. Token accounting is not a reporting feature; it is a runtime gate that refuses a call if it cannot afford to make it.

Per-tenant policy (the cost contract)

tenant: acme-corp
ai_policy:
  monthly_cap_usd: 500
  warn_threshold: 0.80         # ping CFO + tenant admin at 80%
  hard_stop: true              # at 100%, refuse new calls
  emergency_override:
    enabled: true
    requires: ["cfo_signoff", "audit_row"]
  primary_provider: bedrock
  fallback_providers: [anthropic, openai]
  require_zdr: true
  residency: sg-1
  per_model_caps:
    bedrock:anthropic.claude-3.5-sonnet:
      max_tokens_per_call: 8000
      daily_call_cap: 5000
    bedrock:anthropic.claude-3-haiku:
      max_tokens_per_call: 4000
      daily_call_cap: 20000
  per_persona_attribution:
    cuo-cpo@0.4.1: { module: cuo, cost_centre: ops }
    cuo-cdo@0.3.0: { module: kb, cost_centre: ops }
    genie-public@1.2.0: { module: portal, cost_centre: cogs }

Pre + post-call accounting (the 7-step sequence)

Attribution dimensions — who owes for this call?

The naive approach to multi-provider is "wrapper SDK per provider." That works until your tenant in EU-1 needs Vertex (residency) and your tenant in SG-1 needs Bedrock (latency). AI Gateway is built on the LiteLLM router because it solves this once: provider auth, retries, streaming semantics, and pricing are model-aliased; consumers never know which provider answered.

Model-alias resolution table

Failover semantics — what happens when a provider degrades

Geographic residency — provider × region matrix

The compliance posture of every AI-using module is determined by what AI Gateway does before bytes leave the cluster. Four protections are stacked in order: PII redaction → persona-version stamp → ZDR check → audit row emission. Any link in the chain that fails refuses the call entirely; there is no degraded-compliance mode.

The four-link compliance chain

The ai.invocation audit row schema

{
  "seq": 18420,
  "ts_ns": 1763112131_000_000_000,
  "op": "put",
  "path": "meta/ai-invocations/2026-05-15_acme_chat-smart_a3c7.md",
  "extra": {
    "tenant_id": "org:acme-corp",
    "agent_persona": "cuo-cpo@0.4.1",
    "module": "cuo",
    "route_class": "chat",
    "model_alias": "chat.smart",
    "resolved_provider": "bedrock:claude-3.5-sonnet",
    "region": "ap-se-1",
    "failover_path": "primary",
    "cache_state": "miss",
    "prompt_hash": "sha256:a3c7…2b9f",
    "response_hash": "sha256:9f8e…1d2c",
    "usage": { "prompt_tokens": 1240, "completion_tokens": 612, "usd_cost": 0.012 },
    "redaction_applied": true,
    "redaction_count": 3,
    "zdr_confirmed": true,
    "duration_ms": 1843,
    "idempotency_key": "01HZK…"
  },
  "prev_chain": "69e6…488a",
  "chain": "8d2c…5a91"
}

Vietnamese PII recogniser — what extends Presidio

Six internal pipelines between the gRPC ingress and the provider egress: tenant policy resolution, persona stamping, PII redaction, cache lookup, router selection, and accounting. The diagram below shows the full request path for a chat.complete call.

Internal components

AI Gateway is mostly stateless — its source of truth is BRAIN (provider config, tenant policy, persona prompts) and Redis (cache, idempotency keys). DuckDB holds the cost-roll-up for fast dashboard queries. The entities below show the read + write surfaces.

Provider + model matrix (P0)

AI Gateway speaks gRPC internally and exposes a thin REST surface for non-internal callers. A federated GraphQL subgraph publishes the read-side (usage, model catalogue). No public MCP tools — the gateway is infrastructure, not directly agent-callable.

gRPC API (canonical)

syntax = "proto3";
package cyberos.ai.v1;

service AIGateway {
 / Streaming chat completion (SSE end-to-end).
 rpc ChatComplete(ChatRequest) returns (stream ChatChunk);/ Non-streaming variant for batch jobs.
 rpc ChatCompleteSync(ChatRequest) returns (ChatResponse);/ Embedding (single or batch).
 rpc Embed(EmbedRequest) returns (EmbedResponse);/ Reranking.
 rpc Rerank(RerankRequest) returns (RerankResponse);/ Cost lookup for the calling tenant.
 rpc UsageMTD(TenantRef) returns (UsageReport);/ Model catalogue.
 rpc ListModels(Empty) returns (ModelList);
}

message ChatRequest {
 repeated Message messages = 1;
 string persona = 2;/ "cuo" | "genie" | "hr-assistant"
 string route_class = 3;/ "chat" | "reasoning"
 string idempotency_key = 4;
 ModelHint hint = 5;/ optional; tenant policy may override
 bool stream = 6;
 map<string, string> metadata = 7;/ free-form, recorded in audit
}

message Message {
 string role = 1;/ "user" | "assistant" | "system" (system reserved)
 string content = 2;
 repeated Tool tool_calls = 3;
}

message ChatChunk {
 string content_delta = 1;
 bool done = 2;
 Usage usage = 3;/ emitted on final chunk
}

message Usage {
 int32 tokens_in = 1;
 int32 tokens_out = 2;
 string model_id = 3;
 string cache_state = 4;
 double usd_cost = 5;
 bool redaction_applied = 6;
 string persona_version = 7;
 string brain_chain = 8;
}

message EmbedRequest {
 repeated string inputs = 1;
 string model = 2;/ default: "bge-m3"
}

message EmbedResponse {
 repeated Embedding embeddings = 1;
 Usage usage = 2;
}

message Embedding { repeated float vector = 1; }

REST + SSE surface (planned, edge-only)

GraphQL subgraph (read-only)

extend schema
 @link(url: "https://specs.apollo.dev/federation/v2.5", import: ["@key", "@requiresScopes"])

type AIInvocation @key(fields: "id") @requiresScopes(scopes: [["ai.usage_read"]]) {
 id: ID!
 tenantId: ID!
 actor: String!
 routeClass: RouteClass!
 modelId: String!
 personaVersion: String!
 tokensIn: Int!
 tokensOut: Int!
 usdCost: Float!
 latencyMs: Int!
 cacheState: CacheState!
 redactionApplied: Boolean!
 failoverPath: String!
 ts: DateTime!
}

type UsageReport @key(fields: "tenantId month") {
 tenantId: ID!
 month: String! # "2026-05"
 totalCalls: Int!
 totalTokens: Int!
 totalUsdCost: Float!
 capUsdCost: Float!
 percentUsed: Float!
 byModel: [ModelUsage!]!
}

type ModelUsage {
 modelId: String!
 calls: Int!
 tokensIn: Int!
 tokensOut: Int!
 usdCost: Float!
}

enum RouteClass { CHAT REASONING EMBED RERANK IMAGE }
enum CacheState { MISS HIT BYPASS }

type Query {
 aiUsageMTD(tenantId: ID): UsageReport!
 aiInvocations(since: DateTime, limit: Int = 50): [AIInvocation!]!
 @requiresScopes(scopes: [["ai.usage_read"]])
 aiModels: [Model!]!
}

Flow 1 — Streaming chat completion (cache miss)

Flow 2 — Cache hit (deterministic replay)

Flow 3 — Provider failover (primary degraded)

Flow 4 — Per-tenant cost cap enforcement

Flow 5 — PII redaction (Vietnamese CCCD)

A single AI invocation traverses ten states between caller and audit row. Most of the time is in Routing (provider RTT); cache-hit paths skip from CacheCheck straight to Streaming.

Latency budget per route class

Performance NFRs from + reliability from §11.2.2. Cross-referenced at nfr-catalog.html#ai.

AI Gateway depends on three internal services and four external providers (P0). It is depended on by every CyberOS module that calls an LLM.

AI Gateway is the chokepoint for "what did the AI see, and at what cost?" — making it the regulator's first call for AI Act + PDPL questions.

AI Gateway-specific risks tracked in the risk register.

9 KPIs covering latency, cost, redaction quality, and reliability.

Two CLIs: cyberos-ai for operators (tenant policy, cost reports, model catalogue) and the standard OpenAI-compatible curl path for ad-hoc testing.

1. Quick chat call

$ curl https://ai.cyberos.com/v1/chat/completions \
 -H "Authorization: Bearer $CYBEROS_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"model":"claude-3.5-sonnet","messages":[{"role":"user","content":"summarise Q1 OKRs"}],"stream":false}'

{
 "id": "ai_01HZJ8…XK",
 "model": "bedrock:anthropic.claude-3.5-sonnet",
 "persona_version": "genie-v1.0.2",
 "choices": [{"message":{"role":"assistant","content":"Q1 OKRs are …"}}],
 "usage": {"prompt_tokens":120,"completion_tokens":450,"usd_cost":0.0075,
 "cache_state":"miss","redaction_applied":false,"failover_path":"primary"}
}

2. Operator — view MTD usage

$ cyberos-ai usage mtd --tenant stephen-personal

Tenant: stephen-personal
Month: 2026-05
─────────────────────────────
Total calls: 14,823
Tokens-in: 8.2 M
Tokens-out: 3.1 M
USD spent: $97.42 / $150.00 (64.9%)

By model:
 bedrock:claude-3.5-sonnet 11,420 calls $74.20 (76%)
 bedrock:claude-3-haiku 3,210 calls $4.80 (5%)
 bge-m3 (embed) 193 batches free

By cache state:
 hit 4,612 (31.1%) ← above target
 miss 10,211 (68.9%)

3. Operator — update tenant policy

$ cyberos-ai policy set --tenant acme-corp \
 --primary bedrock --fallback anthropic \
 --require-zdr true --cap-usd-monthly 500

[policy updated]
 tenant: acme-corp
 primary: bedrock
 fallback: anthropic
 zdr: required
 cap: $500/month
[audit] brain seq=14841

4. Operator — list models

$ cyberos-ai models list

ID ROUTE ZDR PRICE (in/out per 1k)
bedrock:anthropic.claude-3.5-sonnet chat ✓ $0.003 / $0.015
bedrock:anthropic.claude-3-haiku chat ✓ $0.00025 / $0.00125
anthropic:claude-sonnet-4.5 chat ✓ $0.003 / $0.015
openai:gpt-4o chat ✓ $0.0025 / $0.01
openai:o1-mini reason ✓ $0.003 / $0.012
self-hosted:bge-m3 embed — free
self-hosted:bge-rerank-v2-m3 rerank — free

5. Operator — failover drill

$ cyberos-ai chaos failover --provider bedrock --duration 60s

[chaos] injected 100% error rate on bedrock for 60 s
[detect] primary failure recognised @ +6.2 s
[failover] secondary (anthropic) active @ +6.4 s
[recovery] bedrock errors cleared @ +60 s
[breaker] half-open @ +90 s; closed @ +91 s
[result] (FR pending) PASSED (failover ≤ 30 s)

6. Operator — export monthly invoice

$ cyberos-ai invoice export --tenant acme-corp --month 2026-05 --output invoice.csv

[invoice] tenant=acme-corp month=2026-05 rows=14,823 written invoice.csv (1.2 MB)
[lines] by_model · by_route · by_persona · by_date