Enterprise-local inference needs a control plane: a reference architecture for governed AI

“Run it locally” sounds like an infrastructure decision. In production, it is a placement, trust and state-management decision.

An enterprise can own the GPU and still depend on a remote identity service, model registry, policy engine or DNS zone. It can keep model weights on premises while leaking sensitive prompts into telemetry. It can deploy four local replicas and discover that all four share one switch, key-management service and failure domain. It can also force every task onto scarce local accelerators when a sanitized subproblem could safely use elastic cloud capacity.

The right question is therefore not local or cloud? It is:

Where can this specific task execute while preserving its authoritative context, meeting its capability and latency requirements, and producing evidence that policy was followed?

That question leads to a different architecture: a federation of autonomous inference cells across device, site, private datacenter and approved cloud tiers. A shared control plane distributes signed desired state. Each cell makes bounded local decisions and continues operating when the wide-area control plane is unavailable.

Enterprise-local inference works as a governed federation: context remains anchored in its trust zone, while policy determines whether computation may move toward it or a sanitized subproblem may cross a boundary.

The architectural mistake: placing models instead of placing work

Most local-inference discussions begin with a model and ask where its weights will fit. That is necessary capacity planning, but it is not architecture.

The difficult state is often elsewhere:

• the source document and its access-control list;
• retrieved passages and the exact index version;
• the active conversation and its retention policy;
• tool credentials and approval state;
• KV or prefix cache accumulated during a long session;
• audit evidence explaining why this model, location and policy were selected.

Moving a request away from that state can increase disclosure risk, latency, recomputation and recovery cost even when remote tokens look cheaper. The more useful rule is:

Place inference around authoritative context, then choose the smallest eligible capability that can complete the task.

This is not a mandate to answer every request with a small local model. A device model may classify, redact, retrieve or compress locally before passing an approved subproblem to a larger private or cloud model. A site cell may continue an operational workflow during WAN loss. A private datacenter may serve shared enterprise models. An approved cloud tier may handle rare reasoning tasks or bursts—but only when policy explicitly permits the destination.

Four execution tiers, one logical cell

A durable platform repeats the same logical unit at different scales.

1. Device cell

A laptop, workstation, vehicle or industrial terminal can handle private interaction, extraction, classification, redaction, offline assistance and speculative drafts. Its strengths are zero network round-trip, direct proximity to user context and continued operation during disconnection.

Its constraints are equally real: limited memory, thermal throttling, battery limits, hardware diversity and weaker fleet observability. Device execution should therefore be a governed capability class, not an assumption that any downloaded model is safe.

2. Site or edge cell

A factory, hospital, branch, mine or retail site benefits from low-latency retrieval and inference near operational data. This tier can preserve continuity during WAN degradation and prevent high-volume sensor or video streams from leaving the facility.

But a small accelerator pool creates queueing and maintenance risks. A site also needs local identity-validation material, artifact caches, policy snapshots, DNS, time, retrieval indexes and a telemetry buffer. A GPU in the building does not by itself create an autonomous site.

3. Private-datacenter cell

The datacenter tier provides shared governed models, high-volume RAG, sensitive document processing and enterprise tool access. It can support stronger isolation, clustered capacity and centralized operations.

Its main risks are concentration and contention: one inference gateway can become a global blast radius, and nominal GPU utilization can hide KV-cache exhaustion or tail-latency collapse. Capacity must be admitted against token memory, queue age and deadlines—not merely the number of free GPUs.

4. Approved cloud capability tier

Cloud remains valuable for elastic bursts, evaluation, temporary capacity and tasks that genuinely require larger models. In this architecture, however, cloud is not transparent failover. It is another trust zone with distinct residency, retention, contractual and cost properties.

Overflow must be an explicit policy outcome. If crossing the boundary changes data treatment, model behavior or audit requirements, the application needs to know.

What every inference cell contains

Each cell should be a security and failure boundary with:

1. A local inference gateway for authentication, classification and admission.
2. One or more model runtimes qualified for specific capability classes.
3. A model and artifact cache addressed by immutable digest.
4. A local context plane for permitted retrieval, tools and conversation state.
5. A policy snapshot with activation, expiry and offline rules.
6. Admission control and queues that understand token and KV-cache demand.
7. A bounded telemetry buffer that does not become a shadow prompt corpus.
8. An append-only decision journal recording model, runtime, location, policy and evidence versions.

The cell should keep already-authorized services running when global management is unavailable. It should not require a synchronous round trip to a central policy engine for every token request.

That requirement rules out a “single hybrid cluster” stretched across unreliable WAN links. Cross-site quorum, registry, identity and networking dependencies create hidden coupling precisely when the local tier is expected to survive disruption.

The global control plane distributes desired state

Central services remain important, but their job is governance and fleet coordination:

• model, tokenizer and adapter catalogue;
• task-specific evaluation evidence;
• signed artifact digests and provenance;
• placement and residency policies;
• identity and attestation reference values;
• accelerator inventory and approved hardware classes;
• budget, quota and unit-cost policy;
• rollout rings, rollback and revocation;
• aggregated audit evidence.

The control plane publishes signed, versioned bundles. Cells validate and cache them. Updates carry activation and expiry times. Emergency artifact or credential revocation should have a narrow, separately protected distribution path.

Global unavailability may pause new releases and policy changes. It should not automatically stop an approved local workflow.

Placement is constraint solving, not load balancing

A conventional load balancer chooses among interchangeable endpoints. Enterprise inference endpoints are not interchangeable: models differ in capability; sites differ in trust; replicas hold different adapters and prefixes; cloud destinations have different retention terms.

The placement process must apply hard exclusions first, then optimize among eligible cells.

Hard constraint 1: data boundary

Can the raw prompt, retrieved passages, tool results or output leave the device, facility, legal entity or jurisdiction? The initial prompt may be low sensitivity while retrieved records are restricted. Classification must continue through the workflow rather than occur only at ingress.

Hard constraint 2: measured capability

The task needs a minimum measured capability—not a generic leaderboard score. Eligibility can include modality, language, context length, structured output, tool use, adapter availability and task-specific quality thresholds.

Models should be exposed as capability classes such as internal-document-extraction-v3 or regulated-case-summary-v2, not as arbitrary vendor model names.

Hard constraint 3: operational envelope

Define maximum time to first token, inter-token latency, total deadline, disconnected-operation requirement and recovery objective for conversation state. Interactive assistance, overnight extraction and machine safety decisions are different placement problems.

Hard constraint 4: execution trust

A request may require a specific hardware or firmware posture, signed artifacts, isolated GPU allocation, or successful attestation before keys are released. Network location alone is not proof of trust. NIST SP 800-207 explicitly rejects implicit trust based solely on physical or network location or asset ownership.[1]

Hard constraint 5: regulation and evidence

Some workflows require defined logging, human oversight, retention or deletion. These constraints depend on the use case and jurisdiction. The EU AI Act’s obligations for covered high-risk systems should not be generalized to every chatbot, but they cannot be bolted on after deployment either.[2]

Rank only the eligible cells

Once policy removes ineligible destinations, rank the survivors using operational signals:

• State locality: proximity to source data, retrieval index, tools and active prefix state.
• Warmness: model and adapter loaded, prefix present, enough KV-cache headroom.
• Saturation: queue depth, token rate, memory pressure, thermal state and errors.
• Marginal cost: energy and reserved-capacity opportunity cost locally; tokens, endpoints, networking and egress remotely.
• Failure independence: whether primary and fallback share the same WAN, identity, registry, switch, KMS or power domain.
• Information loss: quality cost of redaction, summarization, quantization or smaller-model fallback.

Kubernetes Dynamic Resource Allocation can express accelerator claims and administrative device classes, and Kubernetes documents DRA as stable from v1.35.[3] That is useful inside a cluster. It is not a multi-site inference-placement policy.

The Gateway API Inference Extension adds InferencePool and endpoint-selection patterns that can consider queue state, KV capacity, adapters and cached prefixes.[4] This is closer to inference-aware routing than round robin, but adopters should pin a release and verify its API maturity. An official project is not the same thing as a universally stable cross-vendor standard.

The router optimizes only after policy establishes eligibility. If no cell qualifies, the result is an explicit degraded mode—not silent overflow to a different trust zone.

Bind the complete execution release

A model name is not a reproducible deployment unit. The safe release unit is:

model + tokenizer + quantization + runtime + adapter + system prompt + retrieval policy + tool schema + safety policy

Changing any element can alter behavior, memory demand or output format. The selected request should be bound to exact versions of all these elements, plus the retrieval-index version and active placement policy.

OCI distribution supports content-addressed blobs and digest verification and is content-type agnostic, making it a useful transport substrate for model-related artifacts.[5] SLSA provenance can describe where, when and how a build artifact was produced.[6] Neither proves that a model’s training data is lawful or its behavior safe. Artifact integrity and behavioral assurance are different controls.

A joint 2026 advisory from ASD and partner national agencies treats the AI/ML supply chain as training data, models, software, infrastructure, hardware and third-party services—not just model files. It recommends AI bills of materials, supplier assessment, threat modelling and continuing validation.[7]

Memory is the real capacity boundary

A model’s weight size is only the floor. Approximate dense weight storage is:

parameters × bits per weight ÷ 8

An 8-billion-parameter model is roughly 16 GB at 16-bit, 8 GB at 8-bit and 4 GB at 4-bit before runtime overhead. A 70-billion-parameter model is roughly 140 GB, 70 GB and 35 GB respectively.

Those numbers exclude quantization metadata, unquantized layers, allocator reservations, activations, communication buffers, CUDA graphs and KV cache. They must not be used as purchasing requirements.

KV cache grows with retained sequence length and active sequences. Hugging Face documents key and value tensors per layer, which makes the relationship linear even though grouped-query attention, paged allocation, prefix reuse and sliding windows change the observed total.[8]

This has three consequences:

1. A service can run out of useful concurrency while weights still fit comfortably.
2. Long contexts can trigger preemption and recomputation, damaging tail latency.
3. Routing should use queue and KV headroom, not GPU utilization alone.

vLLM documents the trade-off between time to first token, inter-token latency and aggregate throughput when tuning batched-token limits.[9] There is no universal best value. Test the real joint distribution of prompt length, output length and arrival bursts, then report p50, p95 and p99—not one offline tokens-per-second number.

Isolation choices are architectural choices

“Shared GPU” can mean very different things.

• Exclusive allocation is simplest for hard latency and strong operational isolation.
• MIG provides hardware-level memory and fault isolation with fixed partition shapes.
• Time-slicing oversubscribes a GPU but, as NVIDIA documents, does not provide memory or fault isolation between replicas.[10]

Time-slicing can be useful for trusted development or low-duty-cycle tasks. It is not a substitute for hard multi-tenant isolation, and requesting multiple replicas does not guarantee proportional compute. Keep separate resource classes for full-GPU production, MIG-isolated services and time-sliced development workloads so an SLO-critical service cannot silently land on the wrong class.

Disaggregated serving is topology, not magic

Prefill and decode stress different resources. Prefill is generally compute-heavy; decode is often memory-bandwidth and KV-capacity sensitive. Disaggregated serving places them in separate worker pools and transfers KV state between them.

This can isolate long prompts from ongoing generation and let each pool scale independently. It also introduces a large state transfer, another routing stage and network-topology requirements. NVIDIA Dynamo’s documentation explicitly notes that aggregated workers can be better at low concurrency and that the optimum depends on model, quantization, accelerator and traffic distribution.[11]

For enterprise-local inference, the rule is simple: keep tensor-parallel and KV-transfer paths inside a measured low-latency fabric. Do not casually disaggregate across sites or trust zones.

Security: attest before releasing the valuable thing

For the highest-sensitivity workloads, hardware-backed attestation can gate model keys, sensitive data or tool credentials.

RFC 9334 defines an architecture with an Attester, Verifier, Evidence, Attestation Result and Relying Party.[12] It deliberately does not prescribe one wire protocol. Implementations remain heterogeneous.

A practical sequence is:

1. The cell produces hardware-backed evidence.
2. A verifier appraises firmware, runtime, loader and configuration against approved reference values.
3. A relying party authorizes one operation.
4. KMS releases a short-lived key or credential.

Attestation is not behavioral safety. A perfectly measured runtime can still execute an unsafe model or overprivileged tool. Retain ordinary zero-trust identity, authorization, sandboxing, evaluation and monitoring.

NVIDIA’s confidential-computing documentation also highlights an operational trade-off: protected modes can disable performance counters, reducing observability.[13] Assurance and diagnosability must both be tested on the exact hardware and driver combination.

Disconnected means more than keeping the pod alive

A site claiming offline autonomy needs local copies of:

• model, tokenizer, adapter and quantization artifacts;
• runtime and platform container images;
• policy bundles and evaluation evidence;
• certificates, validation keys and revocation information;
• DNS, service discovery and time sources;
• identity-validation material;
• retrieval indexes and required tools;
• telemetry buffer and recovery procedures.

Red Hat’s disconnected OpenShift guidance distinguishes connected mirroring from filesystem/removable-media staging and warns that mirror availability affects installation, updates and ordinary workload relocation.[14] Release signatures must travel with artifacts. Apply the same dependency discipline to model files, GPU operators, firmware and vulnerability databases.

A useful resilience exercise is to disable WAN connectivity and then wait for credentials, certificates and caches to age. A ten-minute network test will not reveal what happens after an offline lease expires or a node must be rebuilt.

Cost per outcome, not cost per token

Owned accelerators introduce capital amortization, power, cooling, support, software, operations, idle replicas and availability headroom. Provider inference introduces token charges, endpoint hours, networking, retrieval, logging, guardrails, retries and contractual constraints.

Both sides also incur quality failure, human rework and latency costs.

The FinOps Foundation’s unit-economics guidance moves from early cost-per-token measures toward business outcomes such as cost per assist or case deflected.[15] That is the right direction. A cheap response that must be manually corrected is not cheap.

Local capacity is economical when utilization is predictable. Filling it to nearly 100%, however, destroys latency headroom and failure tolerance. Reserve local capacity for policy-critical and disconnected workloads. Make lower-priority work batchable, preemptible or cloud-eligible under explicit policy.

Failure modes to test before production

The global control plane disappears

Already-approved cells should continue under a signed policy lease. Define what expires, when high-risk operations stop, and how emergency revocation reaches a disconnected site.

Overflow changes the trust boundary

The router must refuse destinations before optimizing latency or price. Do not treat “no training” as identical to zero retention or no human access; endpoint-specific terms and contracts matter.

A stream fails after generation begins

Generation is not safely assumed to be idempotent. Use request IDs, retry budgets, durable application state and duplicate suppression. Decide whether to resume, restart or ask the user.

KV cache reaches a cliff

Admit against token budgets, cap context where justified, reserve decode capacity and expose degraded behavior. GPU utilization alone will not warn early enough.

Cold start becomes a network incident

Simultaneous nodes pulling multi-gigabyte artifacts can saturate a site link. Keep a local registry and model cache, stagger rollout and maintain warm pools for critical capability classes.

Policy splits during disconnection

Use signed bundles with activation and expiry. For sensitive tasks, use bounded offline authorization rather than indefinite stale policy.

Telemetry becomes the data leak

Default to metadata-only traces. Full prompt and output capture needs explicit classification, retention and access policy. Logs should not become an ungoverned copy of the enterprise corpus.

A practical adoption sequence

Phase 1 — Define capability and data classes

Select a small set of valuable tasks. Specify measured quality, latency, modality, context, tool and evidence requirements. Map enterprise data labels directly to eligible execution zones.

Phase 2 — Build one autonomous cell

Implement local gateway, immutable artifact cache, admission control, context adapters, policy snapshot and decision journal. Prove it works without the global management plane.

Phase 3 — Make placement explainable

Apply hard eligibility rules first. Record the candidate set, exclusions, ranking signals and final execution binding. Return degraded-mode metadata to the application.

Phase 4 — Federate cells

Add site and datacenter tiers without stretching one cluster across them. Distribute signed desired state and aggregate audit evidence asynchronously.

Phase 5 — Add governed overflow

Permit specific task and data classes to use approved external capability. Test redaction, retention, budget, latency and partial-failure behavior. Make destination changes visible.

Phase 6 — Exercise loss, expiry and rollback

Test WAN, DNS, registry, KMS, identity and time failures; stale policy; poisoned artifacts; model rollback; and half-completed streams. Recovery evidence matters more than a diagram that says “high availability.”

The conclusion: local is a property of the decision

Enterprise-local inference is not defined by rack ownership. It is defined by whether the organization can place, execute and explain a request within an acceptable trust, state, latency and failure boundary.

The winning architecture will not force every task into one location. It will:

• move computation toward authoritative context;
• distribute governance without centralizing the live token path;
• treat device, site, datacenter and cloud as distinct capability cells;
• enforce hard policy before optimization;
• bind exact release artifacts and produce a decision receipt;
• degrade explicitly when no eligible destination exists.

That turns “local AI” from a hardware purchase into an operable enterprise control system.

Sources

1. NIST, Zero Trust Architecture (SP 800-207): https://csrc.nist.gov/pubs/sp/800/207/final
2. European Union, Regulation (EU) 2024/1689 (AI Act): https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng/pdf
3. Kubernetes, Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/resource-management/dynamic-resource-allocation/
4. Kubernetes SIG Network, Gateway API Inference Extension — InferencePool: https://gateway-api-inference-extension.sigs.k8s.io/api-types/inferencepool/
5. Open Container Initiative, Distribution Specification: https://specs.opencontainers.org/distribution-spec/?v=v1.0.1
6. SLSA, Build Provenance v1.2: https://slsa.dev/spec/v1.2/build-provenance
7. Australian Signals Directorate and partners, AI/ML Supply Chain Risks and Mitigations: https://media.defense.gov/2026/Mar/04/2003882809/-1/-1/0/AI_ML_SUPPLY_CHAIN_RISKS_AND_MITIGATIONS.PDF
8. Hugging Face, Cache explanation: https://huggingface.co/docs/transformers/v5.20.2/en/cache_explanation
9. vLLM, Optimization and Tuning: https://docs.vllm.ai/en/latest/configuration/optimization
10. NVIDIA, GPU Operator — Time-Slicing GPUs: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
11. NVIDIA Dynamo, Disaggregated Serving: https://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/disaggregated-serving
12. IETF, RFC 9334: Remote ATtestation procedureS (RATS) Architecture: https://www.rfc-editor.org/rfc/rfc9334.html
13. NVIDIA, Confidential Computing on H100 GPUs: https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai
14. Red Hat, OpenShift disconnected environments: https://docs.redhat.com/en/documentation/openshift_container_platform/4.22/html/disconnected_environments/installing-mirroring-installation-images
15. FinOps Foundation, Unit Economics: https://www.finops.org/framework/capabilities/unit-economics/