Model Coordination And Valkey

Purpose

The backend can run against one Ollama host or a small cluster of AI boxes. Once multiple API workers or model boxes are involved, process-local admission is not enough: each process would only see its own in-flight work and could overcommit the same model hosts.

Valkey provides the shared coordination plane for this problem. It is not the document source of truth, not the retrieval index, and not a replacement for OpenSearch. It exists to make short-lived scheduling decisions visible to every backend process.

Boundary

The runtime storage responsibilities are intentionally split:

Store	Responsibility
JSON document records	Current source truth for document aggregates and answer/query records
Environment-local OpenSearch	Retrieval indexes, catalog/profile/summary projections, ingestion jobs, conversation memory, and projection-control manifests
Environment-local application Valkey	Application coordination and fail-open planning/retrieval caches for one deployment only
Shared model-coordination Valkey	Model placement, monitor election, fenced node residency, telemetry history, and agent-flow admission for the shared model boxes

Nothing that must survive indefinitely should live only in Valkey. Values stored there must either be reconstructable from source/projection state or safe to expire after their TTL.

Conversation history and conversation memory are canonical durable state. They are not application-cache entries and must not be moved to the shared model pool. Cache loss may cause recomputation; it must not erase a conversation or change its source scope.

ELS Compass Deployment Topology

The development and demo deployments share only the physical model workers and their scheduling control plane:

Participant	Environment-local state	Shared model path
Development API	`.84` OpenSearch and application Valkey	restricted local 6380 tunnel to `.199:6381/2`
Demo API on `.16`	demo OpenSearch and application Valkey on `.199:6379`	restricted local 6380 tunnel to `.199:6381/2`
Primary monitor on `.199`	local logs only	direct `.199:6381/2`
Standby monitor on `.17`	local logs only	restricted local 6380 tunnel to `.199:6381/2`
Model workers	Ollama on `.16` (`box-c`) and `.17` (`box-e`)	fenced telemetry and leases in the shared model pool

The shared pool uses namespace aibs-shared-models and key prefix aibs:shared:model-coordination. Development and demo application caches use different databases, namespaces, prefixes, and credentials. Neither application cache is an authorization source; enabled-source and RBAC checks still run against canonical state.

Each participant has a separately revocable ACL identity. Passwords are read from service-local files and are never embedded in URLs. The SSH tunnel account on .199 has no shell and each key is limited to forwarding 127.0.0.1:6381.

The two monitor candidates provide monitor-process and monitor-host failover. They do not make the single .199 Valkey host highly available. Loss of .199 still makes model placement fail closed until the model pool is restored or moved to an independently replicated Valkey service.

Why Valkey Is Needed

The global model scheduler needs operations that are both fast and atomic:

read current node residency and active lease state;
compare requested resident/KV budget against node capacity;
reserve a node for one model call;
attach a TTL so abandoned leases self-heal;
release the lease after the model call finishes.

Doing that in process memory is only safe for a single API process. Doing it in OpenSearch would be heavier, slower, and less natural for expiring locks. Valkey fits the coordination shape: atomic Lua scripts, server-side time, TTLs, and low latency.

The scheduler uses server-side TIME in the Valkey script so all API processes evaluate lease freshness against the same clock authority instead of their own machine clocks.

What Uses The Coordination Plane

The coordination port is shared by several runtime concerns:

Global model placement. The scheduler reserves a specific node for a model call and returns the selected node URL to the LLM adapter.
Model residency monitoring. Node monitors publish recent Ollama residency observations so stale or unknown nodes are not selected.
Node host telemetry. Optional lightweight agents on each AI box expose CPU, memory, GPU busy, raw VRAM, and raw GTT for the System Health page. The UI presents VRAM and GTT separately; a derived aggregate may be shown only as additional context. These samples are read-only operator telemetry and do not replace the scheduler’s configured capacity budget or fenced residency facts.
Agent-flow admission. Broad agentic workloads use shared queue/active-flow state when a shared coordination backend is configured.
Projection repair fallback. If projection-control writes fail, dirty-state fallback records can be shared across processes.
Best-effort caches. Cache users may treat Valkey failures as misses; safety users must fail closed.

These are coordination facts, not business facts.

Failure Semantics

Valkey-backed model placement is deliberately fail-closed. If the scheduler cannot prove that a node is fresh, has capacity, and has an atomic lease, it must not place work onto that node.

The memory coordination adapter remains useful for local development and tests, but it is process-local. The application must not run AIBS_MODEL_CLUSTER_ENABLED=true with AIBS_COORDINATION_BACKEND=memory, because that combination would advertise global capacity control while every API process still has a private view of the world.

For best-effort cache paths, a Valkey failure should degrade into a cache miss. For lease-protected paths, a Valkey failure should deny or queue work rather than overcommit the model cluster.

Enablement Gate

Before enabling the global scheduler in an environment:

Configure a shared model-coordination Valkey endpoint reachable by every API worker and monitor.
Set AIBS_MODEL_COORDINATION_BACKEND=valkey and configure the canonical AIBS_MODEL_COORDINATION_VALKEY_* settings.
Configure the model cluster node ids, URLs, GPU-memory budgets, and worker weights. The environment variable remains AIBS_MODEL_CLUSTER_NODE_VRAM_GB for compatibility, but the measured value must cover all GPU memory the node can safely use for model residency. Configure AIBS_MODEL_CLUSTER_NODE_MAX_PARALLEL_REQUESTS from the serving process’s verified parallel runner capacity; do not infer it from memory alone.
Run audit_model_capacity until it reports the scheduler as ready.
Run prove_model_monitor_takeover --repeat 3 --handover-gate strict from the target topology. Correctness requires zero pre-kill owner loss and zero successful fenced writes from a deposed owner. The complete timing bound (poll interval + election TTL + standby retry + two poll timeouts) must fit inside the configured residency-staleness window.
Optionally install scripts/aibs_node_telemetry_agent.py on each model node and configure AIBS_MODEL_CLUSTER_NODE_HOST_TELEMETRY_URLS for host-load charts.
Enable AIBS_MODEL_CLUSTER_ENABLED=true only after the gates are green.

The operational runbook is in docs/maintainance_&_operations/model_capacity_audit.md.