# Model Coordination And Valkey

## Purpose

The backend can run against one Ollama host or a small cluster of AI boxes. Once
multiple API workers or model boxes are involved, process-local admission is not
enough: each process would only see its own in-flight work and could overcommit
the same model hosts.

Valkey provides the shared coordination plane for this problem. It is not
the document source of truth, not the retrieval index, and not a replacement for
OpenSearch. It exists to make short-lived scheduling decisions visible to every
backend process.

## Boundary

The runtime storage responsibilities are intentionally split:

| Store | Responsibility |
| --- | --- |
| JSON document records | Current source truth for document aggregates and answer/query records |
| Environment-local OpenSearch | Retrieval indexes, catalog/profile/summary projections, ingestion jobs, conversation memory, and projection-control manifests |
| Environment-local application Valkey | Application coordination and fail-open planning/retrieval caches for one deployment only |
| Shared model-coordination Valkey | Model placement, monitor election, fenced node residency, telemetry history, and agent-flow admission for the shared model boxes |

Nothing that must survive indefinitely should live only in Valkey. Values stored
there must either be reconstructable from source/projection state or safe to
expire after their TTL.

Conversation history and conversation memory are canonical durable state. They
are not application-cache entries and must not be moved to the shared model
pool. Cache loss may cause recomputation; it must not erase a conversation or
change its source scope.

## ELS Compass Deployment Topology

The development and demo deployments share only the physical model workers and
their scheduling control plane:

| Participant | Environment-local state | Shared model path |
| --- | --- | --- |
| Development API | `.84` OpenSearch and application Valkey | restricted local 6380 tunnel to `.199:6381/2` |
| Demo API on `.16` | demo OpenSearch and application Valkey on `.199:6379` | restricted local 6380 tunnel to `.199:6381/2` |
| Primary monitor on `.199` | local logs only | direct `.199:6381/2` |
| Standby monitor on `.17` | local logs only | restricted local 6380 tunnel to `.199:6381/2` |
| Model workers | Ollama on `.16` (`box-c`) and `.17` (`box-e`) | fenced telemetry and leases in the shared model pool |

The shared pool uses namespace `aibs-shared-models` and key prefix
`aibs:shared:model-coordination`. Development and demo application caches use
different databases, namespaces, prefixes, and credentials. Neither
application cache is an authorization source; enabled-source and RBAC checks
still run against canonical state.

Each participant has a separately revocable ACL identity. Passwords are read
from service-local files and are never embedded in URLs. The SSH tunnel account
on `.199` has no shell and each key is limited to forwarding
`127.0.0.1:6381`.

The two monitor candidates provide monitor-process and monitor-host failover.
They do not make the single `.199` Valkey host highly available. Loss of `.199`
still makes model placement fail closed until the model pool is restored or
moved to an independently replicated Valkey service.

## Why Valkey Is Needed

The global model scheduler needs operations that are both fast and atomic:

- read current node residency and active lease state;
- compare requested resident/KV budget against node capacity;
- reserve a node for one model call;
- attach a TTL so abandoned leases self-heal;
- release the lease after the model call finishes.

Doing that in process memory is only safe for a single API process. Doing it in
OpenSearch would be heavier, slower, and less natural for expiring locks. Valkey
fits the coordination shape: atomic Lua scripts, server-side time, TTLs, and low
latency.

The scheduler uses server-side `TIME` in the Valkey script so all API
processes evaluate lease freshness against the same clock authority instead of
their own machine clocks.

## What Uses The Coordination Plane

The coordination port is shared by several runtime concerns:

- **Global model placement.** The scheduler reserves a specific node for a model
  call and returns the selected node URL to the LLM adapter.
- **Model residency monitoring.** Node monitors publish recent Ollama residency
  observations so stale or unknown nodes are not selected.
- **Node host telemetry.** Optional lightweight agents on each AI box expose CPU,
  memory, GPU busy, raw VRAM, and raw GTT for the System Health page. The UI
  presents VRAM and GTT separately; a derived aggregate may be shown only as
  additional context. These samples are read-only operator telemetry and do not
  replace the scheduler's configured capacity budget or fenced residency facts.
- **Agent-flow admission.** Broad agentic workloads use shared queue/active-flow
  state when a shared coordination backend is configured.
- **Projection repair fallback.** If projection-control writes fail, dirty-state
  fallback records can be shared across processes.
- **Best-effort caches.** Cache users may treat Valkey failures as misses; safety
  users must fail closed.

These are coordination facts, not business facts.

## Failure Semantics

Valkey-backed model placement is deliberately fail-closed. If the scheduler
cannot prove that a node is fresh, has capacity, and has an atomic lease, it must
not place work onto that node.

The memory coordination adapter remains useful for local development and tests,
but it is process-local. The application must not run
`AIBS_MODEL_CLUSTER_ENABLED=true` with `AIBS_COORDINATION_BACKEND=memory`,
because that combination would advertise global capacity control while every API
process still has a private view of the world.

For best-effort cache paths, a Valkey failure should degrade into a cache miss.
For lease-protected paths, a Valkey failure should deny or queue work rather than
overcommit the model cluster.

## Enablement Gate

Before enabling the global scheduler in an environment:

1. Configure a shared model-coordination Valkey endpoint reachable by every API
   worker and monitor.
2. Set `AIBS_MODEL_COORDINATION_BACKEND=valkey` and configure the canonical
   `AIBS_MODEL_COORDINATION_VALKEY_*` settings.
3. Configure the model cluster node ids, URLs, GPU-memory budgets, and worker
   weights. The environment variable remains `AIBS_MODEL_CLUSTER_NODE_VRAM_GB`
   for compatibility, but the measured value must cover all GPU memory the node
   can safely use for model residency. Configure
   `AIBS_MODEL_CLUSTER_NODE_MAX_PARALLEL_REQUESTS` from the serving process's
   verified parallel runner capacity; do not infer it from memory alone.
4. Run `audit_model_capacity` until it reports the scheduler as ready.
5. Run `prove_model_monitor_takeover --repeat 3 --handover-gate strict` from
   the target topology. Correctness requires zero pre-kill owner loss and zero
   successful fenced writes from a deposed owner. The complete timing bound
   (`poll interval + election TTL + standby retry + two poll timeouts`) must
   fit inside the configured residency-staleness window.
6. Optionally install `scripts/aibs_node_telemetry_agent.py` on each model node
   and configure `AIBS_MODEL_CLUSTER_NODE_HOST_TELEMETRY_URLS` for host-load
   charts.
7. Enable `AIBS_MODEL_CLUSTER_ENABLED=true` only after the gates are green.

The operational runbook is in
`docs/maintainance_&_operations/model_capacity_audit.md`.