# Standalone Model Residency Monitor Run the residency monitor as a dedicated control-plane process when global model placement is enabled. Do not make its election heartbeat compete with long answer, embedding, reranking, or ingestion work on the API event loop. ## Configuration Use the same shared model-coordination Valkey pool as every API deployment that can schedule work on the configured model nodes: ```bash AIBS_MODEL_CLUSTER_ENABLED=true AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_ENABLED=true AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_RUN_IN_API=false AIBS_MODEL_COORDINATION_BACKEND=valkey AIBS_MODEL_COORDINATION_NAMESPACE=aibs-shared-models AIBS_MODEL_COORDINATION_VALKEY_URL=redis://127.0.0.1:6381/2 AIBS_MODEL_COORDINATION_VALKEY_USERNAME=aibs-model-primary-monitor AIBS_MODEL_COORDINATION_VALKEY_PASSWORD_FILE=/etc/aibs/model-valkey-primary-monitor.secret AIBS_MODEL_COORDINATION_VALKEY_KEY_PREFIX=aibs:shared:model-coordination AIBS_MODEL_COORDINATION_VALKEY_SOCKET_TIMEOUT_SECONDS=1.5 ``` The API still reads the fenced node telemetry and performs atomic placement. It does not start a second model monitor when `RUN_IN_API=false`. It does continue its own environment-local System Health workload sampler; ingestion, answer, and runtime charts must not depend on which process owns the global model-monitor election. Keep application coordination deployment-specific. Dev and demo APIs may share this model pool only when they genuinely share the same physical model nodes; their `AIBS_COORDINATION_NAMESPACE` and application coordination key prefixes must remain different. The standalone process writes its control-plane and HTTP polling logs to `AIBS_MODEL_RESIDENCY_LOG_FILENAME` (default `model_residency.log`) instead of the API process log. Do not point both processes at the same rotating file on Windows; cross-process rotation and file locking can delay lease heartbeats. ## Demo control-plane deployment The demo application Valkey and the global model-coordination Valkey are separate services. Do not restart, upgrade, or repurpose the application container to introduce model scheduling. The repository provides: - `deploy/valkey/model-coordination.conf`; - `deploy/valkey/model-coordination.acl.example`; - `deploy/sysctl/99-aibs-model-valkey.conf`; - `deploy/systemd/aibs-model-valkey.service`; - `deploy/systemd/aibs-model-residency-monitor.service`; - `deploy/systemd/aibs-model-residency-monitor-standby.service`; - `deploy/systemd/aibs-model-valkey-tunnel.service`; - `deploy/systemd/model-residency-monitor.env.example`. The model Valkey service is pinned to `valkey/valkey:8.1.9-alpine`, listens only on `.199` loopback port 6381, stores AOF and snapshots in the named `aibs-model-valkey-data` volume, rejects eviction, and requires a deployment ACL. Generate separate secrets for the development API, demo API, primary monitor, and standby monitor. Store only their SHA-256 verifiers in the Valkey ACL, scope every identity to `aibs:shared:model-coordination:*`, and keep all raw secrets outside Git. Reference each secret through `AIBS_MODEL_COORDINATION_VALKEY_PASSWORD_FILE`; do not embed credentials in URLs or copy one participant's credential to another host. Install each raw secret with owner `root:aibs` and mode `0640` when the service runs as `aibs`. On Windows, remove inherited ACLs and grant the API account read access only. The pinned image runs Valkey as numeric UID/GID `999:1000`; install the ACL with owner `root:1000` and mode `0640` so only root and that container group can read it. Confirm the pinned image still uses those IDs before any image upgrade. Before enabling either service: 1. pull the pinned image explicitly; 2. install the reviewed sysctl file, apply it, and confirm `vm.overcommit_memory = 1`; 3. copy the reviewed Valkey config and systemd unit to `.199`; 4. create `/var/log/aibs` owned by the monitor service account; 5. install the ACL and monitor environment files outside the repository; 6. run `systemd-analyze verify` on both units; 7. start only `aibs-model-valkey.service`; 8. authenticate and verify `PING`, `INFO persistence`, AOF state, the `noeviction` policy, database 2, and restart persistence; 9. deploy one approved immutable backend artifact to the demo API on `.16`, the primary monitor on `.199`, and the standby monitor on `.17`; 10. start the `.199` primary candidate and confirm fresh fenced snapshots; 11. start the `.17` standby candidate; 12. run the three-repeat strict takeover proof and a supervised primary-kill test before restarting either API onto the shared pool. The development API keeps its `.84` OpenSearch/application-Valkey tunnel and uses a separate restricted local 6380 tunnel to `.199:6381` for model coordination. The demo API and `.17` standby use supervised instances of `aibs-model-valkey-tunnel.service` to the same endpoint. The tunnel identity has no shell; each authorized key is limited to forwarding `127.0.0.1:6381`, uses a pinned host key, and is independently revocable. Every participating API and monitor must use the same database, namespace, and prefix. Provision a separate revocable ACL credential for each deployment instead of copying the monitor credential between hosts. The persistent single `.199` model Valkey removes the current unpersisted container and developer-workstation dependencies from model coordination. It does not survive loss of the `.199` host. Genuine infrastructure HA requires a replicated/managed Valkey deployment with an independently hosted failover endpoint; do not label a single-host service as HA. ## Start From the repository root with the backend environment activated: ```bash python -m aibs_backend.interfaces.cli.model_residency_monitor ``` Start this process before the API and manage both with the deployment service supervisor. Run at least two monitor candidates on independent always-on hosts. Multiple standalone monitor processes are safe: Valkey elects one owner and the others remain standby. A second process on the same host does not protect against host or network loss. When moving an existing deployment to a dedicated model-coordination database, do not leave old and new pools active in parallel. Start a monitor against the new pool, confirm fresh snapshots there, and then restart every sharing API and remaining monitor against the same URL, database, namespace, and prefix. Different databases are different schedulers even when the Valkey host is the same. ## Verification 1. Confirm the monitor logs `Model node residency monitor elected`. 2. Run `python -m aibs_backend.interfaces.cli.audit_model_capacity --json`. 3. Confirm fresh healthy node observations and `ready_for_global_scheduler=true`. 4. Run the takeover proof from [model_capacity_audit.md](model-capacity-audit.md) three consecutive times with `--handover-gate strict`. 5. During a long answer request, confirm the monitor continues polling and does not log election loss or renewal failure. 6. Restart the active monitor and confirm a supervised standby takes ownership before the residency staleness window expires. If the monitor stops, placement fails closed once telemetry exceeds `AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS`. Do not bypass that failure with cached health or an API-side optimistic fallback. The API process still starts in a degraded state so health, administration, and durable recovery endpoints remain available. Model-dependent chat returns a typed `503` until the continuous readiness gate observes a fresh healthy node snapshot.