Standalone Model Residency Monitor

Run the residency monitor as a dedicated control-plane process when global model placement is enabled. Do not make its election heartbeat compete with long answer, embedding, reranking, or ingestion work on the API event loop.

Configuration

Use the same shared model-coordination Valkey pool as every API deployment that can schedule work on the configured model nodes:

AIBS_MODEL_CLUSTER_ENABLED=true
AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_ENABLED=true
AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_RUN_IN_API=false
AIBS_MODEL_COORDINATION_BACKEND=valkey
AIBS_MODEL_COORDINATION_NAMESPACE=aibs-shared-models
AIBS_MODEL_COORDINATION_VALKEY_URL=redis://127.0.0.1:6381/2
AIBS_MODEL_COORDINATION_VALKEY_USERNAME=aibs-model-primary-monitor
AIBS_MODEL_COORDINATION_VALKEY_PASSWORD_FILE=/etc/aibs/model-valkey-primary-monitor.secret
AIBS_MODEL_COORDINATION_VALKEY_KEY_PREFIX=aibs:shared:model-coordination
AIBS_MODEL_COORDINATION_VALKEY_SOCKET_TIMEOUT_SECONDS=1.5

The API still reads the fenced node telemetry and performs atomic placement. It does not start a second model monitor when RUN_IN_API=false. It does continue its own environment-local System Health workload sampler; ingestion, answer, and runtime charts must not depend on which process owns the global model-monitor election.

Keep application coordination deployment-specific. Dev and demo APIs may share this model pool only when they genuinely share the same physical model nodes; their AIBS_COORDINATION_NAMESPACE and application coordination key prefixes must remain different.

The standalone process writes its control-plane and HTTP polling logs to AIBS_MODEL_RESIDENCY_LOG_FILENAME (default model_residency.log) instead of the API process log. Do not point both processes at the same rotating file on Windows; cross-process rotation and file locking can delay lease heartbeats.

Demo control-plane deployment

The demo application Valkey and the global model-coordination Valkey are separate services. Do not restart, upgrade, or repurpose the application container to introduce model scheduling.

The repository provides:

  • deploy/valkey/model-coordination.conf;

  • deploy/valkey/model-coordination.acl.example;

  • deploy/sysctl/99-aibs-model-valkey.conf;

  • deploy/systemd/aibs-model-valkey.service;

  • deploy/systemd/aibs-model-residency-monitor.service;

  • deploy/systemd/aibs-model-residency-monitor-standby.service;

  • deploy/systemd/aibs-model-valkey-tunnel.service;

  • deploy/systemd/model-residency-monitor.env.example.

The model Valkey service is pinned to valkey/valkey:8.1.9-alpine, listens only on .199 loopback port 6381, stores AOF and snapshots in the named aibs-model-valkey-data volume, rejects eviction, and requires a deployment ACL. Generate separate secrets for the development API, demo API, primary monitor, and standby monitor. Store only their SHA-256 verifiers in the Valkey ACL, scope every identity to aibs:shared:model-coordination:*, and keep all raw secrets outside Git. Reference each secret through AIBS_MODEL_COORDINATION_VALKEY_PASSWORD_FILE; do not embed credentials in URLs or copy one participant’s credential to another host. Install each raw secret with owner root:aibs and mode 0640 when the service runs as aibs. On Windows, remove inherited ACLs and grant the API account read access only. The pinned image runs Valkey as numeric UID/GID 999:1000; install the ACL with owner root:1000 and mode 0640 so only root and that container group can read it. Confirm the pinned image still uses those IDs before any image upgrade.

Before enabling either service:

  1. pull the pinned image explicitly;

  2. install the reviewed sysctl file, apply it, and confirm vm.overcommit_memory = 1;

  3. copy the reviewed Valkey config and systemd unit to .199;

  4. create /var/log/aibs owned by the monitor service account;

  5. install the ACL and monitor environment files outside the repository;

  6. run systemd-analyze verify on both units;

  7. start only aibs-model-valkey.service;

  8. authenticate and verify PING, INFO persistence, AOF state, the noeviction policy, database 2, and restart persistence;

  9. deploy one approved immutable backend artifact to the demo API on .16, the primary monitor on .199, and the standby monitor on .17;

  10. start the .199 primary candidate and confirm fresh fenced snapshots;

  11. start the .17 standby candidate;

  12. run the three-repeat strict takeover proof and a supervised primary-kill test before restarting either API onto the shared pool.

The development API keeps its .84 OpenSearch/application-Valkey tunnel and uses a separate restricted local 6380 tunnel to .199:6381 for model coordination. The demo API and .17 standby use supervised instances of aibs-model-valkey-tunnel.service to the same endpoint. The tunnel identity has no shell; each authorized key is limited to forwarding 127.0.0.1:6381, uses a pinned host key, and is independently revocable. Every participating API and monitor must use the same database, namespace, and prefix. Provision a separate revocable ACL credential for each deployment instead of copying the monitor credential between hosts.

The persistent single .199 model Valkey removes the current unpersisted container and developer-workstation dependencies from model coordination. It does not survive loss of the .199 host. Genuine infrastructure HA requires a replicated/managed Valkey deployment with an independently hosted failover endpoint; do not label a single-host service as HA.

Start

From the repository root with the backend environment activated:

python -m aibs_backend.interfaces.cli.model_residency_monitor

Start this process before the API and manage both with the deployment service supervisor. Run at least two monitor candidates on independent always-on hosts. Multiple standalone monitor processes are safe: Valkey elects one owner and the others remain standby. A second process on the same host does not protect against host or network loss.

When moving an existing deployment to a dedicated model-coordination database, do not leave old and new pools active in parallel. Start a monitor against the new pool, confirm fresh snapshots there, and then restart every sharing API and remaining monitor against the same URL, database, namespace, and prefix. Different databases are different schedulers even when the Valkey host is the same.

Verification

  1. Confirm the monitor logs Model node residency monitor elected.

  2. Run python -m aibs_backend.interfaces.cli.audit_model_capacity --json.

  3. Confirm fresh healthy node observations and ready_for_global_scheduler=true.

  4. Run the takeover proof from model_capacity_audit.md three consecutive times with --handover-gate strict.

  5. During a long answer request, confirm the monitor continues polling and does not log election loss or renewal failure.

  6. Restart the active monitor and confirm a supervised standby takes ownership before the residency staleness window expires.

If the monitor stops, placement fails closed once telemetry exceeds AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS. Do not bypass that failure with cached health or an API-side optimistic fallback. The API process still starts in a degraded state so health, administration, and durable recovery endpoints remain available. Model-dependent chat returns a typed 503 until the continuous readiness gate observes a fresh healthy node snapshot.