# Model Capacity Audit Use this command before enabling the global model scheduler. It compares the configured worker roster with live Ollama `/api/ps` responses from the serving nodes, reports loaded model residency, and suggests resident-weight settings when a roster model is currently loaded on a node. By default the command is read-only. It does not place work, mutate Valkey state, or change model residency. ## Command ```bash python -m aibs_backend.interfaces.cli.audit_model_capacity ``` Optional ad-hoc node inspection: ```bash python -m aibs_backend.interfaces.cli.audit_model_capacity \ --node-url http://10.4.3.16:11434 \ --node-url http://10.4.3.17:11434 ``` Machine-readable output: ```bash python -m aibs_backend.interfaces.cli.audit_model_capacity --json ``` Active warm-up measurement: ```bash python -m aibs_backend.interfaces.cli.audit_model_capacity \ --warmup \ --warmup-node-id box-c \ --json ``` `--warmup` deliberately sends a tiny non-streaming Ollama request for each configured worker model, samples `/api/ps` while that model is resident, and then unloads the model with `keep_alive=0`. Use it only during an operator measurement window because it consumes model slots and may evict currently loaded models. Omit `--warmup-node-id` to measure every configured node. ## What It Checks - `AIBS_MODEL_ADMISSION_WORKER_ROSTER` remains the source of model lanes and roles. The audit does not introduce a second model catalog. - Each worker should have measured `resident_weight_gb` and `active_parameter_billions`. - Ollama `/api/ps` supplies `size_vram` and `size`; the audit reports those as resident-weight suggestions for matching workers. It does not infer active parameter counts. - `--warmup` can produce resident-weight suggestions even when the boxes are idle and the configured worker models are cold. - Each configured cluster node should expose live `/api/ps` state. - The `ready_to_enable_global_scheduler` gate is independent of `AIBS_MODEL_CLUSTER_ENABLED`. That lets operators audit first, then enable only after Valkey coordination, node budgets, worker measurements, and the node-residency monitor are present. ## Related Settings ```bash AIBS_MODEL_CLUSTER_ENABLED=false AIBS_MODEL_CLUSTER_BACKEND=valkey AIBS_MODEL_CLUSTER_FALLBACK_MODE=fail_closed AIBS_MODEL_CLUSTER_NODE_IDS=box-c,box-e AIBS_MODEL_CLUSTER_NODE_URLS=http://10.4.3.16:11434,http://10.4.3.17:11434 AIBS_MODEL_CLUSTER_NODE_VRAM_GB=90,104 AIBS_MODEL_CLUSTER_NODE_MAX_PARALLEL_REQUESTS=1,1 AIBS_MODEL_CLUSTER_NODE_HOST_TELEMETRY_URLS=http://10.4.3.16:8765/telemetry,http://10.4.3.17:8765/telemetry AIBS_MODEL_CLUSTER_ROLE_AFFINITY_JSON={"semantic_profile":["box-c"]} AIBS_MODEL_CLUSTER_ACTIVE_LEASE_TTL_SECONDS=900 AIBS_MODEL_CLUSTER_HEARTBEAT_INTERVAL_SECONDS=30 AIBS_MODEL_CLUSTER_POLL_INTERVAL_SECONDS=0.1 AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS=20000 AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_ENABLED=true AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_RUN_IN_API=false AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_POLL_INTERVAL_SECONDS=3 AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_POLL_TIMEOUT_SECONDS=2 AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_FAILURE_THRESHOLD=2 AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_LEASE_TTL_SECONDS=8 AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_STANDBY_RETRY_SECONDS=1 AIBS_MODEL_CLUSTER_TELEMETRY_TTL_SECONDS=30 AIBS_MODEL_CLUSTER_HEALTH_HISTORY_WINDOW_SECONDS=900 AIBS_MODEL_CLUSTER_HEALTH_HISTORY_MAX_SAMPLES=180 AIBS_MODEL_CLUSTER_HEALTH_HISTORY_TTL_SECONDS=1800 ``` Do not enable `AIBS_MODEL_CLUSTER_ENABLED` with `AIBS_COORDINATION_BACKEND=memory`. The application rejects that combination because the memory coordination adapter is process-local and cannot enforce global model capacity. `AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_ENABLED=true` is safe to run while `AIBS_MODEL_CLUSTER_ENABLED=false`. The monitor elects one process through Valkey, polls configured Ollama nodes, writes a tiny placement health stamp, and writes richer telemetry to a separate key. Use `AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_RUN_IN_API=false` and the standalone monitor in multi-node or latency-sensitive deployments. This proves fresh residency before routing model calls through the global scheduler. Keep the monitor lease TTL longer than one poll interval plus one poll timeout, and keep the full takeover bound inside the residency staleness window: `poll_interval + lease_ttl + standby_retry + 2 * poll_timeout`. Do not enable `AIBS_MODEL_CLUSTER_ENABLED` until the monitor is running, audit is green, observations stay fresh across a monitor kill/takeover test, worker resident weights are measured, and a live broad query succeeds. The placement primitive fails closed when node residency is missing or older than `AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS`; without the monitor, every request should look stale even when model nodes are idle. Shared-tenancy note: the AI boxes may also host non-AIBS Ollama models. The global scheduler identifies AIBS-owned models from the configured worker roster and treats every other resident model in the fresh node telemetry snapshot as foreign residency. Foreign resident GB is subtracted from the node budget before placement. AIBS roster models visible in `/api/ps` are not subtracted as foreign workloads; they remain governed by AIBS leases and worker sizing. System Health keeps two deliberately separate timelines: - the elected model monitor writes model-node and host telemetry history to the shared model-coordination pool; - each API environment writes ingestion, answer, and runtime workload history to its own application-coordination pool. Workload sampling starts immediately with the API and does not depend on global model-monitor election. A short environment-local lease prevents duplicate writers when multiple API instances share the same application coordination namespace. The history settings control how much of each timeline the API can return to the frontend: - `AIBS_MODEL_CLUSTER_HEALTH_HISTORY_WINDOW_SECONDS` - `AIBS_MODEL_CLUSTER_HEALTH_HISTORY_MAX_SAMPLES` - `AIBS_MODEL_CLUSTER_HEALTH_HISTORY_TTL_SECONDS` Keep the history TTL greater than or equal to the displayed history window. ## Optional Host Telemetry Agent For richer System Health charts, install the lightweight host telemetry agent on each AI box and expose it only on the trusted management network: ```bash python scripts/aibs_node_telemetry_agent.py --host 0.0.0.0 --port 8765 ``` The agent is read-only. It reports CPU, memory, GPU busy, raw VRAM, raw GTT, and combined GPU memory from Linux `/proc` and DRM sysfs. AMD APU/iGPU deployments can place model buffers in GTT, so operator-facing capacity should use combined GPU memory rather than raw VRAM alone. The agent does not run model calls, mutate Valkey, or make placement decisions. The backend polls the URLs configured through `AIBS_MODEL_CLUSTER_NODE_HOST_TELEMETRY_URLS` and stores those values beside the fenced residency snapshots for operator visibility. If the agent is not configured or unavailable, the node remains schedulable when its fenced residency facts are fresh. System Health will show host telemetry as unavailable for that node. ## Valkey Placement Test Gate The global placement primitive requires Valkey 8+. The Valkey Lua scripts use server-side `TIME` while writing placement state so all API processes share one clock authority. Do not enable the global scheduler against older Redis-compatible deployments. Run the atomic placement tests against a real Valkey instance: ```bash AIBS_TEST_VALKEY_URL=redis://localhost:6379/15 \ python -m pytest tests/infrastructure/model_scheduling/test_redis_atomic_placement.py ``` Without `AIBS_TEST_VALKEY_URL`, the real Valkey behavior test is skipped and only the deterministic in-memory/fake-client tests run. The legacy `AIBS_TEST_REDIS_URL` name remains a temporary compatibility fallback. ## Residency Monitor Takeover Gate Before enabling `AIBS_MODEL_CLUSTER_ENABLED=true`, prove that the elected residency monitor can die without leaving stale node health trusted by the placement scheduler: ```bash python -m aibs_backend.interfaces.cli.prove_model_monitor_takeover --json ``` The proof command: 1. creates an isolated Valkey namespace; 2. starts monitor owner A as a subprocess; 3. waits for owner A to publish healthy samples for every enabled model node; 4. starts monitor owner B as a standby; 5. hard-kills owner A, without graceful lease release; 6. waits for owner B to publish newer healthy samples; 7. verifies correctness: no pre-kill owner loss and no successful fenced writes from a deposed owner; 8. measures handover duration separately from correctness; 9. fails the availability gate unless the target deployment topology observes fresh replacement samples inside `AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS`; 10. captures owner stderr and classifies expected shutdown artifacts separately from real owner churn; 11. deletes the isolated proof namespace so proof scheduler hashes do not accumulate in Valkey. The safe timing bound is: ```text residency_monitor_poll_interval + residency_monitor_lease_ttl + residency_monitor_standby_retry + 2 * residency_monitor_poll_timeout <= max_residency_staleness ``` With the current recommended values, `3 + 6 + 1 + 2 * 2 = 14s`, which fits inside the `15s` staleness window. For deployment-grade evidence, run the proof repeatedly and require every run to pass: ```bash python -m aibs_backend.interfaces.cli.prove_model_monitor_takeover --json --repeat 3 ``` When running from the local dev machine through an SSH tunnel to Valkey, use the split contract explicitly: ```bash python -m aibs_backend.interfaces.cli.prove_model_monitor_takeover \ --json --repeat 3 --handover-gate report-only ``` In that mode, correctness still gates the result. Long handover duration is reported as an availability finding because the dev tunnel is not the production transport. On the demo topology, where the backend runs on an AI box and reaches Valkey over LAN, use the default strict handover gate before enabling shared traffic. Treat `passed=true` as a scheduler liveness gate, not as full production approval. It proves election/failover and fresh node telemetry. It does not prove answer quality or routed LLM behavior under real user load. If correctness fails, do not enable the cluster scheduler. If only the dev tunnel handover report fails, document the admission-pause duration and rerun the strict proof from the target topology. ## Controlled Enable Smoke Test After the audit and takeover proof are green, enable the scheduler only in the target test environment: ```bash AIBS_MODEL_CLUSTER_ENABLED=true ``` Then restart the API process. Do not rely on `--reload` to notice a `.env`-only change; some reload configurations watch Python files but not environment files. Run: ```bash python -m aibs_backend.interfaces.cli.audit_model_capacity --json ``` Expected enable-state result: - `cluster_enabled=true` - `ready_to_enable_global_scheduler=true` - `ready_for_global_scheduler=true` Then run the real API path: 1. one narrow or single-document query; 2. one broad or agentic query; 3. two concurrent broad queries from separate conversations or users when possible. Acceptance evidence: - global placement acquisition/release logs appear; - leases include `node_id` and `node_base_url`; - answer quality does not regress; - no request remains stuck in answering state after completion/cancel; - Valkey active leases are released or expire according to TTL. If placement acquisition works but release warnings appear, stop before demo and inspect the placement release path. A scheduler that admits work but leaks capacity is not enterprise-stable.