Model Capacity Audit

Use this command before enabling the global model scheduler. It compares the configured worker roster with live Ollama /api/ps responses from the serving nodes, reports loaded model residency, and suggests resident-weight settings when a roster model is currently loaded on a node.

By default the command is read-only. It does not place work, mutate Valkey state, or change model residency.

Command

python -m aibs_backend.interfaces.cli.audit_model_capacity

Optional ad-hoc node inspection:

python -m aibs_backend.interfaces.cli.audit_model_capacity \
  --node-url http://10.4.3.16:11434 \
  --node-url http://10.4.3.17:11434

Machine-readable output:

python -m aibs_backend.interfaces.cli.audit_model_capacity --json

Active warm-up measurement:

python -m aibs_backend.interfaces.cli.audit_model_capacity \
  --warmup \
  --warmup-node-id box-c \
  --json

--warmup deliberately sends a tiny non-streaming Ollama request for each configured worker model, samples /api/ps while that model is resident, and then unloads the model with keep_alive=0. Use it only during an operator measurement window because it consumes model slots and may evict currently loaded models. Omit --warmup-node-id to measure every configured node.

What It Checks

• AIBS_MODEL_ADMISSION_WORKER_ROSTER remains the source of model lanes and roles. The audit does not introduce a second model catalog.

• Each worker should have measured resident_weight_gb and active_parameter_billions.

• Ollama /api/ps supplies size_vram and size; the audit reports those as resident-weight suggestions for matching workers. It does not infer active parameter counts.

• --warmup can produce resident-weight suggestions even when the boxes are idle and the configured worker models are cold.

• Each configured cluster node should expose live /api/ps state.

• The ready_to_enable_global_scheduler gate is independent of AIBS_MODEL_CLUSTER_ENABLED. That lets operators audit first, then enable only after Valkey coordination, node budgets, worker measurements, and the node-residency monitor are present.

Related Settings

AIBS_MODEL_CLUSTER_ENABLED=false
AIBS_MODEL_CLUSTER_BACKEND=valkey
AIBS_MODEL_CLUSTER_FALLBACK_MODE=fail_closed
AIBS_MODEL_CLUSTER_NODE_IDS=box-c,box-e
AIBS_MODEL_CLUSTER_NODE_URLS=http://10.4.3.16:11434,http://10.4.3.17:11434
AIBS_MODEL_CLUSTER_NODE_VRAM_GB=90,104
AIBS_MODEL_CLUSTER_NODE_MAX_PARALLEL_REQUESTS=1,1
AIBS_MODEL_CLUSTER_NODE_HOST_TELEMETRY_URLS=http://10.4.3.16:8765/telemetry,http://10.4.3.17:8765/telemetry
AIBS_MODEL_CLUSTER_ROLE_AFFINITY_JSON={"semantic_profile":["box-c"]}
AIBS_MODEL_CLUSTER_ACTIVE_LEASE_TTL_SECONDS=900
AIBS_MODEL_CLUSTER_HEARTBEAT_INTERVAL_SECONDS=30
AIBS_MODEL_CLUSTER_POLL_INTERVAL_SECONDS=0.1
AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS=20000
AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_ENABLED=true
AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_RUN_IN_API=false
AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_POLL_INTERVAL_SECONDS=3
AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_POLL_TIMEOUT_SECONDS=2
AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_FAILURE_THRESHOLD=2
AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_LEASE_TTL_SECONDS=8
AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_STANDBY_RETRY_SECONDS=1
AIBS_MODEL_CLUSTER_TELEMETRY_TTL_SECONDS=30
AIBS_MODEL_CLUSTER_HEALTH_HISTORY_WINDOW_SECONDS=900
AIBS_MODEL_CLUSTER_HEALTH_HISTORY_MAX_SAMPLES=180
AIBS_MODEL_CLUSTER_HEALTH_HISTORY_TTL_SECONDS=1800

Do not enable AIBS_MODEL_CLUSTER_ENABLED with AIBS_COORDINATION_BACKEND=memory. The application rejects that combination because the memory coordination adapter is process-local and cannot enforce global model capacity.

AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_ENABLED=true is safe to run while AIBS_MODEL_CLUSTER_ENABLED=false. The monitor elects one process through Valkey, polls configured Ollama nodes, writes a tiny placement health stamp, and writes richer telemetry to a separate key. Use AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_RUN_IN_API=false and the standalone monitor in multi-node or latency-sensitive deployments. This proves fresh residency before routing model calls through the global scheduler. Keep the monitor lease TTL longer than one poll interval plus one poll timeout, and keep the full takeover bound inside the residency staleness window: poll_interval + lease_ttl + standby_retry + 2 * poll_timeout.

Do not enable AIBS_MODEL_CLUSTER_ENABLED until the monitor is running, audit is green, observations stay fresh across a monitor kill/takeover test, worker resident weights are measured, and a live broad query succeeds. The placement primitive fails closed when node residency is missing or older than AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS; without the monitor, every request should look stale even when model nodes are idle.

Shared-tenancy note: the AI boxes may also host non-AIBS Ollama models. The global scheduler identifies AIBS-owned models from the configured worker roster and treats every other resident model in the fresh node telemetry snapshot as foreign residency. Foreign resident GB is subtracted from the node budget before placement. AIBS roster models visible in /api/ps are not subtracted as foreign workloads; they remain governed by AIBS leases and worker sizing.

System Health keeps two deliberately separate timelines:

• the elected model monitor writes model-node and host telemetry history to the shared model-coordination pool;

• each API environment writes ingestion, answer, and runtime workload history to its own application-coordination pool.

Workload sampling starts immediately with the API and does not depend on global model-monitor election. A short environment-local lease prevents duplicate writers when multiple API instances share the same application coordination namespace. The history settings control how much of each timeline the API can return to the frontend:

• AIBS_MODEL_CLUSTER_HEALTH_HISTORY_WINDOW_SECONDS

• AIBS_MODEL_CLUSTER_HEALTH_HISTORY_MAX_SAMPLES

• AIBS_MODEL_CLUSTER_HEALTH_HISTORY_TTL_SECONDS

Keep the history TTL greater than or equal to the displayed history window.

Optional Host Telemetry Agent

For richer System Health charts, install the lightweight host telemetry agent on each AI box and expose it only on the trusted management network:

python scripts/aibs_node_telemetry_agent.py --host 0.0.0.0 --port 8765

The agent is read-only. It reports CPU, memory, GPU busy, raw VRAM, raw GTT, and combined GPU memory from Linux /proc and DRM sysfs. AMD APU/iGPU deployments can place model buffers in GTT, so operator-facing capacity should use combined GPU memory rather than raw VRAM alone. The agent does not run model calls, mutate Valkey, or make placement decisions. The backend polls the URLs configured through AIBS_MODEL_CLUSTER_NODE_HOST_TELEMETRY_URLS and stores those values beside the fenced residency snapshots for operator visibility.

If the agent is not configured or unavailable, the node remains schedulable when its fenced residency facts are fresh. System Health will show host telemetry as unavailable for that node.

Valkey Placement Test Gate

The global placement primitive requires Valkey 8+. The Valkey Lua scripts use server-side TIME while writing placement state so all API processes share one clock authority. Do not enable the global scheduler against older Redis-compatible deployments.

Run the atomic placement tests against a real Valkey instance:

AIBS_TEST_VALKEY_URL=redis://localhost:6379/15 \
python -m pytest tests/infrastructure/model_scheduling/test_redis_atomic_placement.py

Without AIBS_TEST_VALKEY_URL, the real Valkey behavior test is skipped and only the deterministic in-memory/fake-client tests run. The legacy AIBS_TEST_REDIS_URL name remains a temporary compatibility fallback.

Residency Monitor Takeover Gate

Before enabling AIBS_MODEL_CLUSTER_ENABLED=true, prove that the elected residency monitor can die without leaving stale node health trusted by the placement scheduler:

python -m aibs_backend.interfaces.cli.prove_model_monitor_takeover --json

The proof command:

1. creates an isolated Valkey namespace;

2. starts monitor owner A as a subprocess;

3. waits for owner A to publish healthy samples for every enabled model node;

4. starts monitor owner B as a standby;

5. hard-kills owner A, without graceful lease release;

6. waits for owner B to publish newer healthy samples;

7. verifies correctness: no pre-kill owner loss and no successful fenced writes from a deposed owner;

8. measures handover duration separately from correctness;

9. fails the availability gate unless the target deployment topology observes fresh replacement samples inside AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS;

10. captures owner stderr and classifies expected shutdown artifacts separately from real owner churn;

11. deletes the isolated proof namespace so proof scheduler hashes do not accumulate in Valkey.

The safe timing bound is:

residency_monitor_poll_interval
+ residency_monitor_lease_ttl
+ residency_monitor_standby_retry
+ 2 * residency_monitor_poll_timeout
<= max_residency_staleness

With the current recommended values, 3 + 6 + 1 + 2 * 2 = 14s, which fits inside the 15s staleness window.

For deployment-grade evidence, run the proof repeatedly and require every run to pass:

python -m aibs_backend.interfaces.cli.prove_model_monitor_takeover --json --repeat 3

When running from the local dev machine through an SSH tunnel to Valkey, use the split contract explicitly:

python -m aibs_backend.interfaces.cli.prove_model_monitor_takeover \
  --json --repeat 3 --handover-gate report-only

In that mode, correctness still gates the result. Long handover duration is reported as an availability finding because the dev tunnel is not the production transport. On the demo topology, where the backend runs on an AI box and reaches Valkey over LAN, use the default strict handover gate before enabling shared traffic.

Treat passed=true as a scheduler liveness gate, not as full production approval. It proves election/failover and fresh node telemetry. It does not prove answer quality or routed LLM behavior under real user load. If correctness fails, do not enable the cluster scheduler. If only the dev tunnel handover report fails, document the admission-pause duration and rerun the strict proof from the target topology.

Controlled Enable Smoke Test

After the audit and takeover proof are green, enable the scheduler only in the target test environment:

AIBS_MODEL_CLUSTER_ENABLED=true

Then restart the API process. Do not rely on --reload to notice a .env-only change; some reload configurations watch Python files but not environment files.

Run:

python -m aibs_backend.interfaces.cli.audit_model_capacity --json

Expected enable-state result:

• cluster_enabled=true

• ready_to_enable_global_scheduler=true

• ready_for_global_scheduler=true

Then run the real API path:

1. one narrow or single-document query;

2. one broad or agentic query;

3. two concurrent broad queries from separate conversations or users when possible.

Acceptance evidence:

• global placement acquisition/release logs appear;

• leases include node_id and node_base_url;

• answer quality does not regress;

• no request remains stuck in answering state after completion/cancel;

• Valkey active leases are released or expire according to TTL.

If placement acquisition works but release warnings appear, stop before demo and inspect the placement release path. A scheduler that admits work but leaks capacity is not enterprise-stable.