Environment Setup

This guide shows how to create the backend .env file for local development.

Source of truth:

  • the variables documented here are based on what the backend currently reads from src/aibs_backend/settings.py in the backend repository

  • if this page ever drifts from the code, trust src/aibs_backend/settings.py

Before You Start

The backend expects these runtime dependencies:

  • OpenSearch reachable from your machine

  • the local embedding model bundle under models/embeddings/bge-m3

  • the local reranker bundle under models/reranking/bge-reranker-v2-m3 if reranking is enabled

  • local Docling artifacts under models/docling if AIBS_DOCLING_ARTIFACTS_PATH is configured

  • an Ollama-compatible LLM endpoint if grounded answering is enabled

How To Create .env

  1. Create a file named .env in the repository root.

  2. Copy the template below into it.

  3. Replace the placeholder values with your local settings.

  4. Never commit real credentials into git.

Section Notes

API server

  • AIBS_HOST and AIBS_PORT control where FastAPI listens.

  • AIBS_CORS_ORIGINS should include the frontend origin you use during local development.

OpenSearch

  • AIBS_OPENSEARCH_INDEX_TEMPLATE is workspace-scoped and should usually be left as-is.

  • AIBS_OPENSEARCH_REQUEST_TIMEOUT_SECONDS is the general client/search timeout. Use a higher value for agentic, multi-document retrieval because document-local neighbor and coverage searches can legitimately run longer than a small single search request under load.

  • AIBS_OPENSEARCH_BULK_REQUEST_TIMEOUT_SECONDS is used for chunk bulk indexing, where large PDFs can legitimately take longer than search requests

  • AIBS_OPENSEARCH_BULK_BATCH_SIZE controls how many vector records are sent per bulk indexing call; lower it for memory-constrained/local boxes, raise cautiously only after live ingestion stays stable

  • AIBS_OPENSEARCH_MAX_CONNECTIONS raises the OpenSearch HTTP connection pool above the tiny default so concurrent job/search activity does not churn connections unnecessarily

  • AIBS_OPENSEARCH_SEMANTIC_DOCUMENT_CANDIDATE_MULTIPLIER controls how many vector chunk hits are fetched per requested document result before document-level grouping

  • AIBS_OPENSEARCH_SEMANTIC_DOCUMENT_MAX_CANDIDATES caps that semantic search candidate window so large source-list searches do not pull an unbounded number of vectors

  • AIBS_DOCUMENT_CATALOG_INDEX names the lightweight OpenSearch read model used by /documents/catalog, search hydration, catalog summaries, and large-workspace paging

  • AIBS_DOCUMENT_PROFILE_INDEX names the OpenSearch profile projection used for cheap document discovery before chunk retrieval

  • AIBS_SECTION_SUMMARY_INDEX names the OpenSearch section/table summary projection used to narrow broad questions before full chunk evidence is fetched

  • AIBS_PROJECTION_WORKSPACE_ROUTING_ENABLED=true routes catalog/profile/summary projection writes and workspace-scoped reads by workspace_id. Enable it only with a full rebuild of those projection indexes; routed searches will not see documents previously indexed without routing.

  • AIBS_RETRIEVAL_PYRAMID_DISCOVERY_ENABLED=true makes broad discovery read profile/summary hits before chunk evidence. Keep it false for the clean pyramid-off baseline; enable it only for the pyramid-on A/B run.

  • keep AIBS_OPENSEARCH_SSL and AIBS_OPENSEARCH_VERIFY_CERTS aligned with the actual cluster you connect to

  • use real credentials only in your local .env, never in docs or committed files

  • if your OpenSearch cluster lives on a VM, you can forward it locally over SSH and still keep:

    • AIBS_OPENSEARCH_HOST=localhost

    • AIBS_OPENSEARCH_PORT=9200

  • example tunnel:

ssh -fN -L 9200:<YOUR_VM_IP>:9200 <YOUR_VM_USER>@<YOUR_VM_IP>

Embeddings

  • the repository is already set up for the local BGE-M3 embedding model

  • AIBS_EMBEDDING_LOCAL_FILES_ONLY=true is the safest default for offline or controlled environments

  • the current local .env pins AIBS_EMBEDDING_DEVICE=cuda; use auto only when you want portable CPU/GPU fallback across machines

  • supported explicit device values are cpu, cuda, cuda:N, gpu, gpu:N, and auto

  • explicit cuda/gpu values fail fast if torch.cuda is not available; use auto for portable local and VM configs

  • AMD/ROCm PyTorch exposes AMD GPUs through the torch.cuda compatibility API, so do not use a separate rocm value

Reranker

  • reranking improves retrieval quality but depends on the local BGE reranker bundle

  • if you do not want reranking during a debugging session, set AIBS_RERANKER_ENABLED=false

  • AIBS_RERANKER_MAX_CANDIDATES controls how many assembled candidates are rescored by the cross-encoder. The demo/default value is 80, matching the fast multi-document evidence cap so broad pages do not leave selected evidence in an unreredanked tail.

  • AIBS_RERANKER_DIAGNOSTIC_TEXT_PREVIEW_ENABLED=false keeps reranker diagnostic logs to document/chunk identifiers and scores. Enable it only in private dev investigations when text previews are acceptable in logs.

  • AIBS_RERANKER_SCORE_ACTIVATION=sigmoid converts BGE reranker logits into probability-like scores before sufficiency checks. Use identity only when a reranker model already returns calibrated scores on the scale you want to evaluate directly.

  • AIBS_RERANKER_DEVICE accepts the same device values as embeddings: cpu, cuda, cuda:N, gpu, gpu:N, and auto

  • keep AIBS_RERANKER_DEVICE=auto unless you intentionally want startup to fail when CUDA/ROCm torch acceleration is unavailable

Docling PDF parser

  • if AIBS_DOCLING_ARTIFACTS_PATH is set, the parser passes that folder to Docling as its local artifacts_path

  • this is the right setup if you want document parsing to stay independent from live Hugging Face availability after the artifacts have been downloaded locally

  • the current recommended repo-local path is models/docling

  • the current local .env pins AIBS_DOCLING_DEVICE=cuda so parsing fails fast if the expected CUDA/ROCm torch acceleration is unavailable

  • use AIBS_DOCLING_DEVICE=auto when you want the parser to fall back to CPU on machines without GPU-capable torch

  • AIBS_DOCLING_NUM_THREADS and the AIBS_DOCLING_*_BATCH_SIZE values cap Docling’s page-processing pressure; the template favors smoother ingestion over peak throughput

  • AIBS_DOCLING_QUEUE_MAX_SIZE limits Docling’s internal queued work and helps avoid memory spikes on large PDFs

  • use a lower value such as 8 on laptop/local-dev machines that share memory with the API, embedding, reranker, and LLM clients; keep higher values such as 50 for dedicated AI-box style deployments that have enough memory headroom

  • AIBS_PARSING_ENABLE_FORMULA_MODEL=false is the safe default; formula enrichment is expensive and remains disabled unless this flag is explicitly set to true

  • when formula enrichment is enabled, the parser still gates CodeFormulaV2 behind the PDF formula signal threshold, so ordinary PDFs do not load the formula model unnecessarily

  • with AIBS_DOCLING_ARTIFACTS_PATH=models/docling, CodeFormulaV2 must be available locally under models/docling/docling-project--CodeFormulaV2 before enabling formula enrichment

  • when the downloaded RapidOCR bundle uses Docling’s older *_infer filenames, the backend now creates the compatible torch aliases it needs during startup so OCR warmup can still stay local

Chunking

  • AIBS_CHUNKING_ABSTRACTIVE_SUMMARIES_ENABLED=true enables the optional assistant-backed summarization stage during chunking in the current local .env

  • set it to false only when you intentionally want deterministic extractive summaries without model enrichment

Ingestion worker

  • AIBS_INGESTION_WORKER_ENABLED controls whether the API process starts its embedded ingestion worker

  • AIBS_INGESTION_WORKER_COUNT controls how many worker lanes run inside the process; set it to 0 to pause API-hosted ingestion without disabling job submission

  • AIBS_INGESTION_WORKER_CLAIM_SCAN_SIZE controls how many claimable jobs a lane scans when an OpenSearch optimistic-concurrency conflict occurs

  • AIBS_INGESTION_MAX_CONCURRENT_PARSE_STAGES bounds concurrent parse/OCR stages across worker lanes in one process

  • AIBS_INGESTION_MAX_CONCURRENT_CHUNK_STAGES bounds concurrent chunk/summarization stages across worker lanes in one process

  • AIBS_INGESTION_MAX_CONCURRENT_INDEX_STAGES bounds concurrent embedding/indexing stages across worker lanes in one process

  • AIBS_INGESTION_MAX_CONCURRENT_SEMANTIC_PROFILE_STAGES bounds post-index semantic profile enrichment across worker lanes; documents are already searchable before this stage completes

  • AIBS_INGESTION_MAX_CONCURRENT_HEAVY_MODEL_STAGES bounds local heavyweight model stages shared by parse/OCR, embedding/indexing, and memory-heavy chunking so they do not compete for the same GPU/torch memory at the same time

  • AIBS_INGESTION_MEMORY_HEAVY_PAGE_COUNT_THRESHOLD, AIBS_INGESTION_MEMORY_HEAVY_TABLE_COUNT_THRESHOLD, AIBS_INGESTION_MEMORY_HEAVY_SECTION_COUNT_THRESHOLD, and AIBS_INGESTION_MEMORY_HEAVY_CHUNK_COUNT_THRESHOLD classify expanded document artifacts that should reserve the heavy-model lane during chunking; this prevents annual-report-size documents from chunking while another heavy parse is active

  • AIBS_INGESTION_WORKER_IDLE_SLEEP_SECONDS controls how long the background worker sleeps when no job is available

  • AIBS_INGESTION_WORKER_ERROR_SLEEP_SECONDS controls the backoff after worker-loop errors

  • AIBS_INGESTION_WORKER_LEASE_REAP_INTERVAL_SECONDS controls how often an idle worker reaps expired job leases so jobs abandoned by another process become retryable without requiring an API restart

  • AIBS_INGESTION_EXPIRED_LEASE_CLAIM_MULTIPLIER bounds true worker crash loops separately from caught stage failures; with the default 10, a job with max_attempts=3 can survive up to roughly 30 expired-lease claim cycles before the reaper marks it failed as a suspected worker crash loop

  • AIBS_INGESTION_UPLOAD_QUIET_PERIOD_SECONDS pauses API-hosted ingestion claims while document upload requests are active and for the configured quiet window afterward; this is only a fallback for single uploads and clients that do not open an explicit batch

  • AIBS_INGESTION_UPLOAD_BATCH_TTL_SECONDS is the abandoned-browser failsafe for explicit upload batches; each file upload refreshes the batch, and explicit batch completion releases API-hosted ingestion immediately

  • AIBS_INGESTION_JOB_INDEX is the operational queue for durable ingestion jobs, not just a storage name; it is a literal OpenSearch index name and is not automatically derived from AIBS_ENV

  • if AIBS_INGESTION_JOB_INDEX is unset, the backend default is aibs-ingestion-jobs; the sample .env uses aibs-ingestion-jobs-dev only as a development namespace convention

  • use a unique value per local developer, branch sandbox, VM, staging, and production deployment unless all API/worker processes are intentionally cooperating on the same ingestion queue

  • do not run production or demo VM ingestion against aibs-ingestion-jobs-dev unless that is deliberately the shared queue for that deployment

  • sharing OpenSearch credentials or search indices is not the same as sharing ingestion ownership; any process polling the same AIBS_INGESTION_JOB_INDEX can claim and mutate jobs in that queue

  • keep these values conservative on the demo VM to avoid unnecessary polling pressure

  • set AIBS_INGESTION_WORKER_ENABLED=false only for a deliberate API-only deployment where a separate worker process/node owns parse/chunk/index execution; do not use it as a bulk-upload workaround

  • run the standalone worker with python -m aibs_backend.interfaces.cli.ingestion_worker on the worker node; the CLI runs when invoked and still uses the sleep/reap timings above

Conversation memory

  • AIBS_CONVERSATION_MEMORY_ENABLED enables Phase 5 Slice A conversation context.

  • recent turns are read from authoritative query/answer JSON history, not from the memory index.

  • AIBS_CONVERSATION_MEMORY_INDEX is the separate OpenSearch index used only for derived older-turn retrieval.

  • AIBS_CONVERSATION_MEMORY_RECENT_TURNS controls how many latest persisted turns are placed in the planner and answer prompts.

  • AIBS_CONVERSATION_MEMORY_RETRIEVAL_TOP_K controls how many older relevant turns are retrieved for the answer prompt; these are intentionally not shown to the planner.

  • AIBS_CONVERSATION_MEMORY_MAX_TURN_CHARS clips each turn before it is embedded or inserted into prompts.

  • memory indexing is best effort and logs warning telemetry on failure; deleting a conversation purges memory first and fails loudly if that purge fails.

Answer pagination

  • AIBS_ANSWER_PAGE_DOCUMENT_LIMIT controls how many ranked documents are sent into one grounded answer page before the backend returns a continuation token. The default is 10 for Gemma4:26b-era context discipline; raise it only after benchmark evidence shows the larger model/context budget can preserve answer quality.

  • AIBS_ANSWER_CONTINUATION_TOKEN_TTL_SECONDS controls how long continuation tokens remain valid on disk. The default is 86400 seconds (24 hours), which keeps browser reloads and long automatic continuation chains usable while reaping orphaned tokens from failed or abandoned runs.

  • AIBS_ANSWER_MAP_CONCURRENCY_LIMIT caps parallel per-document map calls when a large evidence page uses map-reduce answering. The global agent-flow scheduler may reduce this request-local ceiling when current node capacity is lower. The default is 5; raise it only after VM benchmarks show the worker fleet handles the concurrency.

  • AIBS_DOCUMENT_WORK_MODEL_DEADLINE_SECONDS is the absolute wall-clock budget for all model attempts on one map/refine document. The default is 45 seconds. An assistant exceeding it is cancelled, its model lease is released, and the document is published as a typed reliability gap instead of holding the complete answer indefinitely. This is distinct from the HTTP inactivity timeout, which may reset while a stalled provider streams data.

  • AIBS_ANSWER_MAP_REDUCE_EVIDENCE_LIMIT controls when a large evidence pack switches from single-pass answering to map-reduce answering. The default is 40 evidence items.

  • AIBS_ANSWER_FAST_MULTI_DOC_EVIDENCE_CAP controls how many evidence items a fast-tier multi-document query may retrieve before mapping. This is separate from the map-reduce trigger, so broad non-heavy questions can carry more evidence without changing when map-reduce activates. Keep it at or above AIBS_ANSWER_MAP_REDUCE_EVIDENCE_LIMIT. The default is 80.

  • AIBS_ANSWER_MULTI_DOC_CHUNKS_PER_DOCUMENT controls the per-document evidence target before the total cap is applied. The demo/default value is 8, so a 10-document broad page can use the full 80-item fast multi-document cap.

  • AIBS_ANSWER_MAP_REDUCE_FALLBACK_CHUNK_LIMIT triggers map-reduce when too many raw chunk excerpts are needed because summaries are missing, degraded, or too thin. The default is 20 chunk excerpts.

  • AIBS_STRICT_GROUNDED_QUARANTINED_MODEL_IDS is the comma-separated deny-list for strict Grounded map, refine, and compose model admission. Quarantined models remain available to non-strict Creative work. The default excludes gpt-oss:20b until contract and latency evaluations explicitly clear it.

  • AIBS_ANSWER_METRICS_SINCE resets operational answer metrics without deleting persisted answers. Set it to an ISO-8601 UTC timestamp such as 2026-07-07T10:25:45Z; answer counts, token totals, model token distribution, and the answer trend ignore answers created before that boundary. Leave it blank to report over the normal recent-answer window.

  • AIBS_DOCUMENT_DISCOVERY_MIN_ENTITY_ANCHOR_DOCUMENTS and AIBS_DOCUMENT_DISCOVERY_MIN_UNDELIMITED_ENTITY_ANCHOR_DOCUMENTS control the catalog entity-bound fast path. Delimited identity references such as quoted, parenthesized, or bracketed document names must all resolve before the fast path is allowed, and the first value is the minimum number of anchored documents for that mode. Undelimited identity references use the second, stricter floor so a broad topic/category query does not steal scope from semantic/profile discovery. Defaults: 2, 3.

  • AIBS_DOCUMENT_PROFILE_BINDING_TIMEOUT_SECONDS bounds profile-index lookup when fast-binding explicit document references. Catalog/title binding runs first; slow profile lookup is skipped after this budget and normal discovery continues. The default is 3 seconds.

  • AIBS_PLANNING_CATALOG_CACHE_TTL_SECONDS controls the short in-process cache for dispatcher planning catalogs loaded from lightweight catalog projection entries. The default is 0 (disabled) because knowledge enable/disable and delete operations require a shared invalidation/version signal before this can be safely enabled in multi-worker production.

  • AIBS_PLANNING_CATALOG_CACHE_MAX_WORKSPACES bounds the optional in-process planning-catalog cache when the TTL is enabled. The default is 64.

Document Complexity

Document-local map work is classified from intrinsic document metadata before a worker is admitted. This avoids treating a 2-page table-heavy file and a 150-page narrative file as equivalent work.

  • AIBS_DOCUMENT_COMPLEXITY_MEDIUM_PAGE_COUNT and AIBS_DOCUMENT_COMPLEXITY_LARGE_PAGE_COUNT classify size pressure from parsed page count. Defaults: 25, 100.

  • AIBS_DOCUMENT_COMPLEXITY_MEDIUM_CHUNK_COUNT and AIBS_DOCUMENT_COMPLEXITY_LARGE_CHUNK_COUNT classify size pressure from indexed chunk count. Defaults: 120, 500.

  • AIBS_DOCUMENT_COMPLEXITY_MEDIUM_SECTION_COUNT and AIBS_DOCUMENT_COMPLEXITY_LARGE_SECTION_COUNT classify section-rich documents. Defaults: 50, 200.

  • AIBS_DOCUMENT_COMPLEXITY_MEDIUM_TABLE_COUNT and AIBS_DOCUMENT_COMPLEXITY_LARGE_TABLE_COUNT classify table-heavy documents. Defaults: 4, 20.

  • AIBS_DOCUMENT_COMPLEXITY_MEDIUM_REQUEST_SIGNAL_COUNT and AIBS_DOCUMENT_COMPLEXITY_LARGE_REQUEST_SIGNAL_COUNT classify typed request complexity from the number of requested fields and subjects, not from natural-language keywords. Defaults: 4, 10.

  • AIBS_DOCUMENT_COMPLEXITY_SMALL_EVIDENCE_BUDGET, AIBS_DOCUMENT_COMPLEXITY_MEDIUM_EVIDENCE_BUDGET, AIBS_DOCUMENT_COMPLEXITY_LARGE_EVIDENCE_BUDGET, and AIBS_DOCUMENT_COMPLEXITY_DEGRADED_LARGE_EVIDENCE_BUDGET bound how many evidence items a document-local worker receives for each complexity class. Defaults: 4, 8, 12, 16.

  • AIBS_DOCUMENT_COMPLEXITY_PAGE_CONSTRAINED_EVIDENCE_BUDGET keeps page/section-constrained work compact while still giving the worker enough local context. Default: 6.

  • AIBS_DOCUMENT_COMPLEXITY_STRUCTURED_OUTPUT_MIN_EVIDENCE_BUDGET gives chart and table outputs a minimum evidence budget when enough evidence exists. Default: 10.

Heavy-tier multi-doc combine

The execution planner’s intent signals classify each request into a latency tier. Fast queries (facts, inventory, single-doc, small multi-doc) keep the tight AIBS_ANSWER_PAGE_DOCUMENT_LIMIT page budget. Broad discovered/planner scopes also page first so users receive an inspectable first answer and continuation token instead of waiting for a monolithic all-document synthesis.

Explicit selected-document heavy queries (“compare these named documents”) can cover the selected set in one combined answer:

  • AIBS_ANSWER_HEAVY_COMBINE_DOCUMENT_LIMIT is the maximum number of documents a single explicit heavy combine folds in before it falls back to paging. The default is 64. Retrieval/document binding already narrows the selected set, so this bounds work, not the workspace size. Raise it only if your model/context budget can synthesize more documents at once; lower it to cap explicit heavy-query latency.

  • AIBS_ANSWER_HEAVY_EVIDENCE_CHUNK_CAP is the total evidence-chunk budget for a heavy combine, so every document keeps its full chunks-per-document share instead of being thinned to fit the fast-tier evidence cap (AIBS_ANSWER_MAP_REDUCE_EVIDENCE_LIMIT). The default is 320. It does not change when map-reduce is triggered, only how much evidence a heavy combine may gather.

  • AIBS_SCOPED_MULTI_DOC_CANDIDATE_MULTIPLIER_CAP caps reranker/search overfetch after the document set is already known. It does not change document scoping. Higher values preserve sparse-document recall at higher latency; lower values reduce latency after broad scopes. The default is 4.

Workspace document inventory

  • AIBS_CATALOG_INVENTORY_LIST_LIMIT controls how many documents the deterministic workspace document table (the WORKSPACE ANSWER catalog listing) shows per page. The default is 25. When the workspace holds more documents than one page, the answer states the true total (for example, “Showing the first 25 of 930 documents in this workspace”) and returns a continuation token so the client can request the next page (“show more”). Raise it to show more rows per page; lower it for tighter responses. It does not affect grounded answering, which is governed by AIBS_ANSWER_PAGE_DOCUMENT_LIMIT.

Remote GPU Worker Environment

Use this split when the API node should accept uploads and submit durable jobs, while a separate GPU worker node claims and executes parse/chunk/index.

API node:

AIBS_INGESTION_WORKER_ENABLED=false

GPU worker node:

AIBS_DOCLING_ARTIFACTS_PATH=/opt/aibs/models/docling
AIBS_EMBEDDING_MODEL=BAAI/bge-m3
AIBS_EMBEDDING_MODEL_PATH=/opt/aibs/models/embeddings/bge-m3
AIBS_EMBEDDING_DIMENSION=1024
AIBS_RERANKER_MODEL_PATH=/opt/aibs/models/reranking/bge-reranker-v2-m3
AIBS_RERANKER_MAX_CANDIDATES=80
AIBS_RERANKER_SCORE_ACTIVATION=sigmoid
AIBS_DOCLING_DEVICE=cuda
AIBS_EMBEDDING_DEVICE=cuda
AIBS_RERANKER_DEVICE=auto

AIBS_INGESTION_WORKER_ENABLED is only used by the FastAPI lifespan to decide whether the API process should start its embedded worker. The standalone worker CLI always runs when invoked, so the GPU worker env does not need that flag.

The GPU worker must use the same OpenSearch service as the API process and must be able to read/write the upload and document storage roots that ingestion actually touches.

For the demo topology where the VM owns storage and the AI-on-prem box is compute-only, mount only the VM-owned ingestion roots on the GPU worker. Do not mount over the GPU server’s whole /opt/aibs directory, because that path also holds the local model cache and worker virtual environment:

GPU server local runtime:
  /opt/aibs/models
  /opt/aibs/models/.venvs/aibs-dependecy-gpu-ingestion

VM-owned shared data mounted on GPU server:
  /opt/aibs/.aibs_data/uploads
  /opt/aibs/.aibs_data/documents

These two paths are the required ingestion filesystem boundary:

AIBS_UPLOAD_ROOT=.aibs_data/uploads
AIBS_DOCUMENT_ROOT=.aibs_data/documents

If the worker might be launched from another directory, prefer absolute paths for those two roots on the worker node:

AIBS_UPLOAD_ROOT=/opt/aibs/.aibs_data/uploads
AIBS_DOCUMENT_ROOT=/opt/aibs/.aibs_data/documents

AIBS_WORKSPACE_ROOT, AIBS_CONVERSATION_ROOT, AIBS_QUERY_ROOT, AIBS_ANSWER_ROOT, and AIBS_STAGING_ROOT are API-side storage concerns for workspace management, conversations, query records, answer artifacts, and upload staging. The standalone ingestion worker does not need shared access to them for parse/chunk/index. Mounting the parent .aibs_data directory is still acceptable for operational simplicity, as long as it does not overwrite /opt/aibs/models.

The worker also needs to point at the same OpenSearch service used by the API. OpenSearch is a separate search/storage boundary and may live on its own VM or service host. Do not infer the OpenSearch host from the GPU/LLM/model server IP unless OpenSearch is actually deployed there. In this codebase OpenSearch is configured with the AIBS_OPENSEARCH_* variables, not OPENSEARCH_URL:

AIBS_OPENSEARCH_HOST=<opensearch-host>
AIBS_OPENSEARCH_PORT=9200
AIBS_OPENSEARCH_SSL=false
AIBS_OPENSEARCH_VERIFY_CERTS=false
AIBS_OPENSEARCH_USERNAME=<same-user-as-api>
AIBS_OPENSEARCH_PASSWORD=<same-password-as-api>
AIBS_OPENSEARCH_INDEX_TEMPLATE=.abs-content-{workspace_id}
AIBS_INGESTION_JOB_INDEX=aibs-ingestion-jobs-gpu-worker
AIBS_DOCUMENT_CATALOG_INDEX=aibs-document-catalog
AIBS_DOCUMENT_PROFILE_INDEX=aibs-document-profiles
AIBS_SECTION_SUMMARY_INDEX=aibs-section-summaries
AIBS_RETRIEVAL_PYRAMID_DISCOVERY_ENABLED=true

On AMD/ROCm boxes, cuda and auto both route through PyTorch’s torch.cuda compatibility layer. Keep the current pinned cuda values when you want fail-fast GPU startup; use auto for portable worker images that may run on CPU-only hosts.

For the current AI-on-prem Ryzen APU worker, the validated Python runtime uses AMD’s ROCm 7.2.1 PyTorch wheels from repo.radeon.com, not the public PyTorch ROCm wheels:

torch 2.9.1+rocm7.2.1.lw.gitff65f5bc
torchvision 0.24.0+rocm7.2.1.gitb919bd0c
triton 3.5.1+rocm7.2.1.gita272dfa8

The worker venv lives under the model folder so it stays isolated from system Python:

/opt/aibs/models/.venvs/aibs-dependecy-gpu-ingestion

Local storage

These paths are relative to the repo root by default and back the local JSON/file persistence used in development.

  • AIBS_INGESTION_JOB_INDEX is the OpenSearch index used for durable ingestion job state and worker claiming. Treat it as a queue namespace, not an environment-aware setting: AIBS_ENV=production does not rename the queue. Use one shared value only for processes that are meant to cooperate on the same jobs, and use separate values for local developer machines, branch sandboxes, demo VMs, staging, production, and GPU worker environments that must not claim each other’s work.

  • AIBS_DOCUMENT_CATALOG_INDEX is the OpenSearch index used for the workspace document catalog projection. It is not the aggregate store; rebuild it from JSON documents if it is missing or stale.

  • AIBS_DOCUMENT_PROFILE_INDEX is the OpenSearch profile projection used before broad chunk retrieval.

  • AIBS_SECTION_SUMMARY_INDEX is the OpenSearch section/table summary projection used before broad chunk retrieval.

  • AIBS_PROJECTION_CONTROL_INDEX is the OpenSearch control-plane index used for the catalog projection manifest and mutation outbox. The default is aibs-projection-control. Keep this shared by every API and ingestion worker process that reads/writes the same document store. A missing or dirty manifest intentionally makes broad planning fall back to JSON source truth until reconciliation marks the projection clean.

  • AIBS_PROJECTION_CONTROL_SUCCEEDED_EVENT_RETENTION_DAYS controls pruning for terminal succeeded catalog projection outbox events after successful catalog reconciliation. The default is 7; set 0 to retain succeeded events indefinitely.

  • AIBS_PROJECTION_CONTROL_RECONCILED_EVENT_RETENTION_DAYS controls pruning for terminal reconciled outbox events. The default is 20; set 0 to retain reconciled events indefinitely.

  • Pending and failed events are never pruned by this retention policy. They are unresolved drift/audit signals and must remain visible until reconciliation converts them to terminal reconciled events.

  • AIBS_PROJECTION_WORKSPACE_ROUTING_ENABLED optionally routes catalog/profile/summary projection documents by workspace_id. Leave it false unless you are prepared to rebuild AIBS_DOCUMENT_CATALOG_INDEX, AIBS_DOCUMENT_PROFILE_INDEX, and AIBS_SECTION_SUMMARY_INDEX after enabling it.

Document Catalog Rebuild CLI

The catalog projection is write-through for new document saves and deletes, with retry and best-effort logging if OpenSearch is briefly unavailable. Existing workspaces, restored backups, renamed catalog indexes, or suspected projection drift need a rebuild:

python -m pdm run python -m aibs_backend.interfaces.cli.rebuild_document_catalog [--workspace <id>] [--batch-size 200] [--dry-run]

The command loads .env from the current working directory and writes to the OpenSearch index named by AIBS_DOCUMENT_CATALOG_INDEX.

Successful non---only-missing rebuilds also reconcile the catalog projection manifest in AIBS_PROJECTION_CONTROL_INDEX. This establishes the clean baseline required by the dispatcher to trust projection-backed broad planning. --only-missing does not clear dirty state because it cannot prove stale rows were removed.

If the catalog mapping schema is outdated, run a full rebuild without --workspace. The full rebuild drops and recreates only the catalog projection from JSON source-of-truth documents. A workspace-scoped rebuild refuses stale schemas because the catalog index is shared across workspaces.

Document Catalog Reconciliation CLI

Use this command when the dispatcher logs dirty catalog projection fallback, after restoring backups, after changing projection indexes, or before enabling projection-first planning for existing workspaces:

python -m pdm run python -m aibs_backend.interfaces.cli.reconcile_document_catalog_projection [--workspace <id>] [--batch-size 200]

The reconciler replays the authoritative JSON documents into the catalog projection, removes stale projection rows, refreshes OpenSearch, and marks the workspace catalog manifest clean. It does not offer an --only-missing mode because partial repair cannot prove broad completeness.

Retrieval Pyramid Rebuild CLI

The retrieval pyramid projections are write-through for new document saves and deletes. Existing workspaces, restored backups, renamed projection indexes, or suspected drift need a rebuild:

python -m pdm run python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid [--workspace <id>] [--batch-size 200] [--dry-run]

The command writes to AIBS_DOCUMENT_PROFILE_INDEX and AIBS_SECTION_SUMMARY_INDEX. If either mapping schema is outdated, run a full rebuild without --workspace; scoped rebuilds refuse stale schemas because the indexes are shared across workspaces.

Auth and identity

Auth uses OpenSearch credentials and stores an opaque server-side session id in an HTTP-only cookie. AIBS product access still requires aibs-access, and workspace operations are then governed by the workspace role matrix. The OpenSearch auth adapter supports standard OpenSearch Security authinfo responses and Logserver clusters that authenticate through /_logserver/login.

AIBS_AUTH_DEV_MODE=true
AIBS_AUTH_DEV_ROLES=aibs-access,aibs-admin,developer
AIBS_AUTH_SESSION_COOKIE_NAME=aibs-session
AIBS_AUTH_SESSION_TTL_SECONDS=3600
AIBS_AUTH_COOKIE_SECURE=false
AIBS_AUTH_COOKIE_SAMESITE=lax
AIBS_AUTH_COOKIE_PATH=/
# AIBS_AUTH_COOKIE_DOMAIN=

  • AIBS_AUTH_DEV_MODE=true allows the legacy X-AIBS-User-Id header for local development only.

  • AIBS_AUTH_DEV_ROLES controls the exact roles assigned by the development auth stub. Keep production role grants in OpenSearch/Logserver; do not use dev-mode roles outside local development.

  • For local operator testing of admin/developer-only diagnostics, such as System Health node ids and model-node endpoint visibility, include the required roles explicitly, for example:

AIBS_AUTH_DEV_ROLES=aibs-access,aibs-admin,developer

After changing dev-mode roles, restart the API and sign in again so the process-local session contains the updated principal roles.

  • When AIBS_AUTH_DEV_MODE=false, the header is ignored and requests require a valid session cookie.

  • Browser sessions use a fixed TTL in this phase and are process-local. Keep API deployment single-worker until a persistent session store lands.

  • Bootstrap the first production admin before first login with:

python -m pdm run python -m aibs_backend.interfaces.cli.bootstrap_admin --username <admin-user>

  • If the user does not exist, the CLI prompts for a password and creates the user with aibs-access and aibs-admin.

  • If the user already exists, the CLI grants missing AIBS roles idempotently.

  • Passwords are read through a masked prompt and are never printed.

  • The CLI loads .env from the current working directory before reading AIBS_OPENSEARCH_* settings.

  • AIBS_IDENTITY_HEADER_NAME remains only for the explicit dev-mode compatibility path.

  • Workspace-local roles are owner, admin, member, and viewer. Owners can delete workspaces and manage ownership, owners/admins can manage non-owner members and documents, members can upload/process documents, and viewers can read workspace state and ask questions.

Language Backfill CLI

Newly ingested documents have their primary language detected automatically during parse. Documents ingested before language detection joined the pipeline carry no language metadata and appear under unknown in the workspace inventory by_language breakdown.

Populate the metadata for those documents with:

python -m pdm run python -m aibs_backend.interfaces.cli.backfill_language [--workspace <id>] [--dry-run] [--force]

  • with no flags the CLI processes every workspace

  • --workspace <uuid> restricts the run to a single workspace

  • documents that already have language metadata are skipped unless --force is passed

  • --dry-run prints what would be persisted without writing; every output line is prefixed with [DRY-RUN]

  • the CLI uses the same JsonDocumentRepository and LinguaLanguageDetector the running backend uses, so the .env and AIBS_* settings above govern its behavior

  • AIBS product user administration requires both aibs-access and aibs-admin.

  • Product admins can create users, reset passwords, toggle aibs-access, toggle aibs-admin, and disable users when the backing Logserver/OpenSearch provider supports it.

  • Workspace member management is separate from product user administration. Users should be created or granted aibs-access first, then added to a workspace by username.

  • Workspace member-add still accepts user_id as a deprecated compatibility field, but the intended operator path is username-based and directory-backed.

  • A valid Logserver user without aibs-access should receive AIBS_ACCESS_REQUIRED in non-dev mode. That is an access-policy failure, not an invalid password.

Logging

The backend writes:

  • application logs to logs/aibs.log

  • ingestion logs to logs/ingestion.log

  • retrieval logs to logs/retrieval.log

LLM

  • AIBS_LLM_PROVIDER is currently expected to be ollama

  • AIBS_LLM_BASE_URL should point to your Ollama-compatible endpoint

  • the current demo/default example above uses the dedicated ELS Compass node at 10.4.3.16:11434; global placement also admits 10.4.3.17:11434

  • AIBS_LLM_MODEL can be switched between available server models such as gemma4:26b and gpt-oss:20b depending on the quality/latency tradeoff you want to test

  • AIBS_LLM_MAX_CONCURRENT_REQUESTS serializes assistant calls through a concurrency gate; keep it at 1 when the local serving lane handles one request at a time

  • AIBS_LLM_TEMPERATURE, AIBS_LLM_TOP_P, and AIBS_LLM_TOP_K map directly to Ollama sampling options; the template keeps them aligned with Gemma 4’s recommended conversational sampling settings

  • AIBS_LLM_THINKING_ENABLED=true enables provider thinking for natural-language assistant calls in the current local .env; the backend exposes returned thinking separately from answer text for inspection

  • AIBS_LLM_THINKING_STRATEGY=auto uses Gemma 4’s documented system-token strategy for gemma4* models and Ollama’s native think=true field for other models; explicit values are ollama and gemma4_system_token

  • structured JSON calls, such as execution planning, explicitly send think=false even when this flag is true so machine-readable response contracts stay reliable

  • if a structured JSON call still returns provider thinking without final message content, the backend retries once with a stricter final-content instruction

  • AIBS_LLM_THINKING_MAX_OUTPUT_TOKENS and AIBS_LLM_THINKING_CONTEXT_LENGTH optionally give thinking-enabled answer calls a larger output and context budget than normal calls

  • if a thinking-enabled natural-language call returns thinking but no final answer content, the backend retries that same call once with thinking disabled and keeps the original thinking trace as response metadata

  • AIBS_LLM_MAX_OUTPUT_TOKENS maps to the Ollama output-token option and should be high enough for descriptive grounded answers

  • AIBS_LLM_CONTEXT_LENGTH is optional and maps to the Ollama context-length option when set

  • chat execution planning now uses the main answer model from AIBS_LLM_MODEL so routing, document scope, metadata facts, language, and answer shape come from one typed plan

  • AIBS_LLM_STARTUP_CHECK=true is useful during setup because the app will fail fast if the LLM is unreachable

  • AIBS_LLM_CAPABILITY_CONTEXT is optional; if omitted, the backend uses the branch-default capability context

  • AIBS_LLM_SUMMARIZER_* controls the optional ingestion-time abstractive summarizer lane used when AIBS_CHUNKING_ABSTRACTIVE_SUMMARIES_ENABLED=true

  • summarizer provider, base URL, model, timeout, output, context, keep-alive, top-p, and top-k fall back to the matching AIBS_LLM_* value when omitted

  • AIBS_LLM_SUMMARIZER_MAX_CONCURRENT_REQUESTS is a separate concurrency gate from chat, so long-running document enrichment does not block user answers

  • AIBS_LLM_SUMMARIZER_TEMPERATURE=0.0 is the recommended default because enrichment writes retrieval metadata and should be stable

  • AIBS_LLM_SEMANTIC_PROFILE_* controls the dedicated semantic-profile enrichment lane. Its default model is qwen3:8b; it is intentionally separate from AIBS_LLM_SUMMARIZER_* so profile backfills and ingestion enrichment do not contend with the summarizer/reduce lane used by live answers.

  • AIBS_LLM_SEMANTIC_PROFILE_MAX_CONCURRENT_REQUESTS is the per-model gate for the qwen semantic-profile lane. Role-level admission still coordinates this lane with the global model-residency planner.

  • AIBS_LLM_SEMANTIC_PROFILE_ADMISSION_QUEUE_TIMEOUT_SECONDS controls how long semantic enrichment waits for a model-admission slot. Keep this much higher than the interactive query admission timeout because backfills are expected to serialize behind the same resident model lane instead of failing fast.

  • AIBS_LLM_CONVERSATION_TITLE_* controls asynchronous first-message title enrichment. The title generator should use a small structured-extraction model such as qwen2.5:7b-instruct; it only updates conversation metadata, never the query text, reply language, routing decision, or answer content.

  • Generated titles are skipped when the user has manually renamed the conversation, and model-admission busy states leave the deterministic placeholder title in place instead of delaying the chat.

LLM heavy reasoning lane

The execution planner emits reasoning_effort=high for business/research and deep cross-document analysis; those queries route to the heavy tier. The AIBS_LLM_HEAVY_* lane lets that tier use a more capable model with its own budget, while fast queries stay on the base AIBS_LLM_* model. Every field falls back to the matching AIBS_LLM_* value when omitted, so the lane is dormant by default.

  • AIBS_LLM_HEAVY_MODEL activates a distinct heavy serving lane only when set to a model different from AIBS_LLM_MODEL. Keep it unset/blank unless a measured residency plan proves the distinct model can stay resident safely. When it equals AIBS_LLM_MODEL (or is unset) the heavy tier reuses the base answer lane and its concurrency gate, so a single model is never double-gated. If a future large heavy model is needed, prefer a measured Apache-licensed candidate such as Qwen3-30B-A3B over reintroducing the retired 31B Gemma lane.

  • When the heavy tier reuses the base answer lane, heavy sampling/output/context values are applied as per-call generation overrides on that shared assistant. This preserves the heavy answer budget without creating a second same-model residency or concurrency lane. This is a bridge for the current chat pipeline; broad or heavy work should still move toward Agent Mode execution over typed work items rather than one larger prompt.

  • AIBS_LLM_HEAVY_TEMPERATURE, AIBS_LLM_HEAVY_TOP_P, AIBS_LLM_HEAVY_TOP_K, AIBS_LLM_HEAVY_MAX_OUTPUT_TOKENS, AIBS_LLM_HEAVY_CONTEXT_LENGTH, override the base sampling/output/context for the heavy tier. When provider thinking is enabled, the heavy output/context values also become the thinking output/context budget for the call.

  • AIBS_LLM_HEAVY_KEEP_ALIVE should stay blank while the heavy tier shares AIBS_LLM_MODEL. Ollama residency is per model and last-request-wins; passing a shorter heavy keep-alive for the shared model would evict the base answer model earlier than AIBS_LLM_KEEP_ALIVE intends. Configure a heavy keep-alive only when AIBS_LLM_HEAVY_MODEL names a distinct model with a measured residency plan.

  • AIBS_LLM_HEAVY_THINKING_ENABLED turns provider thinking on for the heavy lane independently of the base lane, since heavy reasoning benefits from it.

  • The heavy lane shares the base AIBS_LLM_MAX_CONCURRENT_REQUESTS limit. Because a distinct heavy model and the base model are different models, allowing one concurrent call each is correct per-model serialization; keep enough resident memory for both to stay loaded so requests do not trigger model reloads.

  • Backend request serialization is not the same thing as Ollama model residency. Ollama can keep several model runners loaded after requests finish, and other clients connected to the same Ollama host bypass this backend’s concurrency gate. Before retrieval-latency benchmarks, check ollama ps and amd-smi; ideally only the base model should be hot before the run starts.

Shared coordination port

The backend exposes shared coordination ports for best-effort cache values and safety-critical expiring leases. Local development can use process-local memory, but multi-worker or multi-box deployments must use Valkey so all API workers share one scheduler/cache state.

AIBS_COORDINATION_BACKEND=valkey
AIBS_COORDINATION_NAMESPACE=aibs
AIBS_COORDINATION_DEFAULT_CACHE_TTL_SECONDS=86400
AIBS_COORDINATION_DEFAULT_LEASE_TTL_SECONDS=300
AIBS_COORDINATION_VALKEY_URL=redis://localhost:6379/0
AIBS_COORDINATION_VALKEY_KEY_PREFIX=aibs:coordination
AIBS_COORDINATION_VALKEY_SOCKET_TIMEOUT_SECONDS=3

  • AIBS_COORDINATION_BACKEND=memory keeps coordination state inside the current API process. This is correct for local development and tests, but it is not a cross-process safety mechanism.

  • Durable cleanup workers use the application coordination pool with a shorter, renewable lease. Configure their crash-recovery and retry budgets separately:

    AIBS_CLEANUP_WORKER_LEASE_TTL_SECONDS=30
AIBS_CLEANUP_WORKER_RETRY_DELAY_SECONDS=5
AIBS_CLEANUP_WORKER_MAX_ATTEMPTS=3

    The cleanup lease is renewed while a job is running. Its TTL controls crash failover time, not the maximum duration of a cleanup operation.

  • AIBS_COORDINATION_BACKEND=valkey stores cache and lease state in Valkey. The redis backend value remains readable only for configuration compatibility.

  • Global model placement requires Valkey 8+. The scheduler Lua scripts use server-side TIME while writing placement state so all API processes share one clock authority.

  • AIBS_COORDINATION_VALKEY_URL points to the shared Valkey endpoint used by every API worker in the deployment.

  • AIBS_COORDINATION_VALKEY_KEY_PREFIX separates environments that share one Valkey service.

  • AIBS_COORDINATION_VALKEY_SOCKET_TIMEOUT_SECONDS bounds coordination calls so scheduler decisions fail closed instead of hanging indefinitely. When the residency monitor is enabled, it must be shorter than one third of the monitor lease TTL so a single socket wait cannot consume the renewal budget. The three-second default tolerates slower Windows-to-Linux Valkey networking while preserving that safety margin for the default monitor lease.

  • AIBS_MODEL_CLUSTER_NODE_MAX_PARALLEL_REQUESTS is an authoritative per-node ceiling. It must not exceed the serving process’s actual OLLAMA_NUM_PARALLEL/runner capacity. The scheduler applies the minimum of this node limit, the model lane limit, current flow admission, and memory capacity.

  • AIBS_COORDINATION_NAMESPACE scopes future shared keys so different deployments or model hosts do not accidentally share scheduler/cache state.

  • Cache users should treat cache failures as best-effort misses. Scheduler users should treat lease failures conservatively because those leases protect model host capacity.

  • Agent-flow queue and active-flow state are already stored through this coordination boundary. With the memory adapter the behavior remains single-process; with Valkey, the same state path is shared across the deployment.

  • Document projection freshness also uses this boundary for fallback dirty-state records when the OpenSearch projection control plane cannot record a mutation. In multi-process or multi-box deployments, use Valkey here; the memory adapter cannot carry projection repair state across API processes.

  • Legacy AIBS_COORDINATION_REDIS_* names remain readable during migration, but canonical AIBS_COORDINATION_VALKEY_* values take precedence.

Shared model coordination

When separate application deployments share the same physical model nodes, they must share one model-scheduling authority without sharing application coordination. Configure the dedicated model pool in both deployments:

AIBS_MODEL_COORDINATION_BACKEND=valkey
AIBS_MODEL_COORDINATION_NAMESPACE=aibs-shared-models
AIBS_MODEL_COORDINATION_VALKEY_URL=redis://127.0.0.1:6380/2
AIBS_MODEL_COORDINATION_VALKEY_USERNAME=aibs-model-dev-api
AIBS_MODEL_COORDINATION_VALKEY_PASSWORD_FILE=C:\path\to\model-valkey-dev-api.secret
AIBS_MODEL_COORDINATION_VALKEY_KEY_PREFIX=aibs:shared:model-coordination
AIBS_MODEL_COORDINATION_VALKEY_SOCKET_TIMEOUT_SECONDS=1.5

  • Model placement, admission leases, node telemetry, monitor election, and agent-flow admission use this pool.

  • All deployments sharing model nodes must use the same model coordination namespace, database, and key prefix.

  • Use a distinct revocable ACL user and password file for each API deployment and monitor. The password file must contain exactly one non-empty secret; the loader URL-encodes it and rejects ambiguous credentials embedded in the URL.

  • Application coordination must retain a deployment-specific namespace and key prefix. Do not point two environments at the same application-coordination keyspace merely because they share GPUs.

  • When model scheduling or residency monitoring is enabled, this dedicated block is mandatory. Startup fails if it is absent; application coordination is never inherited as an implicit model-capacity pool.

  • Migrate every API and monitor that shares the model nodes as one rollout. Start at least one monitor against the new pool first, wait for fresh healthy snapshots, then restart the APIs against exactly the same URL, database, namespace, and prefix. Two live pools create split-brain admission even when they point to the same Valkey server.

  • An SSH-forwarded URL is suitable for development participation, but the always-on deployment and standalone residency monitor must reach the Valkey service without depending on a developer workstation.

Best-effort application cache

Natural-language typed routing, document discovery, and query embeddings use a separate best-effort cache policy. Keep this cache in the owning deployment’s application Valkey, with a separate logical database and key prefix from application coordination. Do not place it in the dedicated shared model-coordination Valkey: cache eviction, cache failure, and environment data must never affect or cross scheduler leases.

AIBS_APPLICATION_CACHE_BACKEND=valkey
AIBS_APPLICATION_CACHE_NAMESPACE=aibs-application-cache
AIBS_APPLICATION_CACHE_DEFAULT_TTL_SECONDS=300
AIBS_APPLICATION_CACHE_VALKEY_URL=redis://localhost:6379/1
AIBS_APPLICATION_CACHE_VALKEY_KEY_PREFIX=aibs:application-cache
AIBS_APPLICATION_CACHE_VALKEY_SOCKET_TIMEOUT_SECONDS=1

  • The default Valkey URL uses logical database 1; coordination defaults to database 0.

  • AIBS_APPLICATION_CACHE_BACKEND=memory is process-local and is suitable for tests or a single local API process.

  • Cache reads, writes, and single-flight lease failures degrade to cache misses. Authoritative catalog, conversation, result-set, and scheduler state never depends on this cache.

  • Conversation history and conversation memory are durable environment-local state, not cache entries. A cache flush may increase latency but cannot erase or alter conversational scope.

  • Route keys include the normalized question, public-mode/source context, bounded recent turns, authoritative conversation routing state, and router version. Reusing a cached classification across different conversational contexts is therefore forbidden by construction.

  • Legacy AIBS_APPLICATION_CACHE_REDIS_* names remain readable during migration, but canonical AIBS_APPLICATION_CACHE_VALKEY_* values take precedence.

Model admission and residency planning

Map-reduce answering can create many role-level model calls from one user question. The model admission boundary is the first control point for that work: each document-map and reduce call asks for a lease before calling the LLM, and whole-query agent-flow admission gates broad map fleets before workers spawn. This prevents one broad query from blindly spawning a full model fleet while another broad query is already running.

AIBS_MODEL_ADMISSION_ENABLED=true
AIBS_MODEL_ADMISSION_SAFE_POOL_GB=0
AIBS_MODEL_ADMISSION_QUEUE_TIMEOUT_SECONDS=0.25
AIBS_MODEL_ADMISSION_KV_GB_PER_ACTIVE_BILLION_PER_TOKEN=0.000008

# Whole-query agent flow admission. This is above per-model admission and
# controls whether broad agentic map fleets start immediately, wait with an ETA,
# or fail fast when the queue is full.
AIBS_AGENT_FLOW_SCHEDULER_ENABLED=true
AIBS_AGENT_FLOW_MAX_CONCURRENT=0
AIBS_AGENT_FLOW_MAX_QUEUED=8
AIBS_AGENT_FLOW_QUEUE_TIMEOUT_SECONDS=300
AIBS_AGENT_FLOW_MODEL_QUEUE_TIMEOUT_SECONDS=30
AIBS_PLANNER_MODEL_QUEUE_TIMEOUT_SECONDS=30
AIBS_AGENT_FLOW_ETA_SECONDS_PER_DOCUMENT=30
AIBS_AGENT_FLOW_ETA_DISPLAY_BUFFER_RATIO=0.15
AIBS_AGENT_FLOW_ACTIVE_LEASE_TTL_SECONDS=60
AIBS_AGENT_FLOW_STATE_LOCK_TTL_SECONDS=5
AIBS_AGENT_FLOW_POLL_INTERVAL_SECONDS=0.25
AIBS_AGENT_FLOW_ACTIVE_STATE_REAP_ENABLED=true
AIBS_AGENT_FLOW_ACTIVE_STATE_REAP_AFTER_SECONDS=3600
AIBS_AGENT_FLOW_RELEASE_RETRY_TIMEOUT_SECONDS=30

# Active model worker roster. If omitted, the backend derives base, summarizer,
# semantic-profile, and optional heavy workers from the AIBS_LLM_* lane settings.
AIBS_MODEL_ADMISSION_WORKER_ROSTER=base,summarizer,semantic-profile,map-small,reasoning-small

# Built-in worker names inherit their model/runtime envelope from the matching
# LLM lane. Keep roles and measured capacity explicit here.
AIBS_WORKER_BASE_ROLES=planner,workspace_answer,document_map,structured_extract,reduce,final_answer
AIBS_WORKER_BASE_RESIDENT_WEIGHT_GB=16.02
AIBS_WORKER_BASE_ACTIVE_PARAM_BILLIONS=26

AIBS_WORKER_SUMMARIZER_ROLES=document_map,structured_extract
AIBS_WORKER_SUMMARIZER_RESIDENT_WEIGHT_GB=11.86
AIBS_WORKER_SUMMARIZER_ACTIVE_PARAM_BILLIONS=3.6

AIBS_WORKER_SEMANTIC_PROFILE_ROLES=semantic_profile
AIBS_WORKER_SEMANTIC_PROFILE_RESIDENT_WEIGHT_GB=4.91
AIBS_WORKER_SEMANTIC_PROFILE_ACTIVE_PARAM_BILLIONS=8

# Document-map subagent workers. These definitions are active when the worker
# name is listed in AIBS_MODEL_ADMISSION_WORKER_ROSTER.
AIBS_WORKER_MAP_SMALL_MODEL_ID=qwen2.5:7b-instruct
AIBS_WORKER_MAP_SMALL_ROLES=document_map,structured_extract
AIBS_WORKER_MAP_SMALL_CONTEXT_LENGTH=16384
AIBS_WORKER_MAP_SMALL_MAX_OUTPUT_TOKENS=2048
AIBS_WORKER_MAP_SMALL_MAX_CONCURRENT_REQUESTS=1
AIBS_WORKER_MAP_SMALL_RESIDENT_WEIGHT_GB=4.31
AIBS_WORKER_MAP_SMALL_ACTIVE_PARAM_BILLIONS=7
AIBS_WORKER_MAP_SMALL_KEEP_ALIVE=5m
AIBS_WORKER_MAP_SMALL_TIMEOUT_SECONDS=240

AIBS_WORKER_REASONING_SMALL_MODEL_ID=qwen3:8b
AIBS_WORKER_REASONING_SMALL_ROLES=document_map,structured_extract,reduce
AIBS_WORKER_REASONING_SMALL_CONTEXT_LENGTH=32768
AIBS_WORKER_REASONING_SMALL_MAX_OUTPUT_TOKENS=4096
AIBS_WORKER_REASONING_SMALL_MAX_CONCURRENT_REQUESTS=1
AIBS_WORKER_REASONING_SMALL_RESIDENT_WEIGHT_GB=4.91
AIBS_WORKER_REASONING_SMALL_ACTIVE_PARAM_BILLIONS=8
AIBS_WORKER_REASONING_SMALL_KEEP_ALIVE=5m
AIBS_WORKER_REASONING_SMALL_TIMEOUT_SECONDS=300

AIBS_DOCUMENT_COMPLEXITY_SMALL_MODEL_IDS=qwen2.5:7b-instruct,qwen3:8b,gpt-oss:20b
AIBS_DOCUMENT_COMPLEXITY_MEDIUM_MODEL_IDS=qwen3:8b,qwen2.5:7b-instruct,gpt-oss:20b
AIBS_DOCUMENT_COMPLEXITY_LARGE_MODEL_IDS=gpt-oss:20b,qwen3:8b,gemma4:26b
AIBS_DOCUMENT_COMPLEXITY_DEGRADED_LARGE_MODEL_IDS=gpt-oss:20b,gemma4:26b,qwen3:8b

# Optional legacy/default-roster sizing. Keep at 0 until measured on the target
# Ollama host. These values feed the fallback base/summarizer/heavy roster only
# when AIBS_MODEL_ADMISSION_WORKER_ROSTER is not set.
AIBS_MODEL_ADMISSION_BASE_RESIDENT_WEIGHT_GB=0
AIBS_MODEL_ADMISSION_BASE_ACTIVE_PARAMS_BILLIONS=0
AIBS_MODEL_ADMISSION_SUMMARIZER_RESIDENT_WEIGHT_GB=0
AIBS_MODEL_ADMISSION_SUMMARIZER_ACTIVE_PARAMS_BILLIONS=0
AIBS_MODEL_ADMISSION_HEAVY_RESIDENT_WEIGHT_GB=0
AIBS_MODEL_ADMISSION_HEAVY_ACTIVE_PARAMS_BILLIONS=0

  • AIBS_MODEL_ADMISSION_ENABLED=true enables the application-level admission boundary. If the planner itself fails, answering continues through a no-op fallback so correctness does not depend on the scheduler.

  • AIBS_MODEL_ADMISSION_SAFE_POOL_GB=0 disables memory-budget admission while preserving per-model concurrency admission. Set a positive value only after measuring the Ollama host’s safe resident-model pool.

  • AIBS_MODEL_ADMISSION_QUEUE_TIMEOUT_SECONDS controls how long a role waits for its preferred model lane before trying a smaller eligible lane. If no eligible lane is available, the request is treated as busy instead of hanging indefinitely.

  • AIBS_AGENT_FLOW_SCHEDULER_ENABLED=true enables whole-query admission for agentic map/reduce work. This is the user-visible queue/ETA layer; per-model admission still controls each individual LLM call after the flow starts.

  • AIBS_AGENT_FLOW_MAX_CONCURRENT=0 derives the active-flow ceiling from the enabled model-node count. A positive value sets an explicit ceiling. The scheduler reduces per-flow map parallelism as active flows increase; atomic model placement still applies live GPU pressure, memory headroom, per-model, per-node, and per-flow limits before every model call.

  • When AIBS_MODEL_ADMISSION_SAFE_POOL_GB is positive, agent-flow admission clamps broad-query map parallelism to the largest value that fits the estimated pool. If even sequential map work cannot fit, the backend returns an explicit capacity response instead of overcommitting the model host.

  • AIBS_AGENT_FLOW_MAX_QUEUED bounds accepted-but-waiting agent flows. When full, the backend returns a busy answer instead of silently hanging.

  • AIBS_AGENT_FLOW_QUEUE_TIMEOUT_SECONDS is the maximum time a queued agentic flow may wait on the open SSE stream before it degrades to busy.

  • AIBS_AGENT_FLOW_MODEL_QUEUE_TIMEOUT_SECONDS is the bounded wait for each map/refine/compose model call after the flow starts. The preferred lane begins degrading after AIBS_MODEL_ADMISSION_QUEUE_TIMEOUT_SECONDS; the longer flow-scoped wait lets live placement pressure serialize work instead of misclassifying temporary saturation as a failed document.

  • AIBS_PLANNER_MODEL_QUEUE_TIMEOUT_SECONDS is the bounded wait for an intent or scope-planning call to obtain scheduler-selected model capacity. Planning uses the same global node placement and lane envelopes as scheduled document work, so a busy or stalled direct LLM host cannot serialize every chat request.

  • AIBS_AGENT_FLOW_ETA_SECONDS_PER_DOCUMENT seeds ETA calculation before the process has observed real completed agent-flow durations. Runtime observations replace this seed for later ETA estimates.

  • AIBS_AGENT_FLOW_ETA_DISPLAY_BUFFER_RATIO pads the user-visible queued ETA without changing scheduling decisions. The default 0.15 shows about 15% more time than the raw estimate so users are more likely to receive an answer before the displayed time expires.

  • AIBS_AGENT_FLOW_ACTIVE_LEASE_TTL_SECONDS, AIBS_AGENT_FLOW_STATE_LOCK_TTL_SECONDS, and AIBS_AGENT_FLOW_POLL_INTERVAL_SECONDS control the coordinated scheduler’s active-flow heartbeat lease, fenced state-lock lease, and queue polling cadence. Keep these conservative unless a saturation run proves the lock/heartbeat windows are too slow.

  • AIBS_AGENT_FLOW_ACTIVE_STATE_REAP_ENABLED=true enables a generous dead-man cleanup for leaked active-flow records. Reap is driven by AIBS_AGENT_FLOW_ACTIVE_STATE_REAP_AFTER_SECONDS, not the short heartbeat lease. The default one-hour ceiling is intentionally much longer than normal broad-query work: a missed heartbeat should not over-admit another map fleet, but a crashed or leaked active record should eventually self-heal.

  • AIBS_AGENT_FLOW_RELEASE_RETRY_TIMEOUT_SECONDS controls how long an authoritative flow release keeps trying to acquire the scheduler state lock before logging a leaked slot. Keep this longer than AIBS_AGENT_FLOW_STATE_LOCK_TTL_SECONDS.

  • For multi-process Valkey, active-flow dead-man liveness must be based on server-side TTL/key expiry rather than API-worker epoch-clock comparisons.

  • AIBS_MODEL_ADMISSION_WORKER_ROSTER is an ordered list of worker names. Each worker is configured through AIBS_WORKER_<NAME>_*, where <NAME> is the upper-case roster name with non-alphanumeric characters converted to underscores. For example, map-small becomes AIBS_WORKER_MAP_SMALL_*.

  • Built-in worker names inherit from the corresponding LLM lane unless an AIBS_WORKER_<NAME>_* override is present:

    • base inherits model, context length, output budget, keep-alive, timeout, and concurrency from AIBS_LLM_*.

    • summarizer inherits from AIBS_LLM_SUMMARIZER_*, then falls back to AIBS_LLM_*.

    • semantic-profile inherits from AIBS_LLM_SEMANTIC_PROFILE_*. Its default role is semantic_profile and its default model is qwen3:8b.

    • heavy inherits from AIBS_LLM_HEAVY_*, then falls back to AIBS_LLM_*, but the default heavy worker is created only when AIBS_LLM_HEAVY_MODEL names a distinct model.

  • Custom worker names, such as map-small, do not have an inherited LLM lane. They must provide at least AIBS_WORKER_<NAME>_MODEL_ID and AIBS_WORKER_<NAME>_ROLES.

  • Custom workers become callable only when listed in AIBS_MODEL_ADMISSION_WORKER_ROSTER. In the current demo roster, qwen2.5:7b-instruct and qwen3:8b can be active document-map subagents alongside gpt-oss:20b; the built-in semantic-profile qwen lane is reserved for ingestion-time semantic profile enrichment.

  • AIBS_DOCUMENT_COMPLEXITY_*_MODEL_IDS controls which active map-agent models are eligible for each document class. This is deterministic structural routing: small/medium documents can use lighter qwen agents, while large or degraded-large documents can prefer gpt-oss:20b before falling back to Gemma. Admission still picks the concrete available model under concurrency and residency constraints.

  • AIBS_WORKER_<NAME>_ROLES controls which answer roles the worker can serve. Supported roles are planner, document_map, structured_extract, reduce, and final_answer. The complexity policy sets eligibility; the admission planner still picks the concrete worker under the memory/concurrency budget.

  • AIBS_WORKER_<NAME>_CONTEXT_LENGTH, MAX_OUTPUT_TOKENS, KEEP_ALIVE, and TIMEOUT_SECONDS are optional overrides for built-in workers and required runtime-envelope declarations for custom workers.

  • AIBS_WORKER_<NAME>_MAX_CONCURRENT_REQUESTS, RESIDENT_WEIGHT_GB, and ACTIVE_PARAM_BILLIONS are used by admission to serialize model calls and estimate safe residency. For MoE models, active parameters should be the active parameter count, not total parameters.

  • *_RESIDENT_WEIGHT_GB should reflect the measured loaded model weight for the lane. *_ACTIVE_PARAMS_BILLIONS is used for context/KV estimates and should be the active parameter count for MoE models, not the total parameter count. For example, a 20B MoE model with about 3.6B active parameters should use 3.6 here for KV budgeting.

  • With AIBS_MODEL_CLUSTER_ENABLED=true, the current local .env uses the Valkey global placement adapter and routes each leased model call to the selected node. Set it to false only for a single-process local deployment where model residency admission should remain a process-local guard.

  • Keep the global model cluster enabled only when all of the following are true: worker resident weights are measured, node VRAM budgets are measured, AIBS_MODEL_COORDINATION_BACKEND=valkey, and a live node-residency monitor is writing fresh health observations. Without fresh residency, the placement primitive fails closed as node_stale.

  • AIBS_MODEL_CLUSTER_ACTIVE_LEASE_TTL_SECONDS controls the lifetime of an active model-placement lease. Keep it longer than the longest expected LLM call timeout.

  • AIBS_MODEL_CLUSTER_HEARTBEAT_INTERVAL_SECONDS controls renewal cadence for active placement leases. It must be shorter than AIBS_MODEL_CLUSTER_ACTIVE_LEASE_TTL_SECONDS; transient renewal failures are retried with bounded backoff.

  • AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS is the maximum age of a node residency health observation accepted by placement. If the monitor stops, the scheduler should fail closed instead of placing work onto unknown node state.

  • AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_ENABLED enables the elected monitor that polls configured Ollama nodes and writes two records: a tiny placement health stamp and a separate telemetry snapshot. Keep AIBS_MODEL_CLUSTER_ENABLED=true only while audit and live tests prove those stamps stay fresh.

  • AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_RUN_IN_API=false delegates that monitor to the standalone python -m aibs_backend.interfaces.cli.model_residency_monitor process. This is the recommended multi-node deployment: answer-model latency or API event loop pressure cannot delay the control-plane election heartbeat. Keep the value true only for a small single-process development environment.

  • API startup waits for at least one fresh healthy residency snapshot whenever global model scheduling is enabled. Stale or absent control-plane state keeps the API unready instead of accepting chats that are guaranteed to fail with node_stale.

  • AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_POLL_INTERVAL_SECONDS must be no more than one third of AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS. AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_POLL_TIMEOUT_SECONDS must be shorter than the poll interval.

  • AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_FAILURE_THRESHOLD is the number of consecutive failed polls before a node is marked down. A single success restores the node to healthy.

  • AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_LEASE_TTL_SECONDS bounds monitor failover and must be longer than one poll interval plus one poll timeout. This ownership TTL may exceed the placement staleness window: placement fails closed as soon as telemetry exceeds AIBS_MODEL_CLUSTER_MAX_RESIDENCY_STALENESS_MS, even while a paused monitor still owns its election lease. This prevents transient host scheduling pauses from creating false leader flaps without trusting stale placement state.

  • AIBS_MODEL_CLUSTER_RESIDENCY_MONITOR_STANDBY_RETRY_SECONDS controls how often standby API processes retry monitor election after another process owns the monitor lease.

  • AIBS_MODEL_CLUSTER_TELEMETRY_TTL_SECONDS controls retention for the richer per-node telemetry snapshot used by audit/health surfaces. It is separate from the placement hash to keep admission reads lean.

  • AIBS_MODEL_CLUSTER_HEALTH_HISTORY_WINDOW_SECONDS, AIBS_MODEL_CLUSTER_HEALTH_HISTORY_MAX_SAMPLES, and AIBS_MODEL_CLUSTER_HEALTH_HISTORY_TTL_SECONDS control the rolling samples used by the System Health page. Model-node history is stored in the shared model-coordination pool. Ingestion, answer, and runtime workload history is sampled independently by each API and stored in that environment’s application-coordination pool. Keep TTL greater than or equal to the displayed history window.

  • Before setting AIBS_MODEL_CLUSTER_ENABLED=true, run python -m aibs_backend.interfaces.cli.audit_model_capacity --json and python -m aibs_backend.interfaces.cli.prove_model_monitor_takeover --json --repeat 3. After enabling, restart the API and run the audit again; the expected state is ready_for_global_scheduler=true.

  • When the standalone monitor is selected, start it before the API and keep it under the same service supervisor as the API. Run at least two supervised monitor candidates on independent always-on hosts; Valkey election keeps one active while the other remains ready to take over. See docs/maintainance_&_operations/model_residency_monitor.md.

Common Setup Checks

Before starting the API, it helps to confirm:

  • OpenSearch is reachable

  • the embedding model path exists

  • the reranker model path exists if reranking is enabled

  • AIBS_DOCLING_ARTIFACTS_PATH exists if local Docling artifacts are configured

  • device values are auto unless this machine is known to have a CUDA/ROCm-enabled torch runtime

  • the LLM endpoint is reachable if AIBS_LLM_ENABLED=true

Then start the backend with:

python -m pdm run uvicorn aibs_backend.interfaces.api.app:app --host 127.0.0.1 --port 5000 --reload

Open:

http://127.0.0.1:5000/docs