# Architecture Overview ## Purpose The backend currently implements the document-ingestion and conversation answering lifecycle end to end for AI Business Search: - accept a document upload - store, register, and queue it for processing - parse it into canonical document artifacts - chunk it into retrieval-oriented records - embed those chunks with a local model - index them into OpenSearch - route a workspace-scoped question through Auto, Creative, or Grounded evidence regimes - retrieve, rerank, and bind enabled source evidence when the selected regime needs it - generate a cited answer through a local Ollama-compatible LLM - persist query and answer lineage for conversation history reload - expose the flow through FastAPI RBAC is implemented as an AIBS-owned boundary. Login can reuse an OpenSearch/Logserver identity source, but product access requires `aibs-access` and workspace access is enforced through local workspace roles. The current OpenSearch adapter accepts either standard Security authinfo payloads or the Logserver `/_logserver/login` response shape. ## Layered Structure ### Domain Lives under `src/aibs_backend/domain/`. Responsibilities: - `Document` aggregate lifecycle and invariants - `ParsedDocument` and `ChunkSet` as attached artifacts - lifecycle transition rules - value objects like `Checksum`, `MediaType`, `ChunkPolicyVersion`, and `DocumentLanguage` (ISO 639-1 primary code, confidence, detected-at timestamp, classifier identity) - the shared chunk boundary (`DocumentChunkLike`) The domain does not parse files, call models, or talk to OpenSearch. ### Application Lives under `src/aibs_backend/application/`. Responsibilities: - register-and-submit document service - upload registration - parsing orchestration - chunking orchestration - embedding/indexing orchestration - full pipeline advancement from the current stage - workspace and conversation services - conversation planning, evidence-regime routing, retrieval, and reranking orchestration - Creative and Grounded answer generation with citation persistence - adaptive chat dispatch across workspace-awareness, evidence binding, and workload execution lanes - conversation history reload from persisted query/answer records This layer depends on ports and repositories, not on HTTP or framework code. ### Infrastructure Lives under `src/aibs_backend/infrastructure/`. Responsibilities: - local raw file storage - JSON-backed document, workspace, conversation, query, and answer persistence - Docling-based parsing - chunk building - sentence-transformers embedding - OpenSearch chunk indexing and vector search - OpenSearch document catalog projection for large-workspace source lists - OpenSearch document profile and section/table summary projections for cheap discovery before chunk retrieval - cross-encoder reranking - Ollama-compatible LLM adapter with a concurrency gate - Valkey/Redis-backed shared coordination for global model placement, expiring leases, model-residency state, and cross-process scheduler state This is where external runtime coupling lives. ### Interfaces Lives under `src/aibs_backend/interfaces/`. Responsibilities: - FastAPI application shell (`interfaces/api/`) - dependency wiring entry points - request/response schemas - auth, workspace, conversation, document, and chat routes - AIBS product-user administration routes - startup readiness checks and in-process worker task management - standalone ingestion worker CLI (`interfaces/cli/ingestion_worker.py`) ## Composition Root `src/aibs_backend/bootstrap.py` is the composition root for the current backend. It wires together: - local file store - JSON repositories for documents, workspaces, conversations, query executions, and answer artifacts - Docling parser - document normalizer - chunk builder - embedding service - OpenSearch chunk repository - OpenSearch document catalog index wrapped around the JSON document repository - OpenSearch retrieval pyramid index wrapped around the JSON document repository - OpenSearch ingestion job repository - optional cross-encoder reranker - optional Ollama-backed answer assistant wrapped in a concurrency-limited decorator (single in-flight request by default) - optional assistant-backed document query expansion service for Smart search - upload / parse / chunk / index services - submit ingestion job service - ingestion job runner (claim-and-run loop, heartbeat, lease reaping) - workspace, conversation, Creative/Grounded answer, shared workload, and adaptive dispatch services - auth/session services and AIBS user-directory services ## Runtime Storage Split ### Local filesystem - uploads: `.aibs_data/uploads` - staged browser uploads: `.aibs_data/staging` - JSON document records: `.aibs_data/documents` ### Models - embeddings: `models/embeddings/bge-m3` - reranking: `models/reranking/bge-reranker-v2-m3` (used when reranking is enabled) ### OpenSearch - chunk index template: `.abs-content-{workspace_id}` - workspace-scoped vector index per workspace - chunk embeddings stored as `knn_vector` - ingestion job index: `aibs-ingestion-jobs` (durable job queue with OCC-based claiming) - document catalog index: `aibs-document-catalog` by default, one lightweight projection record per document for catalog paging, filtering, summaries, and search-result hydration - document profile index: `aibs-document-profiles` by default, one profile per document for broad discovery - section/table summary index: `aibs-section-summaries` by default, one row per structural summary for broad discovery - retrieval pyramid discovery is read only when `AIBS_RETRIEVAL_PYRAMID_DISCOVERY_ENABLED=true` - configured identity store for login and AIBS user-directory administration ### Valkey/Redis - optional shared coordination plane for multi-process or multi-box deployments - required when `AIBS_MODEL_CLUSTER_ENABLED=true` - stores volatile coordination state such as model placement leases, node residency samples, scheduler queue state, and best-effort cache values - does not store document source truth, retrieval indexes, or answer lineage - architecture details: `docs/architecture/model-coordination.md` ### Auth And RBAC - OpenSearch/Logserver credentials authenticate the principal - `aibs-access` is required to enter AIBS - `aibs-admin` plus `aibs-access` is required for product user administration - workspace membership remains separate from product administration - owners/admins manage workspace members, but only directory-backed users can be added by username - the first production admin is bootstrapped with `python -m pdm run python -m aibs_backend.interfaces.cli.bootstrap_admin --username ` ### LLM - optional Ollama-compatible endpoint for conversation answer generation - model and base URL are environment-driven - access is serialized through a concurrency gate to match single-lane local serving shapes - when the global scheduler is enabled, model calls are placed through the Valkey-backed scheduler and routed to the selected Ollama node ## Request Flow ```mermaid flowchart TD Browser --> UploadRoute["POST /api/aibs/documents"] UploadRoute --> StageFile["Stage file under .aibs_data/staging"] StageFile --> UploadService["RegisterAndSubmitDocumentService"] UploadService --> FileStore["LocalFileStore"] UploadService --> DocRepo["JsonDocumentRepository"] UploadService --> SubmitJob["SubmitIngestionJobService"] Browser --> ProcessRoute["POST /api/aibs/documents/{id}/process"] ProcessRoute --> SubmitJob["SubmitIngestionJobService"] SubmitJob --> JobRepo["OpenSearchIngestionJobRepository"] JobRepo --> OpenSearch WorkerLoop["IngestionWorkerLoop (API task or standalone CLI)"] --> JobRunner["IngestionJobRunner.claim_and_run_once"] JobRunner --> JobRepo JobRunner --> ParseService["ParseDocumentService"] JobRunner --> ChunkService["ChunkDocumentService"] JobRunner --> IndexService["IndexDocumentService"] ParseService --> Parser["DoclingParser + DocumentNormalizer"] ChunkService --> ChunkBuilder["ChunkingBuilder"] IndexService --> Embedder["EmbeddingService"] IndexService --> ChunkRepo["OpenSearchChunkRepository"] ChunkRepo --> OpenSearch["OpenSearch"] Browser --> CatalogRoute["GET /api/aibs/documents/catalog/search"] CatalogRoute --> CatalogRepo["CatalogIndexedDocumentRepository"] CatalogRepo --> CatalogIndex["OpenSearch document catalog"] CatalogRepo --> DocRepo CatalogIndex --> OpenSearch Browser --> ChatRoute["POST /api/aibs/chat/"] ChatRoute --> Dispatcher["DispatchChatService"] Dispatcher --> DirectWorkspace["Direct workspace answer check"] DirectWorkspace --> WorkspaceAwareness["WorkspaceAwarenessService"] Dispatcher --> Planner["Planner classifies intent"] Planner --> Regime{"Auto / Creative / Grounded"} Regime --> Creative["Creative evidence regime"] Regime --> GroundedAnswer["Grounded evidence regime"] Creative --> EvidenceBinding["Enabled-source evidence binding"] GroundedAnswer --> EvidenceBinding EvidenceBinding --> Retrieval["Retrieval + Reranking when evidence is needed"] Retrieval --> ChunkRepo EvidenceBinding --> Workload["Workload planner"] Workload --> ChatWork["chat"] Workload --> MapReduce["chat_map_reduce"] Workload --> AgentWork["agent_map_refine_compose"] ChatWork --> LLM["Ollama assistant (concurrency-gated)"] MapReduce --> LLM AgentWork --> LLM LLM --> Contracts["Language + citation + table + support contracts"] Contracts --> QueryRepo["Query + Answer JSON repos"] ``` ## Startup Behavior The FastAPI lifespan hook currently does these checks before serving: - OpenSearch connectivity (always) - document catalog index readiness (always) - retrieval pyramid index readiness (always) - embedding model readiness (always) - reranker readiness (only when `AIBS_RERANKER_ENABLED=true`) - document parser warmup (only when `AIBS_INGESTION_WORKER_ENABLED=true`) - LLM readiness (only when `AIBS_LLM_ENABLED=true` and `AIBS_LLM_STARTUP_CHECK=true`) If any enabled check fails, startup fails. When `AIBS_INGESTION_WORKER_ENABLED=true` (the default), the lifespan also starts an in-process `IngestionWorkerLoop` background task. When set to `false`, the API only submits jobs — a separate process running `python -m aibs_backend.interfaces.cli.ingestion_worker` claims and executes them. That standalone worker process runs its own identical startup checks (excluding LLM unless abstractive summaries are enabled), including the document catalog index because ingestion writes document-state changes through that projection. That is intentional. Advertising healthy startup while a declared capability is unreachable would be misleading. ## Current Boundaries That Matter - Workspace ownership is enforced at the API boundary for document-specific endpoints. - Upload workflow is backend-owned on the happy path: `POST /api/aibs/documents` submits the ingestion job, while `/api/aibs/documents/{id}/process` is for manual retry/recovery only. - Ingestion is now driven by a durable OpenSearch-backed job queue with OCC-based claiming, lease heartbeats, retry scheduling, and stale lease reaping. Multiple worker processes can safely compete for jobs without double-execution. - The worker can run in-process within the API (default) or as a standalone CLI process on a separate machine with access to the same OpenSearch and shared storage roots. - Workspace source-list reads use an OpenSearch catalog projection, while JSON document files remain the aggregate source of truth. Backfill the projection with `python -m aibs_backend.interfaces.cli.rebuild_document_catalog` when enabling it for existing workspaces; run a full rebuild without `--workspace` for catalog mapping schema upgrades. - Document metadata persistence (workspace, conversation, document records) is local JSON, which is good for current wiring and local validation but not yet the final production persistence story.