Architecture Overview

Purpose

The backend currently implements the document-ingestion and conversation answering lifecycle end to end for AI Business Search:

• accept a document upload

• store, register, and queue it for processing

• parse it into canonical document artifacts

• chunk it into retrieval-oriented records

• embed those chunks with a local model

• index them into OpenSearch

• route a workspace-scoped question through Auto, Creative, or Grounded evidence regimes

• retrieve, rerank, and bind enabled source evidence when the selected regime needs it

• generate a cited answer through a local Ollama-compatible LLM

• persist query and answer lineage for conversation history reload

• expose the flow through FastAPI

RBAC is implemented as an AIBS-owned boundary. Login can reuse an OpenSearch/Logserver identity source, but product access requires aibs-access and workspace access is enforced through local workspace roles. The current OpenSearch adapter accepts either standard Security authinfo payloads or the Logserver /_logserver/login response shape.

Layered Structure

Domain

Lives under src/aibs_backend/domain/.

Responsibilities:

• Document aggregate lifecycle and invariants

• ParsedDocument and ChunkSet as attached artifacts

• lifecycle transition rules

• value objects like Checksum, MediaType, ChunkPolicyVersion, and DocumentLanguage (ISO 639-1 primary code, confidence, detected-at timestamp, classifier identity)

• the shared chunk boundary (DocumentChunkLike)

The domain does not parse files, call models, or talk to OpenSearch.

Application

Lives under src/aibs_backend/application/.

Responsibilities:

• register-and-submit document service

• upload registration

• parsing orchestration

• chunking orchestration

• embedding/indexing orchestration

• full pipeline advancement from the current stage

• workspace and conversation services

• conversation planning, evidence-regime routing, retrieval, and reranking orchestration

• Creative and Grounded answer generation with citation persistence

• adaptive chat dispatch across workspace-awareness, evidence binding, and workload execution lanes

• conversation history reload from persisted query/answer records

This layer depends on ports and repositories, not on HTTP or framework code.

Infrastructure

Lives under src/aibs_backend/infrastructure/.

Responsibilities:

• local raw file storage

• JSON-backed document, workspace, conversation, query, and answer persistence

• Docling-based parsing

• chunk building

• sentence-transformers embedding

• OpenSearch chunk indexing and vector search

• OpenSearch document catalog projection for large-workspace source lists

• OpenSearch document profile and section/table summary projections for cheap discovery before chunk retrieval

• cross-encoder reranking

• Ollama-compatible LLM adapter with a concurrency gate

• Valkey/Redis-backed shared coordination for global model placement, expiring leases, model-residency state, and cross-process scheduler state

This is where external runtime coupling lives.

Interfaces

Lives under src/aibs_backend/interfaces/.

Responsibilities:

• FastAPI application shell (interfaces/api/)

• dependency wiring entry points

• request/response schemas

• auth, workspace, conversation, document, and chat routes

• AIBS product-user administration routes

• startup readiness checks and in-process worker task management

• standalone ingestion worker CLI (interfaces/cli/ingestion_worker.py)

Composition Root

src/aibs_backend/bootstrap.py is the composition root for the current backend.

It wires together:

• local file store

• JSON repositories for documents, workspaces, conversations, query executions, and answer artifacts

• Docling parser

• document normalizer

• chunk builder

• embedding service

• OpenSearch chunk repository

• OpenSearch document catalog index wrapped around the JSON document repository

• OpenSearch retrieval pyramid index wrapped around the JSON document repository

• OpenSearch ingestion job repository

• optional cross-encoder reranker

• optional Ollama-backed answer assistant wrapped in a concurrency-limited decorator (single in-flight request by default)

• optional assistant-backed document query expansion service for Smart search

• upload / parse / chunk / index services

• submit ingestion job service

• ingestion job runner (claim-and-run loop, heartbeat, lease reaping)

• workspace, conversation, Creative/Grounded answer, shared workload, and adaptive dispatch services

• auth/session services and AIBS user-directory services

Runtime Storage Split

Local filesystem

• uploads: .aibs_data/uploads

• staged browser uploads: .aibs_data/staging

• JSON document records: .aibs_data/documents

Models

• embeddings: models/embeddings/bge-m3

• reranking: models/reranking/bge-reranker-v2-m3 (used when reranking is enabled)

OpenSearch

• chunk index template: .abs-content-{workspace_id}

• workspace-scoped vector index per workspace

• chunk embeddings stored as knn_vector

• ingestion job index: aibs-ingestion-jobs (durable job queue with OCC-based claiming)

• document catalog index: aibs-document-catalog by default, one lightweight projection record per document for catalog paging, filtering, summaries, and search-result hydration

• document profile index: aibs-document-profiles by default, one profile per document for broad discovery

• section/table summary index: aibs-section-summaries by default, one row per structural summary for broad discovery

• retrieval pyramid discovery is read only when AIBS_RETRIEVAL_PYRAMID_DISCOVERY_ENABLED=true

• configured identity store for login and AIBS user-directory administration

Valkey/Redis

• optional shared coordination plane for multi-process or multi-box deployments

• required when AIBS_MODEL_CLUSTER_ENABLED=true

• stores volatile coordination state such as model placement leases, node residency samples, scheduler queue state, and best-effort cache values

• does not store document source truth, retrieval indexes, or answer lineage

• architecture details: docs/architecture/model-coordination.md

Auth And RBAC

• OpenSearch/Logserver credentials authenticate the principal

• aibs-access is required to enter AIBS

• aibs-admin plus aibs-access is required for product user administration

• workspace membership remains separate from product administration

• owners/admins manage workspace members, but only directory-backed users can be added by username

• the first production admin is bootstrapped with python -m pdm run python -m aibs_backend.interfaces.cli.bootstrap_admin --username <admin-user>

LLM

• optional Ollama-compatible endpoint for conversation answer generation

• model and base URL are environment-driven

• access is serialized through a concurrency gate to match single-lane local serving shapes

• when the global scheduler is enabled, model calls are placed through the Valkey-backed scheduler and routed to the selected Ollama node

Request Flow

        flowchart TD
    Browser --> UploadRoute["POST /api/aibs/documents"]
    UploadRoute --> StageFile["Stage file under .aibs_data/staging"]
    StageFile --> UploadService["RegisterAndSubmitDocumentService"]
    UploadService --> FileStore["LocalFileStore"]
    UploadService --> DocRepo["JsonDocumentRepository"]
    UploadService --> SubmitJob["SubmitIngestionJobService"]

    Browser --> ProcessRoute["POST /api/aibs/documents/{id}/process"]
    ProcessRoute --> SubmitJob["SubmitIngestionJobService"]
    SubmitJob --> JobRepo["OpenSearchIngestionJobRepository"]
    JobRepo --> OpenSearch

    WorkerLoop["IngestionWorkerLoop (API task or standalone CLI)"] --> JobRunner["IngestionJobRunner.claim_and_run_once"]
    JobRunner --> JobRepo
    JobRunner --> ParseService["ParseDocumentService"]
    JobRunner --> ChunkService["ChunkDocumentService"]
    JobRunner --> IndexService["IndexDocumentService"]

    ParseService --> Parser["DoclingParser + DocumentNormalizer"]
    ChunkService --> ChunkBuilder["ChunkingBuilder"]
    IndexService --> Embedder["EmbeddingService"]
    IndexService --> ChunkRepo["OpenSearchChunkRepository"]
    ChunkRepo --> OpenSearch["OpenSearch"]

    Browser --> CatalogRoute["GET /api/aibs/documents/catalog/search"]
    CatalogRoute --> CatalogRepo["CatalogIndexedDocumentRepository"]
    CatalogRepo --> CatalogIndex["OpenSearch document catalog"]
    CatalogRepo --> DocRepo
    CatalogIndex --> OpenSearch

    Browser --> ChatRoute["POST /api/aibs/chat/"]
    ChatRoute --> Dispatcher["DispatchChatService"]
    Dispatcher --> DirectWorkspace["Direct workspace answer check"]
    DirectWorkspace --> WorkspaceAwareness["WorkspaceAwarenessService"]
    Dispatcher --> Planner["Planner classifies intent"]
    Planner --> Regime{"Auto / Creative / Grounded"}
    Regime --> Creative["Creative evidence regime"]
    Regime --> GroundedAnswer["Grounded evidence regime"]
    Creative --> EvidenceBinding["Enabled-source evidence binding"]
    GroundedAnswer --> EvidenceBinding
    EvidenceBinding --> Retrieval["Retrieval + Reranking when evidence is needed"]
    Retrieval --> ChunkRepo
    EvidenceBinding --> Workload["Workload planner"]
    Workload --> ChatWork["chat"]
    Workload --> MapReduce["chat_map_reduce"]
    Workload --> AgentWork["agent_map_refine_compose"]
    ChatWork --> LLM["Ollama assistant (concurrency-gated)"]
    MapReduce --> LLM
    AgentWork --> LLM
    LLM --> Contracts["Language + citation + table + support contracts"]
    Contracts --> QueryRepo["Query + Answer JSON repos"]
    

Startup Behavior

The FastAPI lifespan hook currently does these checks before serving:

• OpenSearch connectivity (always)

• document catalog index readiness (always)

• retrieval pyramid index readiness (always)

• embedding model readiness (always)

• reranker readiness (only when AIBS_RERANKER_ENABLED=true)

• document parser warmup (only when AIBS_INGESTION_WORKER_ENABLED=true)

• LLM readiness (only when AIBS_LLM_ENABLED=true and AIBS_LLM_STARTUP_CHECK=true)

If any enabled check fails, startup fails.

When AIBS_INGESTION_WORKER_ENABLED=true (the default), the lifespan also starts an in-process IngestionWorkerLoop background task. When set to false, the API only submits jobs — a separate process running python -m aibs_backend.interfaces.cli.ingestion_worker claims and executes them. That standalone worker process runs its own identical startup checks (excluding LLM unless abstractive summaries are enabled), including the document catalog index because ingestion writes document-state changes through that projection.

That is intentional. Advertising healthy startup while a declared capability is unreachable would be misleading.

Current Boundaries That Matter

• Workspace ownership is enforced at the API boundary for document-specific endpoints.

• Upload workflow is backend-owned on the happy path: POST /api/aibs/documents submits the ingestion job, while /api/aibs/documents/{id}/process is for manual retry/recovery only.

• Ingestion is now driven by a durable OpenSearch-backed job queue with OCC-based claiming, lease heartbeats, retry scheduling, and stale lease reaping. Multiple worker processes can safely compete for jobs without double-execution.

• The worker can run in-process within the API (default) or as a standalone CLI process on a separate machine with access to the same OpenSearch and shared storage roots.

• Workspace source-list reads use an OpenSearch catalog projection, while JSON document files remain the aggregate source of truth. Backfill the projection with python -m aibs_backend.interfaces.cli.rebuild_document_catalog when enabling it for existing workspaces; run a full rebuild without --workspace for catalog mapping schema upgrades.

• Document metadata persistence (workspace, conversation, document records) is local JSON, which is good for current wiring and local validation but not yet the final production persistence story.