# Current Implementation Decisions This page records the important choices that shape the backend today. It is not a full ADR set yet. It is a truthful snapshot of decisions that developers should know before changing the ingestion slice. ## 1. FastAPI is the current HTTP framework Why: - the repo dependencies and actual interface code are FastAPI-based - OpenAPI generation and browser testing through `/docs` matter for the current frontend-wiring phase - current planning and implementation context converged on FastAPI Implication: - new HTTP work should follow the FastAPI interface layer already in `interfaces/` ## 2. Uploads are staged before registration Why: - browser uploads should not be read fully into memory just to hand them to the application layer - checksum calculation and managed storage copy can happen from a staged file Implication: - the upload route stages files under `.aibs_data/staging` - the file store port accepts staged paths ## 3. The local model directory is the default embedding source Why: - local/backend smoke tests should be reproducible - live model downloads are fragile in enterprise or partially offline setups - model availability is a first-class runtime dependency for this slice Implication: - the app expects the embedding model under `models/embeddings/bge-m3` - startup checks fail if the embedder is not ready ## 4. Workspace, conversation, and document metadata are JSON-backed for this phase Why: - local validation and branch iteration are easier with simple file-backed persistence - this keeps the current wiring slice lightweight while the domain, access, and API boundaries are still settling Implication: - duplicate detection is best-effort, not transactional - workspace records, conversation records, and document metadata all use the same JSON-backed persistence approach for now - this is good enough for current wiring, not the final production store ## 5. Chunk indexing is OpenSearch-backed now Why: - the ingestion slice is not complete unless embedding and indexing are proven against the real search boundary - retrieval work later depends on trustworthy chunk persistence now Implication: - startup checks OpenSearch connectivity - chunk indices are workspace-scoped - chunk documents include embeddings as `knn_vector` ## 6. Ingestion is driven by a durable OpenSearch-backed job queue Why: - the browser should not wait for parse/chunk/embed/index completion - jobs must survive worker crashes and API restarts without getting stuck - multiple worker processes (local in-process or remote GPU worker) must be able to compete for jobs without double-execution Implication: - `POST /process` calls `SubmitIngestionJobService` and returns `202 Accepted` - the job is persisted to OpenSearch with OCC-based claiming (`if_seq_no` / `if_primary_term`) - `IngestionJobRunner` advances jobs through parse → chunk → index idempotently - each active stage is heartbeated; expired leases are reaped and retried - the worker runs in-process within the API by default (`AIBS_INGESTION_WORKER_ENABLED=true`) - when that flag is `false`, the standalone CLI `python -m aibs_backend.interfaces.cli.ingestion_worker` owns job execution - `job_id` is returned in the `ProcessAcceptedResponse` for traceability ## 7. Workspace access is enforced through the AIBS auth boundary Why: - AIBS is a separate product even when it reuses an OpenSearch/Logserver-backed identity store - authenticated users still need explicit AIBS product access - workspace-scoped routes need a stable backend user identity before conversation memory and ownership can be trusted Implication: - login validates credentials through OpenSearch/Logserver and requires `aibs-access` - AIBS product administration requires both `aibs-access` and `aibs-admin` - browser auth uses an HTTP-only opaque session cookie - the API constructs `UserContext` from the authenticated principal - workspace-scoped routes enforce access through `WorkspaceAccessFilter` - workspace membership roles are `owner`, `admin`, `member`, and `viewer` - product user management is explicit through `/api/aibs/admin/users` - workspace member management is explicit through workspace member routes - workspace member-add by username resolves through the AIBS user directory so free-form ghost members cannot be created - owner-role changes are owner-only; admins manage non-owner membership - destructive workspace deletion remains owner-only - `X-AIBS-User-Id` is accepted only when `AIBS_AUTH_DEV_MODE=true` - `GET /documents/{id}` and `POST /documents/{id}/process` require `workspace_id` - first-admin bootstrap is now a controlled CLI: `python -m pdm run python -m aibs_backend.interfaces.cli.bootstrap_admin --username ` Provider migration note: the external auth provider name is part of stable AIBS user-id derivation for non-UUID subjects. If a deployment migrates from the OpenSearch auth provider to another provider later, the migration runbook must preserve or explicitly map provider/subject pairs so existing workspace ownership is not orphaned. Provider limitation note: the current OpenSearch/Logserver adapter still uses `client.info()` as the initial credential check before trying authinfo or `/_logserver/login` role discovery. This supports the current Logserver-backed cluster where credentials can authenticate to OpenSearch but roles come from Logserver. A pure Logserver-only deployment where `client.info()` rejects the same credentials needs a separate provider slice that treats `/_logserver/login` as the primary authentication path. ## 8. Workspace deletion closes the in-process workspace boundary first Why: - governance delete must not race with new workspace-scoped operations - archive and delete remain separate lifecycle concepts - the current phase still uses in-process coordination for background work Implication: - workspace delete marks the workspace as "delete in progress" at the interface boundary before destructive cleanup starts - workspace-scoped routes reject new operations while delete is in progress - this coordination is truthful for the current single-process wiring phase, not yet a durable multi-worker lock model ## 9. Workspace archive is a governance-owned lifecycle operation Why: - the domain already owns the `archive()` transition - destructive delete and soft archive should not be blurred together - this keeps workspace CRUD lighter while governance owns lifecycle controls Implication: - archive is handled through the governance application layer - destructive workspace cleanup remains a separate governance flow ## 10. Conversation records are lightweight backend-owned session containers Why: - the frontend should not be the long-term source of truth for workspace chat sessions - the current branch needs real backend conversation identity before adaptive answering and history reload can be trustworthy - conversation lifecycle remains intentionally lighter than workspace Implication: - conversation records are persisted and workspace-bound - new conversation records are private to their creator within the workspace - conversation list, read, history, lineage, and structured follow-up referents are scoped to the conversation creator, with a permissive fallback only for legacy records that have no `created_by` - the backend currently supports create/get/list/rename/delete for those records - recovered conversation history is rebuilt from persisted `QueryExecution` and `AnswerArtifact` records - the frontend reloads conversation turns from that backend history surface ## 11. Evidence-regime routing is now the stable query loop Why: - the product needs more than one evidence posture to stay both useful and truthful - Creative answers, Grounded answers, lineage-backed conversation follow-ups, and insufficient-evidence turns are all first-class outcomes now - `chat` and `agent` are workload strategies, not user-facing evidence modes Implication: - public user-facing modes are `auto`, `creative`, and `grounded` - internal workload strategies are `chat`, `chat_map_reduce`, and `agent_map_refine_compose` - conversation turns can complete as: - `creative` - `document_grounded` - `conversation_context` - `insufficient` - source-backed Creative and Grounded turns publish fresh answer artifacts with citations - conversation-context turns publish fallback artifacts that reuse explicit prior citation lineage - insufficient turns persist query truth without weakening the answer aggregate ## 12. Shared chunk typing lives above infrastructure now Why: - chunk shape should not be treated as a private OpenSearch adapter concern - chunking, embedding, persistence, and indexing all need the same stable seam Implication: - the code uses `DocumentChunkLike` as the shared chunk boundary - review-driven type tightening happened at the domain/application seam, not only inside one adapter ## 13. LLM access is serialized through a concurrency gate Why: - the local Ollama serving lane typically handles one request at a time - classifier plus answer calls across overlapping turns can otherwise pile up and surface as latency or adapter errors Implication: - the bootstrapped answer assistant is wrapped in `ConcurrencyLimitedAnswerAssistant` - concurrency is driven by `AIBS_LLM_MAX_CONCURRENT_REQUESTS` (default `1`) - raising this value is only meaningful when a multi-lane serving setup is confirmed, not as a blanket performance knob ## 14. Document language is a structured aggregate field, not a free-text hint Why: - language identity drives routing decisions (e.g. counting documents by language) and prompt construction; an inferred free-text hint is too weak to support either - the detection step should run at ingestion, not at query time, because the parsed body is the strongest available evidence and the answer path should not pay re-detection cost Implication: - `Document` carries a typed `DocumentLanguage` value object (ISO 639-1 primary code, confidence, detected-at timestamp, classifier identity), or `None` when detection had no confident verdict - detection runs after parse and before chunking, outside the parse try/except, so detector failures cannot mark a document as a failed parse - `PlanningDocument.language` and `WorkspaceDocumentFact.language` carry the structured ISO code; the legacy `language_evidence` free-text hint stays for backwards compatibility with documents ingested before this step joined the pipeline - the backfill CLI populates structured language for pre-existing documents without re-running parse: `python -m aibs_backend.interfaces.cli.backfill_language` ## 15. Execution planning is split into intent and scope phases Why: - routing questions such as inventory, capability, and clarification should not require a document-list prompt - document-scoped questions still need title, alias, language, and summary metadata so the planner can resolve concrete document ids - a failed or truncated planner response should never silently route into full workspace retrieval, because that lane is the most expensive one and is also where wrong-but-confident answers are easiest to produce Implication: - the execution planner first builds an `IntentPlan` from only the question and bounded recent conversation turns - terminal intents (`inventory`, `capability`, `needs_clarification`) bypass document-scope planning - scoped intents (`metadata_fact`, `single_doc_retrieval`, `multi_doc_retrieval`, `workspace_retrieval`) run a second scope-planning call with the capped document prompt, lineage ids, and recent turns - the prior deterministic inventory regex pre-router was an interim measure and has been retired - when either planner phase returns output that cannot be parsed into a valid plan, the fallback is `answer_mode="needs_clarification"`, not workspace retrieval ## 16. Document writes are atomic and validated at startup Why: - a crashed write must not leave the document JSON in a half-serialized state that breaks every subsequent load - corruption introduced offline (manual edits, disk faults) should be surfaced at startup rather than as a stream of 500s once the system is serving traffic Implication: - the JSON document repository writes through a same-directory temp file and then `os.replace`; if any step before the rename fails, the original file is untouched and the temp file is cleaned up - on startup the repository loads every persisted document JSON and fails fast if any file is malformed, raising before the API begins accepting requests ## 17. Workspace document catalog reads use an OpenSearch projection Why: - scanning every document JSON file to render a 50-row source-list page does not scale on large workspaces, especially over shared VM/NTFS storage while ingestion is active - document discovery needs metadata search, filtering, summaries, and search result hydration without loading full aggregates - the projection can be rebuilt from the authoritative JSON documents, so it is safe to treat it as a read model instead of as source of truth Implication: - `CatalogIndexedDocumentRepository` wraps `JsonDocumentRepository` - saves and deletes retry writes to `aibs-document-catalog` by default; if the projection remains unavailable, the JSON source-of-truth operation still succeeds and the projection is repaired by rebuild - `AIBS_DOCUMENT_CATALOG_INDEX` controls the projection index name - `/documents/catalog` and search-result hydration read lightweight catalog entries from OpenSearch - the rebuild CLI repairs or initializes existing workspaces without clearing a workspace first: `python -m aibs_backend.interfaces.cli.rebuild_document_catalog` - catalog mappings carry a schema version; stale schemas require a full rebuild without `--workspace` because recreating the shared catalog index from a scoped command would affect other workspaces - catalog title/failure text uses a custom analyzer for common filename separators instead of storing a duplicated normalized search field - startup and standalone-worker readiness include the catalog index because ingestion changes document state and therefore writes the projection ## 18. Smart document search is hybrid search with guarded AI expansion Why: - users should not have to guess whether a title, exact text, or semantic query is the right search mode for a workspace source list - query expansion can improve recall for vague natural-language searches, but letting the LLM rewrite every catalog search would add latency and surprise Implication: - backend `mode=hybrid` is the frontend `Smart` mode - Smart combines metadata/title/language, chunk full-text, and semantic chunk signals with Reciprocal Rank Fusion (`k=60`) so rank consensus works across workspaces without per-corpus score calibration - assistant-backed query expansion is best-effort, bounded, non-thinking, and used only inside Smart mode - exact-looking queries such as filenames, ids, short numeric terms, and short language codes skip expansion - expanded matches are lower weight and marked with `ai_query_expansion` - text and semantic modes remain deterministic, direct modes ## 19. Broad document discovery uses a retrieval pyramid before chunk search Why: - broad questions should not start by scanning/fetching large chunk candidate windows when ingestion already produced document and section summaries - the document catalog projection is built for source-list paging, not for summary-aware discovery - the chunk index should remain the final evidence layer after profile/summary narrowing, not the first expensive step for every broad question Implication: - new document saves/deletes update two additional OpenSearch projections: `aibs-document-profiles` and `aibs-section-summaries` - `AIBS_DOCUMENT_PROFILE_INDEX` and `AIBS_SECTION_SUMMARY_INDEX` control those projection names - profiles store title, aliases, language, document family/type, ingestion summary, topics, status, and parser/chunker signals - section/table summaries store section title, breadcrumb, summary text, page span, document ID, type, language, and topic/entity hints - discovery tries profile/summary hits first and falls back to chunk evidence only when `AIBS_RETRIEVAL_PYRAMID_DISCOVERY_ENABLED=true`; with the flag disabled, pyramid indexes can be populated without changing retrieval behavior - existing workspaces can be backfilled with `python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid` ## Deliberate Deferrals Still intentionally deferred: - persistent auth session store for multi-worker/restart-surviving deployment - cross-workspace knowledge sharing rules - production-grade document metadata persistence (workspace/conversation/document records are currently local JSON; document catalog is an OpenSearch read model) - revisiting the leading-dot OpenSearch index naming before production hardening