Answering Pipeline

Purpose

Turn a workspace-scoped natural language question into a truthful conversation answer. The selected evidence regime decides whether evidence is optional (creative) or mandatory (grounded), and the downstream workload strategy decides whether the answer can be composed directly, through bounded map-reduce, or through map-refine-compose agent work.

Entry Point

POST /api/aibs/chat/

The route enforces workspace membership, resolves or creates a persisted conversation for the current identity, then calls DispatchChatService and streams the result as server-sent events (SSE). If a request supplies an existing conversation id, the backend confirms the conversation belongs to the requester before planner lineage, inventory referents, or answer history can be reused.

Dispatch Layer

DispatchChatService routes each question through one conversation spine:

Direct workspace answer check - terminal for trusted identity, capability, and workspace metadata answers that do not need the planner.
Planner intent classification - emits typed intent/evidence/output signals without treating chat or agent as public modes.
Evidence-regime resolution - chooses creative or grounded, either from the user override or from auto arbitration.
Evidence binding - binds only enabled workspace sources when the selected regime needs or can use evidence.
Workload planning - chooses chat, chat_map_reduce, or agent_map_refine_compose after evidence binding.
Shared output contracts - language, citation, table/figure/chart, and support checks run before publication.

Two-Phase Execution Planning

Planning is split so basic routing does not pay for document-scope context:

Phase A intent classifier receives only the user question plus bounded recent conversation turns. It does not receive workspace_documents, document metadata, or document ids. It chooses a typed task family and detects reply_language.
Phase B scope planner runs only for document-scoped modes: metadata_fact, single_doc_retrieval, multi_doc_retrieval, and workspace_retrieval. It receives the capped planner document payload, lineage ids, and recent turns, then resolves document ids, metadata facts, answer shape, depth, operation, and section ordinal.

Terminal direct workspace answers bypass both planner phases. Planner terminal families such as inventory, capability, and needs_clarification can still bypass Phase B when no document scope is required. A question such as “how many Polish documents are in this workspace?” routes to inventory without sending the document list into the planner. A semantic question such as “which documents are related to road transport?” routes to discovery/retrieval because the answer depends on document content.

The previous deterministic inventory regex pre-router is now retired. Inventory routing is handled by Phase A instead of a hardcoded English pattern list.

Planner Fallback

When either planner phase returns output that cannot be parsed into a valid plan, the system does not silently fall through to full workspace retrieval. The fallback plan is answer_mode="needs_clarification" with a clarification question prompt.

Execution Planner

AssistantAnswerExecutionPlanner first asks the LLM for an IntentPlan, then asks for an AnswerExecutionPlan only when document scope is required. The final plan identifies:

answer_mode - the typed task family used by orchestration
document_ids - canonical document UUIDs for document-scoped questions
section_ordinal - a positive integer when the user asks for a specific chapter or ordinal section
fact_kind - metadata fact type for metadata_fact answers
reply_language - answer language hint
preferred_answer_shape, requested_depth, requested_operation, requested_scope, and reasoning_effort - answer-shaping and execution hints for Creative and Grounded answering

DocumentContext builds an alias map from the documents in the workspace. For each document the planner inspects the first 20 chunk texts to find abbreviation pairs such as General Data Protection Regulation (GDPR) and derives title initialisms. Phase B receives these aliases so the model can resolve short-form references (GDPR, OGD) to canonical document UUIDs without the user needing to type exact titles.

Each PlanningDocument also exposes a structured language field carrying the ISO 639-1 primary code from Document.language.primary, or null when no structured language metadata is present. The planner instruction prefers structured language over the free-text language_evidence hint and falls back to language_evidence only when language is missing.

Evidence Regime And Workload Boundary

The client can request auto, creative, or grounded.

auto lets the backend choose the evidence regime.
creative allows model reasoning/opinion and optional enabled-source evidence. When source evidence is used, source-backed claims need citations.
grounded requires permitted enabled evidence for factual claims and must abstain or clarify when support is insufficient.

After the evidence regime and evidence binding are resolved, the workload planner chooses the internal physical strategy:

chat for bounded direct execution.
chat_map_reduce for large context that still only needs simple synthesis.
agent_map_refine_compose for broad, high-reasoning, multi-step, or multi-source work.

The workload resolver is deterministic over typed plan fields such as answer_mode, requested_depth, reasoning_effort, selected document count, and evidence-context size. However, requested_depth and reasoning_effort are LLM-planner-derived typed intent fields, not deterministic keyword matches. That means route decisions are traceable in logs, but still need benchmark coverage when planner intent shifts across equivalent phrasings.

Agent Flow Admission And Queue ETA

agent_map_refine_compose work has a whole-flow admission layer above individual model-call admission. This prevents each broad query from independently opening a full document-map fleet when another broad query is already running.

The flow scheduler emits progress events over the chat SSE stream:

admitting_workload / accepted when the request enters admission,
admitting_workload / queued with queue position and ETA when capacity is saturated,
admitting_workload / running when the query receives a flow lease,
generating_answer / streaming when document-map work starts under the granted parallelism.

The user-facing queued message is explicit: Arbiter is at capacity, the query is queued, and an estimated reply time is provided. ETA starts from configured seconds-per-document and is refined from observed completed agent-flow durations inside the process. The displayed ETA applies a small configurable buffer so the product promise is conservative without changing scheduler ordering or capacity decisions.

The scheduler stores queue and active-flow state behind the shared coordination port. With the default in-memory adapter this is still process-local; with AIBS_COORDINATION_BACKEND=valkey or redis, the same scheduler state path is shared across API workers. It allocates a per-flow map parallelism limit and can reduce broad-query map parallelism when multiple agent flows are admitted. Per-model admission still enforces resident weight and in-flight KV estimates for each LLM call.

Short heartbeat timestamps are not used as a slot-release authority. A missed heartbeat must not make the scheduler admit another broad flow while the original map fleet is still running. Active-flow cleanup instead uses a separate generous dead-man ceiling that is refreshed by the heartbeat and defaults to one hour. That preserves the fail-closed behavior for transient heartbeat stalls while still self-healing a leaked active slot after a crash or failed cleanup. A future multi-process deployment should keep this liveness authority in the shared coordination backend rather than in client-local clocks.

When a positive safe-pool budget is configured, the flow scheduler clamps map parallelism down to the largest value that fits the estimated budget before admitting a flow. If even sequential map work plus the reduce estimate cannot fit, admission fails explicitly as capacity/misconfiguration instead of silently overcommitting the model host. Admission progress delivery is best-effort: a failed progress sink is logged, but it cannot wedge the queue or leak an active flow slot.

The application has split coordination contracts for best-effort cache values and safety-critical leases. Scheduler state writes are fenced by the held state-lock token, so a stale scheduler owner cannot overwrite queue/active-flow state after its lock expires. Global model placement uses Valkey/Redis-backed atomic leases and fails closed when shared coordination is unavailable or node residency is stale. The dedicated coordination architecture is documented in docs/architecture/model-coordination.md.

Planner Task Families

Mode	Description
`metadata_fact`	Direct repository metadata fact about one known document
`inventory`	Workspace document inventory, counts, statuses, languages, and search readiness
`capability`	Answer about what the system can do
`single_doc_retrieval`	One-document retrieval and grounded answer
`multi_doc_retrieval`	Multi-document retrieval and grounded answer
`workspace_retrieval`	Retrieval across the workspace when no specific scope is fixed
`needs_clarification`	Question is too ambiguous to answer without more detail

Workspace Awareness Service

Handles inventory, capability, and needs_clarification modes without running retrieval.

For inventory questions, the service queries the workspace document list and applies the is_searchable signal. A document is searchable when its status is ready or degraded because both have indexed chunks. The prompt guidance uses searchable=true as the filter, not status=ready, so degraded documents are correctly reported as queryable.

The inventory summary computed for the LLM includes exact counts across the full workspace, not from the capped planner prompt sample. The summary surfaces:

total_documents
searchable_documents
by_media_type
by_status
by_language, computed from each document’s structured Document.language.primary; documents without language metadata are bucketed as unknown

Per-document inventory facts include a language=<iso-code> line only when the document has structured language metadata; documents bucketed as unknown omit the field to avoid noisy facts.

Deterministic catalog document table

When the planner classifies an inventory request as a document listing (answer_mode=inventory, inventory_fact_kind=unknown, table/document shape), the dispatcher renders a deterministic WORKSPACE ANSWER table (provider deterministic, model workspace_catalog) without calling the answer LLM. Three behaviours matter for large workspaces:

Honest totals. The heading always states the true workspace total, not an opaque “first N”: for example, Showing the first 25 of 930 documents in this workspace - the closest matches..., or Showing all 8 of 8 documents in this workspace: when nothing is hidden. The total comes from the catalog summary, so the answer never hides how many documents exist.
Pagination. When a full page is returned and more documents remain, the result carries a continuation (token + total + remaining). The client can request the next page with that continuation_token; the dispatcher re-runs the catalog candidate search at the next offset and renders Showing documents 26-50 of 930.... Search offset windows are disjoint, so pages never overlap. The per-page cap is AIBS_CATALOG_INVENTORY_LIST_LIMIT.

Grounded Answer Service

Handles strict-evidence retrieval families such as single_doc_retrieval, multi_doc_retrieval, and workspace_retrieval. Source-backed Creative answers reuse the same enabled-source/evidence and shared workload boundaries where appropriate, but the Creative regime may also add bounded model reasoning or opinion around cited evidence.

Document Scope Resolution

Resolves the planner’s document_ids into a concrete retrieval scope:

loads document metadata records
builds DocumentRetrievalPolicy with workspace filter, document id list, and optional section_ordinal

For broad questions without a fixed document scope, document discovery first tries the retrieval pyramid projections: document profiles and section/table summaries. The chunk index remains the final evidence layer after that narrowing step, with broad chunk evidence used only as fallback when the pyramid has no usable hit or is unavailable. This read path is gated by AIBS_RETRIEVAL_PYRAMID_DISCOVERY_ENABLED; with the flag disabled, broad discovery uses the existing chunk-evidence path even if the profile/summary indexes are populated.

Retrieval

DocumentRetriever (OpenSearch-backed) runs a hybrid search:

KNN vector search - dense passage retrieval using the embedded query vector; no primary-segment pre-filter to preserve multi-vector reach across documents
Lexical BM25 search - keyword match, filtered by workspace_id and optionally section_ordinal
Section ordinal pre-filter - when set, applied to the lexical segment only; restricts results to chunks from the heading-derived chapter or section number matching the planner’s section_ordinal field

After combining KNN and lexical results, the retriever applies neighbor expansion: adjacent chunks from the same section are added to give the LLM more context. The neighbor guard filters out chunks whose section_ordinal does not match when an ordinal filter is active, preventing cross-chapter bleed during expansion.

Document-Local Agent Research

For broad agent_map_refine_compose multi-document work, answering builds typed document work items. Each work item is tied to one assigned document and carries:

document id and title
requested subjects and fields from the planner
evidence budget derived from document complexity
eligible worker model ids
neighbor expansion depth
summary vs structured/fact-seeking operation

The document worker is not allowed to wander across the workspace. It retrieves within its assigned document only. For structured or fact-seeking work, requested subjects and fields drive the document-local search first, then neighbor expansion adds adjacent chunks around those targeted hits. For genuine overview or summary requests, the worker can use the original document-level question instead of forcing a subject filter.

Neighbor expansion depends on indexed chunk_sequence metadata. If old chunks were indexed without sequence metadata, the system cannot honestly infer neighbors from storage order; those documents should be reingested or repaired before relying on document-local expansion tests.

Document complexity decides evidence budget and model eligibility from structural signals such as page count, chunk count, section count, table count, degraded state, and requested-field/subject pressure. Small documents can use lighter map workers. Large or degraded documents can prefer larger reasoning workers. The model admission boundary still serializes calls and applies process-local residency/concurrency controls before the worker calls the LLM.

Reranking

When AIBS_RERANKER_ENABLED=true, CrossEncoderDocumentReranker scores each retrieved chunk against the query using a local cross-encoder model and re-orders by relevance score.

Evidence Sufficiency

Before calling the LLM, the service evaluates whether the retrieved chunks contain enough evidence to justify a grounded answer. If not, the turn completes as insufficient and no LLM call is made.

Answer Generation

The answer prompt is built from:

the question
retrieved and optionally reranked chunks with their source metadata
document-level summaries as context preamble

The LLM (OllamaAnswerAssistant) generates a grounded answer with inline citations. The concurrency gate (ConcurrencyLimitedAnswerAssistant) serializes concurrent calls to a single in-flight request by default (AIBS_LLM_MAX_CONCURRENT_REQUESTS=1).

When AIBS_LLM_THINKING_ENABLED=true, the assistant uses the model’s reasoning trace for natural-language answer calls and returns thinking separately from the answer text as response metadata. Structured JSON calls, including planning and summaries, always send think=false regardless of this flag.

Citation Persistence

After the answer is generated:

a QueryExecution record is persisted with query text, retrieval results, and answer mode
an AnswerArtifact is persisted with answer text, confidence label, and citations mapped back to source document and chunk ids
the conversation history service can later rebuild full conversation turns from these records without a separate transcript store
conversation history and lineage reads are scoped to the conversation creator, with only legacy created_by=None records remaining readable by workspace members

Answer Stream Contract

The SSE response includes:

lifecycle progress events
provisional answer text when the current path can stream it safely
final validated answer blocks
final answer mode/regime truth, such as creative, document_grounded, insufficient, inventory, capability, or clarification
query id and answer id for lineage
citations with document title, page reference, and chunk id

Execution Planning

Execution planning uses the configured AIBS_LLM_MODEL through the same assistant boundary as answer generation. Planning calls request compact JSON output with thinking disabled, while answer calls use the normal grounded-answer prompt path.