Chunking Pipeline

Purpose

Transform a parsed canonical document into retrieval-optimised chunk records that carry enough context for both dense (vector) and lexical (BM25) search.

Entry Point

ChunkDocumentService.execute(ChunkDocumentCommand) — called by IngestionJobRunner in the chunk stage after successful parsing.

Stage Flow

parsed document
  → ChunkingBuilder.build_chunks(document)
      → ChunkingService (section + table + document chunks)
      → ExtractiveSummaryBuilder (preamble source for the document chunk)
  → [optional] AssistantBackedDocumentSummarizer (abstractive summaries)
  → ChunkSet attached to document aggregate

Chunk Layers

Every chunk record carries a layer field identifying its origin:

Layer	Source	Purpose
`section`	A heading-bounded text section of the document	Core retrieval unit for prose
`table_row`	A row or row group from a parsed table	Structured data retrieval
`document`	The whole document synthesised into one chunk	Workspace-level overview retrieval

Section Chunks

ChunkingService walks the canonical SectionNode tree. For each section it builds one ChunkRecord whose retrieval_text is structured as:

Document summary: <extractive or abstractive overview>
Document: <title>
Section summary: <section summary if available>
Content:
<section text>

Each section chunk carries two ordinal fields:

ordinal — global sequential position of the section across the whole document (1, 2, 3, … regardless of heading level or numbering)
section_ordinal — the number embedded in the heading itself, extracted by terminal_heading_number() (e.g. “Chapter 3” → 3, “Foreword” → 0)

section_ordinal is what the execution planner emits and the document retriever filters on. It maps user intent (“first chapter”, “Chapter 3”) to the correct set of chunks without relying on position in the parse tree.

Table Chunks

TableChunker builds one or more ChunkRecord entries per table. Each row group becomes a chunk carrying the table caption, column headers, and row data. parent_table_id links table-row chunks back to their source table for abstractive summary injection.

Document Chunk

One document-layer chunk is built per document. It carries:

the document-level summary (extractive or abstractive) as its primary content
a summary_packet in its metadata with global_synopsis, overview_summary, and retrieval_lenses (overview, table summary, section guide)

This chunk is used for workspace-level “what is this about?” queries without needing to retrieve individual sections.

Extractive Summary Source

ExtractiveSummaryBuilder selects a representative subset of sections and tables to feed into the document-level summary. It avoids:

orphan sections (TOC entries, index pages, back matter detected by the _orphans section id suffix)
injecting the TOC into every chunk’s preamble — long narrative sequences (books) are capped to one representative section to prevent the table of contents from dominating the preamble

The builder detects long narrative sequences by counting numbered heading sequences. When a document has more than twice the representative section limit of numbered headings (as in a novel with many short chapters), only one section is used in the extractive preamble.

Abstractive Summaries (Optional)

When AIBS_CHUNKING_ABSTRACTIVE_SUMMARIES_ENABLED=true, after extractive chunking AssistantBackedDocumentSummarizer uses the dedicated AIBS_LLM_SUMMARIZER_* assistant lane. It sends one overview call, then section batches of 3, then table batches of 3. The calls are sequential within one document, while the summarizer lane has its own concurrency gate so ingestion enrichment cannot consume the interactive chat lane.

The summarizer receives:

overview_summary — document-level overview replacing the extractive preamble
section_summaries — one summary per section id, injected into each matching section chunk’s retrieval_text as Section summary: …
table_summary — cross-table summary added to the document chunk
table_summaries — one summary per table id, injected into matching table-row chunks as Table summary: …

The summarizer guards against LLM hallucination by:

validating all returned section_id and table_id values against the valid_ids frozenset from the extractive source
deduplicating repeated ids with a seen_ids set
clipping per-section and per-table prompt input before each batch
passing the detected ISO language code directly when available, or asking the model to infer language from the extractive text when no code is available
raising out only when the overview call fails; malformed section/table batch responses are recorded as partial enrichment failures

If abstractive summarization fails (LLM offline, malformed JSON, timeout), the service logs a warning and falls back to the extractive summary. The chunk set is still persisted and the document advances to indexed status.

If only a section/table batch fails, the document chunk keeps the existing metadata["abstractive_summary"] block and adds partial-enrichment fields: overview_present, section_summary_count, table_summary_count, missing_section_ids, missing_table_ids, failed_batches, and degraded_summary. A degraded summary remains searchable; Stage 1 extractive metadata is the fallback.

The gate condition requires either selected_sections or selected_tables to be non-empty. Table-only documents (e.g. data sheets with no narrative sections) still receive an abstractive table summary.

Chunk Metadata Fields

Key fields on ChunkRecord used downstream by retrieval and answering:

Field	Type	Description
`layer`	str	`section`, `table_row`, or `document`
`ordinal`	int	Global sequential position in the document
`section_ordinal`	int	Heading-derived chapter/section number (0 = unnumbered)
`retrieval_text`	str	Full structured text sent to both embedding and BM25
`summary_text`	str	One-sentence summary of this chunk
`parent_section_id`	str \| None	Section id for table-row chunks
`parent_table_id`	str \| None	Table id for table-row chunks
`fingerprint`	str	Section id for section-layer chunks (used as the summary key)

Chunk Policy Version

ChunkingBuilder.build_policy_version() returns a hash of the current chunking configuration. It is stored on the ChunkSet and used to detect when a document’s chunks are stale relative to a new chunking policy version, enabling future selective re-chunking without full re-ingestion.