# Chunking Pipeline

## Purpose

Transform a parsed canonical document into retrieval-optimised chunk records
that carry enough context for both dense (vector) and lexical (BM25) search.

## Entry Point

`ChunkDocumentService.execute(ChunkDocumentCommand)` — called by
`IngestionJobRunner` in the chunk stage after successful parsing.

## Stage Flow

```
parsed document
  → ChunkingBuilder.build_chunks(document)
      → ChunkingService (section + table + document chunks)
      → ExtractiveSummaryBuilder (preamble source for the document chunk)
  → [optional] AssistantBackedDocumentSummarizer (abstractive summaries)
  → ChunkSet attached to document aggregate
```

## Chunk Layers

Every chunk record carries a `layer` field identifying its origin:

| Layer | Source | Purpose |
| --- | --- | --- |
| `section` | A heading-bounded text section of the document | Core retrieval unit for prose |
| `table_row` | A row or row group from a parsed table | Structured data retrieval |
| `document` | The whole document synthesised into one chunk | Workspace-level overview retrieval |

## Section Chunks

`ChunkingService` walks the canonical `SectionNode` tree. For each section it
builds one `ChunkRecord` whose `retrieval_text` is structured as:

```
Document summary: <extractive or abstractive overview>
Document: <title>
Section summary: <section summary if available>
Content:
<section text>
```

Each section chunk carries two ordinal fields:

- `ordinal` — global sequential position of the section across the whole document
  (1, 2, 3, … regardless of heading level or numbering)
- `section_ordinal` — the number embedded in the heading itself, extracted by
  `terminal_heading_number()` (e.g. "Chapter 3" → `3`, "Foreword" → `0`)

`section_ordinal` is what the execution planner emits and the document retriever
filters on. It maps user intent ("first chapter", "Chapter 3") to the correct
set of chunks without relying on position in the parse tree.

## Table Chunks

`TableChunker` builds one or more `ChunkRecord` entries per table. Each row
group becomes a chunk carrying the table caption, column headers, and row data.
`parent_table_id` links table-row chunks back to their source table for abstractive
summary injection.

## Document Chunk

One `document`-layer chunk is built per document. It carries:

- the document-level summary (extractive or abstractive) as its primary content
- a `summary_packet` in its metadata with `global_synopsis`, `overview_summary`,
  and `retrieval_lenses` (overview, table summary, section guide)

This chunk is used for workspace-level "what is this about?" queries without
needing to retrieve individual sections.

## Extractive Summary Source

`ExtractiveSummaryBuilder` selects a representative subset of sections and tables
to feed into the document-level summary. It avoids:

- orphan sections (TOC entries, index pages, back matter detected by the
  `_orphans` section id suffix)
- injecting the TOC into every chunk's preamble — long narrative sequences
  (books) are capped to one representative section to prevent the table of
  contents from dominating the preamble

The builder detects long narrative sequences by counting numbered heading
sequences. When a document has more than twice the representative section limit
of numbered headings (as in a novel with many short chapters), only one section
is used in the extractive preamble.

## Abstractive Summaries (Optional)

When `AIBS_CHUNKING_ABSTRACTIVE_SUMMARIES_ENABLED=true`, after extractive chunking
`AssistantBackedDocumentSummarizer` uses the dedicated `AIBS_LLM_SUMMARIZER_*`
assistant lane. It sends one overview call, then section batches of 3, then
table batches of 3. The calls are sequential within one document, while the
summarizer lane has its own concurrency gate so ingestion enrichment cannot
consume the interactive chat lane.

The summarizer receives:

- `overview_summary` — document-level overview replacing the extractive preamble
- `section_summaries` — one summary per section id, injected into each matching
  section chunk's `retrieval_text` as `Section summary: …`
- `table_summary` — cross-table summary added to the document chunk
- `table_summaries` — one summary per table id, injected into matching table-row
  chunks as `Table summary: …`

The summarizer guards against LLM hallucination by:

- validating all returned `section_id` and `table_id` values against the
  `valid_ids` frozenset from the extractive source
- deduplicating repeated ids with a `seen_ids` set
- clipping per-section and per-table prompt input before each batch
- passing the detected ISO language code directly when available, or asking the
  model to infer language from the extractive text when no code is available
- raising out only when the overview call fails; malformed section/table batch
  responses are recorded as partial enrichment failures

If abstractive summarization fails (LLM offline, malformed JSON, timeout), the
service logs a warning and falls back to the extractive summary. The chunk set
is still persisted and the document advances to indexed status.

If only a section/table batch fails, the document chunk keeps the existing
`metadata["abstractive_summary"]` block and adds partial-enrichment fields:
`overview_present`, `section_summary_count`, `table_summary_count`,
`missing_section_ids`, `missing_table_ids`, `failed_batches`, and
`degraded_summary`. A degraded summary remains searchable; Stage 1 extractive
metadata is the fallback.

The gate condition requires either `selected_sections` or `selected_tables` to be
non-empty. Table-only documents (e.g. data sheets with no narrative sections)
still receive an abstractive table summary.

## Chunk Metadata Fields

Key fields on `ChunkRecord` used downstream by retrieval and answering:

| Field | Type | Description |
| --- | --- | --- |
| `layer` | str | `section`, `table_row`, or `document` |
| `ordinal` | int | Global sequential position in the document |
| `section_ordinal` | int | Heading-derived chapter/section number (0 = unnumbered) |
| `retrieval_text` | str | Full structured text sent to both embedding and BM25 |
| `summary_text` | str | One-sentence summary of this chunk |
| `parent_section_id` | str \| None | Section id for table-row chunks |
| `parent_table_id` | str \| None | Table id for table-row chunks |
| `fingerprint` | str | Section id for section-layer chunks (used as the summary key) |

## Chunk Policy Version

`ChunkingBuilder.build_policy_version()` returns a hash of the current chunking
configuration. It is stored on the `ChunkSet` and used to detect when a document's
chunks are stale relative to a new chunking policy version, enabling future
selective re-chunking without full re-ingestion.