# Rebuild Retrieval Pyramid CLI

Module: `aibs_backend.interfaces.cli.rebuild_retrieval_pyramid`

## Purpose

Rebuild or validate retrieval projection indexes:

- document profile index: `AIBS_DOCUMENT_PROFILE_INDEX`
- section/table summary index: `AIBS_SECTION_SUMMARY_INDEX`

This CLI is a projection repair tool. It reads persisted documents and writes
OpenSearch read models. It does not create semantic profiles. Run
[backfill_semantic_profiles.md](backfill-semantic-profiles.md) first when old
documents have missing, legacy, or unavailable semantic profiles.

## Command

```bash
pdm run python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid \
  [--workspace <uuid>] [--document <uuid>] [--batch-size 200] \
  [--dry-run|--validate-only] [--recreate] [--only-missing] \
  [--profile-version N]
```

or:

```bash
python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid \
  [--workspace <uuid>] [--document <uuid>] [--batch-size 200] \
  [--dry-run|--validate-only] [--recreate] [--only-missing] \
  [--profile-version N]
```

## Options

| Option | Meaning |
| --- | --- |
| `--workspace <uuid>` | Rebuild one workspace. Omit to process all workspaces. |
| `--document <uuid>` | Repair or validate one document's profile/summary projections inside `--workspace`. |
| `--batch-size <int>` | Batch size, clamped to `1..1000`; default is `200`. |
| `--dry-run` | Validate/count without OpenSearch writes. |
| `--validate-only` | Alias for read-only validation. |
| `--recreate` | Drop and recreate both retrieval projection indexes. |
| `--only-missing` | Skip documents that already have current profile rows. |
| `--profile-version <int>` | Rebuild only documents whose chunk policy major version differs from this value. |

Invalid combinations:

- `--document` without `--workspace`
- `--recreate` with `--workspace`
- `--recreate` with `--document`
- `--recreate` with `--dry-run` or `--validate-only`
- `--recreate` with `--only-missing`
- `--only-missing` with `--profile-version`
- `--profile-version <= 0`

## Execution Flow

1. Validate arguments and optional workspace/document IDs.
2. Build the application container from `.env`.
3. Resolve `document_projection_maintenance.rebuild_retrieval_pyramid_index`.
4. Run validation mode or rebuild mode.
5. Refresh written indexes.
6. Print a report.

## What Gets Projected

The document profile index projects planner/retrieval fields from the persisted
document identity profile, including source title, aboutness, language, semantic
profile facets, and completeness/confidence fields.

The section/table summary index projects section and table summaries used for
retrieval and evidence localization.

Only knowledge-active documents are indexed.

## Report Fields

Validation reports include expected/projected/matched/missing/stale profile
counts, semantic/source-truth completeness counts, summary counts, schema
versions, and recreate requirements.

Rebuild reports include deleted, indexed, skipped-existing,
skipped-current-version, and recreate flags.

## Schema Upgrade

Current schemas:

- document profile index: `7`
- section/table summary index: `1`

When a deployed retrieval projection index is older than the current schema, run
an unscoped recreate rebuild:

```bash
pdm run python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid --recreate
pdm run python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid --validate-only
```

Do not pass `--workspace` for schema recreation. The retrieval projection indexes
are shared.

## Operating Notes

- Run semantic profile backfill first for documents missing current semantic
  profiles.
- Use `--profile-version` after chunk policy changes when a full recreate is not
  required.
- Use `--workspace <uuid> --document <uuid>` for one-document profile/summary
  repair after a single ingestion or delete issue. This reprojects or removes
  only that document's retrieval rows and does not certify the whole workspace.
- Use `--only-missing` for outage repair when schema is current and stale rows
  are not the problem.
- In multi-process or multi-box deployments, set
  `AIBS_COORDINATION_BACKEND=redis` or `valkey` so projection maintenance and API
  processes share the same repair/dirty-state coordination boundary.