Rebuild Retrieval Pyramid CLI

Module: aibs_backend.interfaces.cli.rebuild_retrieval_pyramid

Purpose

Rebuild or validate retrieval projection indexes:

document profile index: AIBS_DOCUMENT_PROFILE_INDEX
section/table summary index: AIBS_SECTION_SUMMARY_INDEX

This CLI is a projection repair tool. It reads persisted documents and writes OpenSearch read models. It does not create semantic profiles. Run backfill_semantic_profiles.md first when old documents have missing, legacy, or unavailable semantic profiles.

Command

pdm run python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid \
  [--workspace <uuid>] [--document <uuid>] [--batch-size 200] \
  [--dry-run|--validate-only] [--recreate] [--only-missing] \
  [--profile-version N]

or:

python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid \
  [--workspace <uuid>] [--document <uuid>] [--batch-size 200] \
  [--dry-run|--validate-only] [--recreate] [--only-missing] \
  [--profile-version N]

Options

Option	Meaning
`--workspace <uuid>`	Rebuild one workspace. Omit to process all workspaces.
`--document <uuid>`	Repair or validate one document’s profile/summary projections inside `--workspace`.
`--batch-size <int>`	Batch size, clamped to `1..1000`; default is `200`.
`--dry-run`	Validate/count without OpenSearch writes.
`--validate-only`	Alias for read-only validation.
`--recreate`	Drop and recreate both retrieval projection indexes.
`--only-missing`	Skip documents that already have current profile rows.
`--profile-version <int>`	Rebuild only documents whose chunk policy major version differs from this value.

Invalid combinations:

--document without --workspace
--recreate with --workspace
--recreate with --document
--recreate with --dry-run or --validate-only
--recreate with --only-missing
--only-missing with --profile-version
--profile-version <= 0

Execution Flow

Validate arguments and optional workspace/document IDs.
Build the application container from .env.
Resolve document_projection_maintenance.rebuild_retrieval_pyramid_index.
Run validation mode or rebuild mode.
Refresh written indexes.
Print a report.

What Gets Projected

The document profile index projects planner/retrieval fields from the persisted document identity profile, including source title, aboutness, language, semantic profile facets, and completeness/confidence fields.

The section/table summary index projects section and table summaries used for retrieval and evidence localization.

Only knowledge-active documents are indexed.

Report Fields

Validation reports include expected/projected/matched/missing/stale profile counts, semantic/source-truth completeness counts, summary counts, schema versions, and recreate requirements.

Rebuild reports include deleted, indexed, skipped-existing, skipped-current-version, and recreate flags.

Schema Upgrade

Current schemas:

document profile index: 7
section/table summary index: 1

When a deployed retrieval projection index is older than the current schema, run an unscoped recreate rebuild:

pdm run python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid --recreate
pdm run python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid --validate-only

Do not pass --workspace for schema recreation. The retrieval projection indexes are shared.

Operating Notes

Run semantic profile backfill first for documents missing current semantic profiles.
Use --profile-version after chunk policy changes when a full recreate is not required.
Use --workspace <uuid> --document <uuid> for one-document profile/summary repair after a single ingestion or delete issue. This reprojects or removes only that document’s retrieval rows and does not certify the whole workspace.
Use --only-missing for outage repair when schema is current and stale rows are not the problem.
In multi-process or multi-box deployments, set AIBS_COORDINATION_BACKEND=redis or valkey so projection maintenance and API processes share the same repair/dirty-state coordination boundary.