Rebuild Document Catalog CLI

Module: aibs_backend.interfaces.cli.rebuild_document_catalog

Purpose

Rebuild or validate the OpenSearch document catalog projection from persisted documents.

This CLI is a projection repair tool. It does not generate semantic profiles, repair parsed documents, or infer new document truth. If existing documents need semantic profiles, run backfill_semantic_profiles.md first.

Command

pdm run python -m aibs_backend.interfaces.cli.rebuild_document_catalog \
  [--workspace <uuid>] [--document <uuid>] [--batch-size 200] \
  [--dry-run|--validate-only] [--recreate] [--only-missing]

or:

python -m aibs_backend.interfaces.cli.rebuild_document_catalog \
  [--workspace <uuid>] [--document <uuid>] [--batch-size 200] \
  [--dry-run|--validate-only] [--recreate] [--only-missing]

Options

Option	Meaning
`--workspace <uuid>`	Rebuild one workspace. Omit to process all workspaces.
`--document <uuid>`	Repair or validate one document projection inside `--workspace`.
`--batch-size <int>`	Batch size, clamped to `1..1000`; default is `200`.
`--dry-run`	Validate/count without OpenSearch writes.
`--validate-only`	Alias for read-only validation.
`--recreate`	Drop and recreate the shared catalog index before rebuilding.
`--only-missing`	Skip catalog rows that already exist.

Invalid combinations:

--document without --workspace
--recreate with --workspace
--recreate with --document
--recreate with --dry-run or --validate-only
--recreate with --only-missing

Execution Flow

Validate arguments and optional workspace/document IDs.
Build the application container from .env.
Resolve document_projection_maintenance.rebuild_catalog_index.
Run validation mode or rebuild mode.
Refresh the index after writes.
Print a report.

What Gets Projected

Catalog schema 5 projects document identity and planning fields from persisted document state, including:

recovered content title and aliases
aboutness summary
language/source/date/version fields
processing document_family for diagnostics
semantic profile fields and completeness/confidence
compact search text for planner/catalog discovery

The processing document_family is not a semantic category. It is preserved for diagnostics and ingestion behavior, while semantic category signals come from the document-owned semantic_profile.

Report Fields

Validation reports include counts for expected, projected, matched, missing, and stale documents plus profile/source-truth completeness fields. Rebuild reports include deleted, indexed, skipped-existing, and recreate flags.

Read-only modes print a validation prefix and do not write to OpenSearch.

Schema Upgrade

Current catalog schema: 5.

When a deployed catalog index is older than schema 5, run an unscoped recreate rebuild:

pdm run python -m aibs_backend.interfaces.cli.rebuild_document_catalog --recreate
pdm run python -m aibs_backend.interfaces.cli.rebuild_document_catalog --validate-only

Do not pass --workspace for schema recreation. The catalog index is shared.

Operating Notes

Run semantic profile backfill first for documents missing current semantic profiles.
Use --validate-only before and after a recreate.
Use --workspace <uuid> --document <uuid> for one-row repair after a single upload/delete/status issue. This reprojects or deletes only that document row. It does not mark the workspace manifest clean; run workspace validation or workspace repair to certify the whole workspace.
Use --only-missing for outage repair when schema is current and stale rows are not the problem.
In multi-process or multi-box deployments, set AIBS_COORDINATION_BACKEND=redis or valkey. Catalog projection repair clears shared dirty-state records through the coordination backend; process-local memory is only suitable for local development and tests.