# Backfill Semantic Profiles CLI

Module: `aibs_backend.interfaces.cli.backfill_semantic_profiles`

## Purpose

Build persisted semantic document profiles for already-ingested documents.

Semantic profiles are source-grounded document metadata used by routing,
category document-set discovery, catalog planning, and retrieval projections.
They are not route-memory records and they are not parser/processing family labels.

Use this CLI for documents ingested before semantic profiles existed, after
semantic profile schema/quality changes, or after degraded semantic profiles
were persisted during the normal ingestion path.

## Command

```bash
pdm run python -m aibs_backend.interfaces.cli.backfill_semantic_profiles \
  [--workspace <uuid>] [--dry-run] [--force] [--limit N] \
  [--page-size 50] [--concurrency 2]
```

or:

```bash
python -m aibs_backend.interfaces.cli.backfill_semantic_profiles \
  [--workspace <uuid>] [--dry-run] [--force] [--limit N] \
  [--page-size 50] [--concurrency 2]
```

## Options

| Option | Meaning |
| --- | --- |
| `--workspace <uuid>` | Process one workspace. Omit to process all workspaces. |
| `--dry-run` | Run enrichment and report results without saving document changes. |
| `--force` | Rebuild even when a current non-legacy profile already exists. |
| `--limit N` | Maximum number of source documents to inspect in this run. Omit for full coverage. |
| `--page-size N` | Repository page size. Default is `50`; must be positive. |
| `--concurrency N` | Maximum documents enriched concurrently by this CLI process. Default is `2`; must be positive. |

## Semantic Profile Contract

The persisted profile schema is `SemanticDocumentProfileV1` and includes:

- `document_roles`
- `domain_families`
- `business_types`
- `topics`
- `entities`
- `key_sections`
- `document_set_keys`
- `legal_references`
- `completeness`
- `confidence`
- `evidence_refs`
- `built_by`
- `schema_version`

Values must come from the document's own source packet: parsed metadata,
document context, section headings, and chunk samples. File names and titles may
help identify the document, but they must not be treated as enough evidence for
semantic category truth unless document content also supports the label.

Processing labels such as `narrative_document`, `research_pdf`, or
`technical_manual` belong to ingestion behavior and must not be copied into
semantic fields.

## Model Lane

The default semantic-profile model is `qwen3:8b`.

Relevant settings:

| Setting | Meaning |
| --- | --- |
| `AIBS_LLM_SEMANTIC_PROFILE_PROVIDER` | Optional provider override. |
| `AIBS_LLM_SEMANTIC_PROFILE_BASE_URL` | Optional model endpoint override. |
| `AIBS_LLM_SEMANTIC_PROFILE_MODEL` | Model used for enrichment. Default: `qwen3:8b`. |
| `AIBS_LLM_SEMANTIC_PROFILE_MAX_CONCURRENT_REQUESTS` | Per-model request gate. |
| `AIBS_LLM_SEMANTIC_PROFILE_REQUEST_TIMEOUT_SECONDS` | Request timeout. |
| `AIBS_LLM_SEMANTIC_PROFILE_ADMISSION_QUEUE_TIMEOUT_SECONDS` | Admission wait limit for semantic-profile model work. |
| `AIBS_LLM_SEMANTIC_PROFILE_MAX_OUTPUT_TOKENS` | JSON output budget. |
| `AIBS_LLM_SEMANTIC_PROFILE_CONTEXT_LENGTH` | Context budget for the source packet. |
| `AIBS_LLM_SEMANTIC_PROFILE_KEEP_ALIVE` | Model keep-alive hint. |
| `AIBS_MODEL_ADMISSION_SEMANTIC_PROFILE_*` | Global admission weighting for the semantic-profile role. |

## Execution Flow

1. Validate arguments and optional workspace ID.
2. Build the application container from `.env`.
3. Page documents from the selected workspace or all workspaces.
4. Skip failed/deleted documents.
5. Skip documents without parsed and chunked artifacts.
6. Skip current non-legacy profiles unless `--force` is set.
7. Build a baseline artifact profile from existing metadata, context, headings,
   chunks, and identifier patterns.
8. Call the configured semantic-profile enricher with retries.
9. Persist the enriched `semantic_profile` on the document unless `--dry-run` is
   set.
10. Print progress and a final report.

## Failure Behavior

Live ingestion and this maintenance CLI intentionally differ:

- Live ingestion retries semantic enrichment and can persist an unavailable
  profile so the durable ingestion job completes degraded.
- This CLI reports failed documents and does not save an unavailable replacement.

That behavior is deliberate. Backfill is a repair tool, so failures should stay
visible to the operator.

## Report Fields

| Field | Meaning |
| --- | --- |
| `total` | Documents inspected. |
| `enriched` | Documents with a newly saved or dry-run enriched profile. |
| `unchanged` | Existing profile matched the enriched result. |
| `skipped_existing` | Current profile exists and `--force` was not set. |
| `skipped_missing_artifacts` | Parsed document or chunk set is missing. |
| `skipped_terminal` | Document is failed or deleted. |
| `failed` | Enrichment failed after retries. |

Dry runs prefix output with `[DRY-RUN]` and print `No documents were persisted.`

## Recommended Maintenance Sequence

For a scoped backfill:

```bash
pdm run python -m aibs_backend.interfaces.cli.backfill_semantic_profiles \
  --workspace <uuid> --dry-run

pdm run python -m aibs_backend.interfaces.cli.backfill_semantic_profiles \
  --workspace <uuid>

pdm run python -m aibs_backend.interfaces.cli.rebuild_document_catalog \
  --workspace <uuid> --validate-only

pdm run python -m aibs_backend.interfaces.cli.rebuild_document_catalog \
  --workspace <uuid>

pdm run python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid \
  --workspace <uuid> --validate-only

pdm run python -m aibs_backend.interfaces.cli.rebuild_retrieval_pyramid \
  --workspace <uuid>
```

Use `--force` only when semantic profile quality or schema changed and existing
profiles must be regenerated.

## Validation

Use code-owned artifacts and CLI reports for validation:

- the CLI report should have `failed=0`
- `skipped_missing_artifacts` should be explained by missing parsed/chunked state
- sampled persisted documents should contain `parsed_document.metadata.semantic_profile`
- current profiles should have `schema_version=1`
- profiles with `completeness=missing` or `confidence=low` should be reviewed
- projection validation counts should be clean after catalog and retrieval rebuilds

Do not validate a full selected scope with `--limit` unless the goal is only a
smoke test.