Backfill Language CLI

Module: aibs_backend.interfaces.cli.backfill_language

Purpose

Populate or repair persisted Document.language values for already-ingested documents. Use this when documents predate language detection, parser output has changed, or a workspace has incomplete language metadata.

This CLI changes document metadata only. It does not rebuild catalog or retrieval indexes; run the projection rebuild CLIs afterward if projected language fields need to be refreshed.

Command

pdm run python -m aibs_backend.interfaces.cli.backfill_language \
  [--workspace <uuid>] [--dry-run] [--force]

or:

python -m aibs_backend.interfaces.cli.backfill_language \
  [--workspace <uuid>] [--dry-run] [--force]

Options

Option

Meaning

--workspace <uuid>

Process only one workspace. Omit to process all workspaces.

--dry-run

Detect and report without saving document changes.

--force

Re-detect even when Document.language already exists.

Execution Flow

  1. Validate the optional workspace ID.

  2. Build the application container from .env.

  3. Page documents through the document repository.

  4. Skip documents that already have language unless --force is set.

  5. Build a language-detection sample from parsed content, metadata, title, and chunk summaries.

  6. Run the configured language detector.

  7. Save changed documents unless --dry-run is set.

  8. Print a summary report.

Report Fields

Field

Meaning

total

Documents inspected.

detected

Documents updated with a detected language.

unchanged

Detector matched the existing value.

skipped_existing

Existing language was present and --force was not set.

skipped_confidence

Detector returned no acceptable result.

skipped_no_text

No usable text sample was available.

failed

Per-document detection failures.

Exit Codes

Code

Meaning

0

Completed successfully.

2

Invalid workspace ID or workspace not found.

Unexpected runtime exceptions exit non-zero through the Python process.

Operating Notes

  • Run --dry-run first on large workspaces.

  • Use --force only when existing language values are suspected stale.

  • Follow with catalog/retrieval projection rebuilds if the UI or planner depends on projected language fields.