Backfill Conversation Memory CLI

Module: aibs_backend.interfaces.cli.backfill_conversation_memory

Purpose

Populate the derived OpenSearch conversation-memory index from authoritative query and answer history. This index supports semantic retrieval of relevant older turns. Recent conversation history remains authoritative in query and answer persistence.

This is an explicit, one-shot maintenance operation. API processes do not run historical conversation-memory backfills during startup. The continuous outbox worker is separate and continues to index newly completed answers.

Prerequisites

  • AIBS_CONVERSATION_MEMORY_ENABLED=true

  • OpenSearch and the configured embedding model are available.

  • The process can read the same workspace, query, and answer persistence used by the API.

  • The process can write the configured conversation-memory OpenSearch index and the local checkpoint file.

Command

From the repository root with the backend virtual environment:

.\aibs\Scripts\python.exe -m aibs_backend.interfaces.cli.backfill_conversation_memory

With PDM:

pdm run python -m aibs_backend.interfaces.cli.backfill_conversation_memory

Execution Flow

  1. Load .env from the repository root and build the normal application container.

  2. Read the durable checkpoint and skip workspaces already marked complete.

  3. Page authoritative query history by workspace.

  4. Select the preferred persisted answer for each query.

  5. Exclude turns that are not eligible for semantic conversation memory.

  6. Build and index derived memory episodes in OpenSearch.

  7. Mark each completed workspace in the checkpoint.

  8. Print the number of episodes indexed during this invocation.

The default checkpoint is .aibs_data/conversation_memory_backfill_v2.json. When AIBS_CONVERSATION_MEMORY_OUTBOX_ROOT is customized, the checkpoint is named conversation_memory_backfill_v2.json in that path’s parent directory.

Idempotence And Recovery

The checkpoint records completed workspace IDs. Re-running the command with the same checkpoint skips those workspaces and normally reports indexed_count: 0. OpenSearch writes use stable memory IDs, so retrying an interrupted workspace replaces already-indexed episodes instead of creating duplicates.

The checkpoint and OpenSearch index are separate stores. If the derived index is deleted or recreated, retaining the old checkpoint would incorrectly skip completed workspaces. In that recovery case:

  1. Stop any concurrent conversation-memory backfill command.

  2. Preserve a copy of the checkpoint for audit and rollback.

  3. Remove the active conversation_memory_backfill_v2.json checkpoint.

  4. Run the command once and verify its final indexed count and logs.

Do not reset the checkpoint for routine reruns.

Output And Exit Behavior

Successful runs print:

Conversation memory backfill completed.
indexed_count: <number>

Exit behavior

Meaning

0

Backfill completed successfully.

2

Conversation memory is disabled or its worker is unavailable.

Other non-zero

An unexpected persistence, OpenSearch, or embedding failure occurred.

Failures are not swallowed. The command closes application resources and exits non-zero so operators and automation can detect incomplete work.