# Backfill Conversation Memory CLI

Module: `aibs_backend.interfaces.cli.backfill_conversation_memory`

## Purpose

Populate the derived OpenSearch conversation-memory index from authoritative
query and answer history. This index supports semantic retrieval of relevant
older turns. Recent conversation history remains authoritative in query and
answer persistence.

This is an explicit, one-shot maintenance operation. API processes do not run
historical conversation-memory backfills during startup. The continuous outbox
worker is separate and continues to index newly completed answers.

## Prerequisites

- `AIBS_CONVERSATION_MEMORY_ENABLED=true`
- OpenSearch and the configured embedding model are available.
- The process can read the same workspace, query, and answer persistence used by
  the API.
- The process can write the configured conversation-memory OpenSearch index and
  the local checkpoint file.

## Command

From the repository root with the backend virtual environment:

```powershell
.\aibs\Scripts\python.exe -m aibs_backend.interfaces.cli.backfill_conversation_memory
```

With PDM:

```bash
pdm run python -m aibs_backend.interfaces.cli.backfill_conversation_memory
```

## Execution Flow

1. Load `.env` from the repository root and build the normal application
   container.
2. Read the durable checkpoint and skip workspaces already marked complete.
3. Page authoritative query history by workspace.
4. Select the preferred persisted answer for each query.
5. Exclude turns that are not eligible for semantic conversation memory.
6. Build and index derived memory episodes in OpenSearch.
7. Mark each completed workspace in the checkpoint.
8. Print the number of episodes indexed during this invocation.

The default checkpoint is
`.aibs_data/conversation_memory_backfill_v2.json`. When
`AIBS_CONVERSATION_MEMORY_OUTBOX_ROOT` is customized, the checkpoint is named
`conversation_memory_backfill_v2.json` in that path's parent directory.

## Idempotence And Recovery

The checkpoint records completed workspace IDs. Re-running the command with the
same checkpoint skips those workspaces and normally reports `indexed_count: 0`.
OpenSearch writes use stable memory IDs, so retrying an interrupted workspace
replaces already-indexed episodes instead of creating duplicates.

The checkpoint and OpenSearch index are separate stores. If the derived index is
deleted or recreated, retaining the old checkpoint would incorrectly skip
completed workspaces. In that recovery case:

1. Stop any concurrent conversation-memory backfill command.
2. Preserve a copy of the checkpoint for audit and rollback.
3. Remove the active `conversation_memory_backfill_v2.json` checkpoint.
4. Run the command once and verify its final indexed count and logs.

Do not reset the checkpoint for routine reruns.

## Output And Exit Behavior

Successful runs print:

```text
Conversation memory backfill completed.
indexed_count: <number>
```

| Exit behavior | Meaning |
| --- | --- |
| `0` | Backfill completed successfully. |
| `2` | Conversation memory is disabled or its worker is unavailable. |
| Other non-zero | An unexpected persistence, OpenSearch, or embedding failure occurred. |

Failures are not swallowed. The command closes application resources and exits
non-zero so operators and automation can detect incomplete work.