# Worker Split Deployment

## Purpose

Run the API process and ingestion worker process separately. This is a deployment
shape, not a different ingestion pipeline.

Use split mode when the API should stay responsive while ingestion uses separate
CPU, GPU, parser, or model resources.

## Process Roles

| Process | Responsibility |
| --- | --- |
| API | Uploads, workspace/chat requests, user-facing HTTP traffic. |
| Worker | Durable ingestion jobs: parse, chunk, index, semantic_profile. |

Both processes must point at the same durable storage and OpenSearch indexes for
the target environment.

## API Node

Disable the embedded worker:

```ini
AIBS_INGESTION_WORKER_ENABLED=false
```

Start the API normally.

## Worker Node

Run:

```bash
pdm run python -m aibs_backend.interfaces.cli.ingestion_worker
```

or:

```bash
python -m aibs_backend.interfaces.cli.ingestion_worker
```

The worker performs readiness checks before claiming any job.

## Shared Resources

API and worker nodes must share:

- `AIBS_UPLOAD_ROOT`
- `AIBS_DOCUMENT_ROOT`
- ingestion job index
- document/chunk/catalog/retrieval-pyramid indexes
- embedding/reranker/model configuration appropriate to the deployment

Other local roots should be reviewed case by case. Do not assume a worker can
see API-local filesystem state unless the path is explicitly shared.

## Semantic Profile Lane

Semantic profiles are part of ingestion. In split mode, the worker node must be
able to reach the semantic-profile LLM lane configured by:

- `AIBS_LLM_SEMANTIC_PROFILE_MODEL`
- `AIBS_LLM_SEMANTIC_PROFILE_BASE_URL`, when using a separate provider endpoint
- `AIBS_LLM_SEMANTIC_PROFILE_MAX_CONCURRENT_REQUESTS`
- model admission settings for the `semantic_profile` role

If the semantic profile model is unavailable, new ingestion jobs can complete
degraded with an unavailable profile. Existing degraded profiles should be
repaired later with [backfill_semantic_profiles.md](backfill-semantic-profiles.md).

## Validation Sequence

1. Start API with embedded worker disabled.
2. Start worker CLI and confirm readiness logs.
3. Upload a test document.
4. Confirm the worker, not the API, claims the ingestion job.
5. Confirm stage results include parse, chunk, index, and semantic_profile.
6. Kill the worker during a controlled test job and confirm lease reaping after
   restart.
7. Run catalog and retrieval validation CLIs if projection repair was part of the
   deployment change.

## Scaling Notes

- `AIBS_INGESTION_WORKER_COUNT` scales worker loops inside one process.
- Multiple worker processes can share the same durable job index.
- Stage-level semaphores are process-local. Size global model capacity through
  model admission and deployment-level limits, not only worker process counts.
- Avoid running local/dev workers against production indexes.