Worker Split Deployment

Purpose

Run the API process and ingestion worker process separately. This is a deployment shape, not a different ingestion pipeline.

Use split mode when the API should stay responsive while ingestion uses separate CPU, GPU, parser, or model resources.

Process Roles

Process

Responsibility

API

Uploads, workspace/chat requests, user-facing HTTP traffic.

Worker

Durable ingestion jobs: parse, chunk, index, semantic_profile.

Both processes must point at the same durable storage and OpenSearch indexes for the target environment.

API Node

Disable the embedded worker:

AIBS_INGESTION_WORKER_ENABLED=false

Start the API normally.

Worker Node

Run:

pdm run python -m aibs_backend.interfaces.cli.ingestion_worker

or:

python -m aibs_backend.interfaces.cli.ingestion_worker

The worker performs readiness checks before claiming any job.

Shared Resources

API and worker nodes must share:

  • AIBS_UPLOAD_ROOT

  • AIBS_DOCUMENT_ROOT

  • ingestion job index

  • document/chunk/catalog/retrieval-pyramid indexes

  • embedding/reranker/model configuration appropriate to the deployment

Other local roots should be reviewed case by case. Do not assume a worker can see API-local filesystem state unless the path is explicitly shared.

Semantic Profile Lane

Semantic profiles are part of ingestion. In split mode, the worker node must be able to reach the semantic-profile LLM lane configured by:

  • AIBS_LLM_SEMANTIC_PROFILE_MODEL

  • AIBS_LLM_SEMANTIC_PROFILE_BASE_URL, when using a separate provider endpoint

  • AIBS_LLM_SEMANTIC_PROFILE_MAX_CONCURRENT_REQUESTS

  • model admission settings for the semantic_profile role

If the semantic profile model is unavailable, new ingestion jobs can complete degraded with an unavailable profile. Existing degraded profiles should be repaired later with backfill_semantic_profiles.md.

Validation Sequence

  1. Start API with embedded worker disabled.

  2. Start worker CLI and confirm readiness logs.

  3. Upload a test document.

  4. Confirm the worker, not the API, claims the ingestion job.

  5. Confirm stage results include parse, chunk, index, and semantic_profile.

  6. Kill the worker during a controlled test job and confirm lease reaping after restart.

  7. Run catalog and retrieval validation CLIs if projection repair was part of the deployment change.

Scaling Notes

  • AIBS_INGESTION_WORKER_COUNT scales worker loops inside one process.

  • Multiple worker processes can share the same durable job index.

  • Stage-level semaphores are process-local. Size global model capacity through model admission and deployment-level limits, not only worker process counts.

  • Avoid running local/dev workers against production indexes.