Troubleshooting

This chapter collects the failure modes documented across the rest of the manual and organises them by symptom. Each entry points at the page that explains the mechanism in full.

Two things are worth knowing before you start.

The platform fails loudly on purpose. Startup aborts when an enabled capability is unreachable, model placement denies work it cannot prove is safe, and the repair tools report failures instead of writing a degraded replacement. A component that refuses to start is usually telling you the truth about its dependencies.

degraded is not the same as broken. A document in that state is searchable and carries known quality loss. See Document lifecycle.

Startup Failures

The FastAPI lifespan hook runs a fixed set of checks before serving. If any enabled check fails, startup fails. That is deliberate: advertising healthy startup while a declared capability is unreachable would be misleading.

Check	Runs when	Look at
Document repository integrity	always, and first	the JSON document store under the configured data root
OpenSearch connectivity	always	endpoint, credentials, network path to OpenSearch
Document catalog index readiness	always	Rebuild document catalog
Retrieval pyramid index readiness	always	Rebuild retrieval pyramid
Embedding model readiness	always	model path under `models/embeddings`
Reranker readiness	`AIBS_RERANKER_ENABLED=true`	model path under `models/reranking`
Document parser warmup	`AIBS_INGESTION_WORKER_ENABLED=true`	`AIBS_DOCLING_ARTIFACTS_PATH` when local artifacts are configured
LLM readiness	`AIBS_LLM_ENABLED=true` and `AIBS_LLM_STARTUP_CHECK=true`	LLM endpoint reachability
Model residency control plane	when a control plane is configured	Model residency monitor. A deployment without a readiness gate is reported rather than failed.

Before starting the API, confirm the basics:

OpenSearch answers.
The embedding model path exists.
The reranker model path exists, if reranking is enabled.
AIBS_DOCLING_ARTIFACTS_PATH exists, if local Docling artifacts are configured.
Device values are auto, unless the machine is known to have a CUDA or ROCm enabled torch runtime.
The LLM endpoint answers, if AIBS_LLM_ENABLED=true.

A standalone ingestion worker runs its own copy of these checks, excluding the LLM check unless abstractive summaries are enabled. It also checks the document catalog index, because ingestion writes document-state changes through that projection.

Full settings reference: Configuration.

Startup Succeeds But The Deployment Is Wrong

One combination passes the checks and is still incorrect: AIBS_MODEL_CLUSTER_ENABLED=true together with AIBS_COORDINATION_BACKEND=memory. The memory adapter is process-local, so this pair advertises global capacity control while every API process keeps a private view of the cluster. Use the Valkey backend for any multi-process deployment. See Model coordination.

Login And Access Problems

Authentication and authorisation are separate gates. Valid credentials are not enough to enter the product, and product administration does not grant workspace access.

Symptom	Likely cause
Credentials work in Energy Logserver but Compass rejects them	The account lacks the `aibs-access` entitlement.
The user gets in but sees no workspaces	Workspace visibility comes from workspace membership. An `aibs-admin` does not automatically see every workspace.
Administration routes return 403 for an administrator	Product administration needs both `aibs-access` and `aibs-admin`. Either alone is not enough.
Nobody can administer anything on a fresh deployment	The first administrator has to be bootstrapped before first non-dev login. See Bootstrap admin.
Sessions drop unpredictably	Sessions currently use a fixed TTL and process-local memory, so a single-worker API deployment is required until a persistent session store lands.
Requests authenticate through a header instead of a cookie	The legacy `X-AIBS-User-Id` header is accepted only when `AIBS_AUTH_DEV_MODE=true`. It does not belong in production.
A workspace member cannot be added by username	Only directory-backed users resolve by username. Free-form text no longer creates a membership.
An owner cannot be removed	Only an owner may demote or remove another owner, and the last enabled administrator cannot be removed or disabled.

The bootstrap CLI reports its outcome through exit codes:

Code	Meaning
`0`	Created, upgraded, or confirmed unchanged.
`2`	Input validation or user-directory provider failure.

Mechanism and rules: Authentication and RBAC.

Document Ingestion Problems

Ingestion runs as a durable job queue in OpenSearch with optimistic-concurrency claiming, lease heartbeats, retry scheduling and stale lease reaping. Several workers can compete for jobs without double execution.

Symptom	Where to look
Uploaded documents never leave `registered`	Nothing is claiming jobs. Either `AIBS_INGESTION_WORKER_ENABLED=false` with no standalone worker running, or the worker cannot reach OpenSearch. See Worker split.
Jobs are claimed and then go quiet	Loop-level exceptions are logged and retried after `AIBS_INGESTION_WORKER_ERROR_SLEEP_SECONDS`. A repeating cycle points at a stage failure, not at the queue.
Documents finish as `degraded` rather than `ready`	Expected when quality signals fire: parser quality, chunk partiality, embedding or indexing warnings, or fewer indexed chunks than expected.
Many documents carry unavailable semantic profiles	Live ingestion retries semantic enrichment, then persists an unavailable profile so the durable job can complete degraded. Repair with Backfill semantic profiles.
A repeated upload is rejected	Duplicate upload rejection is intended behaviour.
A document needs reprocessing	The aggregate supports `ready` or `degraded` back to `registered` for reparse and to `parsed` for rechunk, but the HTTP surface exposes no dedicated reparse or rechunk command yet.
Document operations fail inside an archived workspace	Workspace archive blocks new active document operations by design.

Worker behaviour and controls: Ingestion worker.

Reading The Backfill Report

The semantic profile backfill refuses to hide problems. It reports failed documents instead of saving an unavailable replacement, because a repair tool that quietly writes bad data is worse than one that stops.

Status	Meaning
`skipped_existing`	A current profile exists and `--force` was not set.
`skipped_missing_artifacts`	The parsed document or chunk set is missing.
`skipped_terminal`	The document is failed or deleted.
`failed`	Enrichment failed after retries.

A healthy run reports failed=0, and every skipped_missing_artifacts should be explained by genuinely missing parsed or chunked state.

Retrieval And Projection Problems

Document JSON stays the aggregate source of truth. Catalog and pyramid indexes are projections over it, so they can drift.

Symptom	Action
Workspace source lists are empty or incomplete after enabling the catalog	Backfill the projection for existing workspaces with Rebuild document catalog.
Search misses documents that clearly exist	Run the rebuild in validation mode first and compare expected, projected, matched, missing and stale counts.
Results are wrong after a mapping or schema change	Run a full rebuild without `--workspace`. Schema upgrades need a recreate, not a partial pass.
Broad discovery returns nothing	Retrieval pyramid discovery is read only when `AIBS_RETRIEVAL_PYRAMID_DISCOVERY_ENABLED=true`.
A pyramid rebuild reports missing semantic profiles	Run the semantic profile backfill first, then rebuild. The order matters.
The rebuild rejects the command line	Some combinations are refused on purpose: `--recreate` with `--only-missing`, `--only-missing` with `--profile-version`, and `--profile-version` at or below zero.

Use --only-missing for outage repair when the schema is current and stale rows are not the problem. Use --profile-version after chunk policy changes, when a full recreate is not warranted.

Answering Problems

Symptom	Likely cause
Answers slow down under concurrent use	LLM access is serialised through a concurrency gate, one in-flight request by default, matching single-lane local serving.
Answers arrive without citations	Check which evidence regime handled the question. Creative and Grounded bind evidence differently. See Answering.
Chat fails while everything else works	The LLM endpoint is optional and environment-driven. With `AIBS_LLM_ENABLED=false`, answer generation is not wired at all.
A question about a named document drifts to another one	Document binding is under active hardening. Confirm the document is `ready` or `degraded` and that its catalog projection is current.

Model Cluster Problems

Valkey-backed model placement is fail-closed. If the scheduler cannot prove that a node is fresh, has capacity and holds an atomic lease, it will not place work there. Denied work is the safety mechanism working, not a bug.

Symptom	Meaning
Work is queued or denied instead of dispatched	A lease-protected path could not reach Valkey, or a node could not be proven fresh.
Cache-backed paths degrade but keep working	Best-effort cache paths treat a Valkey failure as a miss. The asymmetry with lease-protected paths is intentional.
The scheduler never reports ready	Run Model capacity audit until it reports the scheduler as ready.
Residency state looks stale, or ownership flaps	See Model residency monitor. Run at least two supervised monitor candidates on independent always-on hosts.

Do not set AIBS_MODEL_CLUSTER_ENABLED=true before the enablement gates are green. The sequence is in Model coordination.

Reading API Errors

The API maps internal errors into a stable JSON shape:

{
  "error": "Error message description",
  "code": "stable_error_code"
}

Status	Meaning
`400`	Validation error.
`403`	Authorization failure. Check entitlements and workspace membership.
`404`	Missing entity.
`409`	Duplicate, or an invalid state transition.
`422`	Invariant violation.
`502`	External service error. OpenSearch or the LLM endpoint is the usual source.
`500`	Persistence or unexpected backend failure.

Full reference: HTTP API.

Not Covered Yet

Two areas belong in this chapter and cannot be written from current material.

Log locations and what to look for. No page documents where each component writes its logs or which messages matter. This needs input from whoever operates a deployment.

Latency diagnosis. Per-stage ingestion, retrieval and answer latency reporting, degraded artifact reports, projection drift reports and query execution traces are planned rather than available. Until they land, slow broad questions on large workspaces can only be approached through the catalog and profile filtering paths described in Answering.