# Troubleshooting

This chapter collects the failure modes documented across the rest of the manual
and organises them by symptom. Each entry points at the page that explains the
mechanism in full.

Two things are worth knowing before you start.

The platform fails loudly on purpose. Startup aborts when an enabled capability
is unreachable, model placement denies work it cannot prove is safe, and the
repair tools report failures instead of writing a degraded replacement. A
component that refuses to start is usually telling you the truth about its
dependencies.

`degraded` is not the same as broken. A document in that state is searchable and
carries known quality loss. See
[Document lifecycle](../03-0-0-Architecture/document-lifecycle.md).

## Startup Failures

The FastAPI lifespan hook runs a fixed set of checks before serving. If any
enabled check fails, startup fails. That is deliberate: advertising healthy
startup while a declared capability is unreachable would be misleading.

| Check | Runs when | Look at |
| --- | --- | --- |
| Document repository integrity | always, and first | the JSON document store under the configured data root |
| OpenSearch connectivity | always | endpoint, credentials, network path to OpenSearch |
| Document catalog index readiness | always | [Rebuild document catalog](../06-0-0-Operations/rebuild-document-catalog.md) |
| Retrieval pyramid index readiness | always | [Rebuild retrieval pyramid](../06-0-0-Operations/rebuild-retrieval-pyramid.md) |
| Embedding model readiness | always | model path under `models/embeddings` |
| Reranker readiness | `AIBS_RERANKER_ENABLED=true` | model path under `models/reranking` |
| Document parser warmup | `AIBS_INGESTION_WORKER_ENABLED=true` | `AIBS_DOCLING_ARTIFACTS_PATH` when local artifacts are configured |
| LLM readiness | `AIBS_LLM_ENABLED=true` and `AIBS_LLM_STARTUP_CHECK=true` | LLM endpoint reachability |
| Model residency control plane | when a control plane is configured | [Model residency monitor](../06-0-0-Operations/model-residency-monitor.md). A deployment without a readiness gate is reported rather than failed. |

Before starting the API, confirm the basics:

- OpenSearch answers.
- The embedding model path exists.
- The reranker model path exists, if reranking is enabled.
- `AIBS_DOCLING_ARTIFACTS_PATH` exists, if local Docling artifacts are configured.
- Device values are `auto`, unless the machine is known to have a CUDA or ROCm
  enabled torch runtime.
- The LLM endpoint answers, if `AIBS_LLM_ENABLED=true`.

A standalone ingestion worker runs its own copy of these checks, excluding the
LLM check unless abstractive summaries are enabled. It also checks the document
catalog index, because ingestion writes document-state changes through that
projection.

Full settings reference:
[Configuration](../04-0-0-Configuration/environment-settings.md).

### Startup Succeeds But The Deployment Is Wrong

One combination passes the checks and is still incorrect:
`AIBS_MODEL_CLUSTER_ENABLED=true` together with
`AIBS_COORDINATION_BACKEND=memory`. The memory adapter is process-local, so this
pair advertises global capacity control while every API process keeps a private
view of the cluster. Use the Valkey backend for any multi-process deployment.
See [Model coordination](../03-0-0-Architecture/model-coordination.md).

## Login And Access Problems

Authentication and authorisation are separate gates. Valid credentials are not
enough to enter the product, and product administration does not grant workspace
access.

| Symptom | Likely cause |
| --- | --- |
| Credentials work in Energy Logserver but Compass rejects them | The account lacks the `aibs-access` entitlement. |
| The user gets in but sees no workspaces | Workspace visibility comes from workspace membership. An `aibs-admin` does not automatically see every workspace. |
| Administration routes return 403 for an administrator | Product administration needs both `aibs-access` and `aibs-admin`. Either alone is not enough. |
| Nobody can administer anything on a fresh deployment | The first administrator has to be bootstrapped before first non-dev login. See [Bootstrap admin](../06-0-0-Operations/bootstrap-admin.md). |
| Sessions drop unpredictably | Sessions currently use a fixed TTL and process-local memory, so a single-worker API deployment is required until a persistent session store lands. |
| Requests authenticate through a header instead of a cookie | The legacy `X-AIBS-User-Id` header is accepted only when `AIBS_AUTH_DEV_MODE=true`. It does not belong in production. |
| A workspace member cannot be added by username | Only directory-backed users resolve by username. Free-form text no longer creates a membership. |
| An owner cannot be removed | Only an owner may demote or remove another owner, and the last enabled administrator cannot be removed or disabled. |

The bootstrap CLI reports its outcome through exit codes:

| Code | Meaning |
| --- | --- |
| `0` | Created, upgraded, or confirmed unchanged. |
| `2` | Input validation or user-directory provider failure. |

Mechanism and rules:
[Authentication and RBAC](../08-0-0-Security/auth-rbac.md).

## Document Ingestion Problems

Ingestion runs as a durable job queue in OpenSearch with optimistic-concurrency
claiming, lease heartbeats, retry scheduling and stale lease reaping. Several
workers can compete for jobs without double execution.

| Symptom | Where to look |
| --- | --- |
| Uploaded documents never leave `registered` | Nothing is claiming jobs. Either `AIBS_INGESTION_WORKER_ENABLED=false` with no standalone worker running, or the worker cannot reach OpenSearch. See [Worker split](../06-0-0-Operations/worker-split.md). |
| Jobs are claimed and then go quiet | Loop-level exceptions are logged and retried after `AIBS_INGESTION_WORKER_ERROR_SLEEP_SECONDS`. A repeating cycle points at a stage failure, not at the queue. |
| Documents finish as `degraded` rather than `ready` | Expected when quality signals fire: parser quality, chunk partiality, embedding or indexing warnings, or fewer indexed chunks than expected. |
| Many documents carry unavailable semantic profiles | Live ingestion retries semantic enrichment, then persists an unavailable profile so the durable job can complete degraded. Repair with [Backfill semantic profiles](../06-0-0-Operations/backfill-semantic-profiles.md). |
| A repeated upload is rejected | Duplicate upload rejection is intended behaviour. |
| A document needs reprocessing | The aggregate supports `ready` or `degraded` back to `registered` for reparse and to `parsed` for rechunk, but the HTTP surface exposes no dedicated reparse or rechunk command yet. |
| Document operations fail inside an archived workspace | Workspace archive blocks new active document operations by design. |

Worker behaviour and controls:
[Ingestion worker](../06-0-0-Operations/ingestion-worker.md).

### Reading The Backfill Report

The semantic profile backfill refuses to hide problems. It reports failed
documents instead of saving an unavailable replacement, because a repair tool
that quietly writes bad data is worse than one that stops.

| Status | Meaning |
| --- | --- |
| `skipped_existing` | A current profile exists and `--force` was not set. |
| `skipped_missing_artifacts` | The parsed document or chunk set is missing. |
| `skipped_terminal` | The document is failed or deleted. |
| `failed` | Enrichment failed after retries. |

A healthy run reports `failed=0`, and every `skipped_missing_artifacts` should be
explained by genuinely missing parsed or chunked state.

## Retrieval And Projection Problems

Document JSON stays the aggregate source of truth. Catalog and pyramid indexes
are projections over it, so they can drift.

| Symptom | Action |
| --- | --- |
| Workspace source lists are empty or incomplete after enabling the catalog | Backfill the projection for existing workspaces with [Rebuild document catalog](../06-0-0-Operations/rebuild-document-catalog.md). |
| Search misses documents that clearly exist | Run the rebuild in validation mode first and compare expected, projected, matched, missing and stale counts. |
| Results are wrong after a mapping or schema change | Run a full rebuild without `--workspace`. Schema upgrades need a recreate, not a partial pass. |
| Broad discovery returns nothing | Retrieval pyramid discovery is read only when `AIBS_RETRIEVAL_PYRAMID_DISCOVERY_ENABLED=true`. |
| A pyramid rebuild reports missing semantic profiles | Run the semantic profile backfill first, then rebuild. The order matters. |
| The rebuild rejects the command line | Some combinations are refused on purpose: `--recreate` with `--only-missing`, `--only-missing` with `--profile-version`, and `--profile-version` at or below zero. |

Use `--only-missing` for outage repair when the schema is current and stale rows
are not the problem. Use `--profile-version` after chunk policy changes, when a
full recreate is not warranted.

## Answering Problems

| Symptom | Likely cause |
| --- | --- |
| Answers slow down under concurrent use | LLM access is serialised through a concurrency gate, one in-flight request by default, matching single-lane local serving. |
| Answers arrive without citations | Check which evidence regime handled the question. Creative and Grounded bind evidence differently. See [Answering](../03-0-0-Architecture/answering.md). |
| Chat fails while everything else works | The LLM endpoint is optional and environment-driven. With `AIBS_LLM_ENABLED=false`, answer generation is not wired at all. |
| A question about a named document drifts to another one | Document binding is under active hardening. Confirm the document is `ready` or `degraded` and that its catalog projection is current. |

## Model Cluster Problems

Valkey-backed model placement is fail-closed. If the scheduler cannot prove that
a node is fresh, has capacity and holds an atomic lease, it will not place work
there. Denied work is the safety mechanism working, not a bug.

| Symptom | Meaning |
| --- | --- |
| Work is queued or denied instead of dispatched | A lease-protected path could not reach Valkey, or a node could not be proven fresh. |
| Cache-backed paths degrade but keep working | Best-effort cache paths treat a Valkey failure as a miss. The asymmetry with lease-protected paths is intentional. |
| The scheduler never reports ready | Run [Model capacity audit](../06-0-0-Operations/model-capacity-audit.md) until it reports the scheduler as ready. |
| Residency state looks stale, or ownership flaps | See [Model residency monitor](../06-0-0-Operations/model-residency-monitor.md). Run at least two supervised monitor candidates on independent always-on hosts. |

Do not set `AIBS_MODEL_CLUSTER_ENABLED=true` before the enablement gates are
green. The sequence is in
[Model coordination](../03-0-0-Architecture/model-coordination.md).

## Reading API Errors

The API maps internal errors into a stable JSON shape:

```json
{
  "error": "Error message description",
  "code": "stable_error_code"
}
```

| Status | Meaning |
| --- | --- |
| `400` | Validation error. |
| `403` | Authorization failure. Check entitlements and workspace membership. |
| `404` | Missing entity. |
| `409` | Duplicate, or an invalid state transition. |
| `422` | Invariant violation. |
| `502` | External service error. OpenSearch or the LLM endpoint is the usual source. |
| `500` | Persistence or unexpected backend failure. |

Full reference: [HTTP API](../09-0-0-API/http-api.md).

## Not Covered Yet

Two areas belong in this chapter and cannot be written from current material.

**Log locations and what to look for.** No page documents where each component
writes its logs or which messages matter. This needs input from whoever operates
a deployment.

**Latency diagnosis.** Per-stage ingestion, retrieval and answer latency
reporting, degraded artifact reports, projection drift reports and query
execution traces are planned rather than available. Until they land, slow broad
questions on large workspaces can only be approached through the catalog and
profile filtering paths described in
[Answering](../03-0-0-Architecture/answering.md).