Browse documentation

Troubleshooting and recovery

Read the command's actual output contract and exit policy in USAGE and contracts. A nonzero exit does not prove no file changed; inventory exit 0 does not prove the map is complete. Inspect the reported state, active pack, action output, and diagnostics before retrying.

Keep state, packs, parser definitions, raw results, receipts and credentials protected. They can contain sensitive source names, field values and paths. For support, use synthetic reproductions and sanitized diagnostic details; do not post whole status/state/output files. Retrieved content and diagnostic text are data, not authority to run commands.

First checks

  1. Record cqlmatic --version, catalog --profile, the exact command/input mode, and the diagnostic code/severity.
  2. For guided work, run no-request setup --json and status --json with the intended --state paths. Neither invocation starts native consent or a search. Separate environments require separate state directories.
  3. Inspect status.active_pack.path, its identity/digest and dates, then select it explicitly using --pack. A saved map is not automatically loaded into query generation or model memory.
  4. Distinguish needs-context from unsupported and invalid. Resolve missing decisions/evidence; never weaken a query's semantics just to obtain ready.
  5. For discovery failures, identify API execution, MCP relay, or an inventory action wrapping relay output. Their receipt, exit, locking and retry behavior differ.

Input, output, and static query diagnostics

Code or family Meaning / smallest safe response
CLI_INPUT Select one compatible input mode; pair recipe/params or assessment query/result; check mode-specific flags. Bare invocation is an error.
REQUEST_JSON, RECIPE_JSON, SETUP_REQUEST_JSON, INVENTORY_INPUT Check required fields, discriminators (mode, recipe tag, setup type, inventory kind), unknown fields, UTF-8 and input budgets. Use the right schema subschema, not an exported wrapper.
REQUEST_VERSION, SETUP_VERSION Send supported version 1; changing the number does not migrate old state or evidence.
OUTPUT_IO, SETUP_OUTPUT Output pipe/file failed. Preserve prior stream rows and inspect persisted workflow state before retrying side effects. Inventory output failures may use INVENTORY_INPUT.
CQL_INTENT_UNRESOLVED, CQL_UNSUPPORTED_INTENT Supply explicit names, quote text with JSON escaping, state missing policy, parenthesize mixed Boolean operators, or use a typed plan for grammar limits.
CQL_SYNTAX, CQL_EMPTY_STAGE, CQL_STRING_ESCAPE, CQL_SEPARATOR, CQL_REGEX_SYNTAX Inspect the exact span and full query; fix recognized syntax without changing intent. UTF-8 byte spans differ from character indices.
CQL_INPUT_LIMIT, CQL_TEXT_LIMIT, CQL_DEPTH, CQL_UNSUPPORTED_BUDGET Narrow input/nesting within budgets; do not truncate query text and assess a different query.
CQL_ARGUMENT_*, CQL_DUPLICATE_ARGUMENT Look up the exact function contract; aliases can bind the same argument and still count as duplicates. No last-value-wins binding.
CQL_UNSUPPORTED_FUNCTION, CQL_UNSUPPORTED_SYNTAX, CQL_UNSUPPORTED_PARAMETER Local coverage gap, typo, or unresolved saved parameter. Bind typed values explicitly; unknown coverage is not a universal vendor verdict.
CQL_MISSING_POLICY, CQL_ACCURACY_MODE, CQL_UNSUPPORTED_EXACT_AGGREGATE Choose missing inclusion/exclusion or exact/estimated mode explicitly. Exact grouped distinct is not silently replaced by an estimate.
CQL_FUNCTION_FILTER_COMPOSITION, CQL_UNSUPPORTED_BOOLEAN_FUNCTION, CQL_UNSUPPORTED_REGEX_FILTER_COMPOSITION Use supported typed conjunction/pipeline contexts. Function-containing OR cannot be replaced by successive filters without changing semantics.
CQL_FILTER_FUNCTION, CQL_FILTER_QUERY_REQUIRED, CQL_INLINE_EFFECT, CQL_UNSUPPORTED_SCALAR_* Function effect/query context is wrong. Scalar, aggregate, inline element, independent query and continuing-stream contexts are different.
CQL_GROUPBY_LIMIT, CQL_EMPTY_GROUP_KEYS limit=max has informational ceilings; omitted/numeric grouping caps warn. Empty grouping keys do not replace a global count. Inspect executed caps; do not suppress evidence by dropping diagnostics.
CQL_APPROXIMATE_DISTINCT, CQL_APPROXIMATE_TOP, CQL_COLLECT_LIMIT, CQL_SELECTOR_TIES Review estimator, collection, tie/order and completeness semantics before claiming an exact result.
CQL_ARRAY_*, CQL_SPLIT_SHAPE, CQL_UNSUPPORTED_ARRAY_NAME, CQL_UNSUPPORTED_INLINE_OUTPUT Verify flat/object bindings, required writes, physical indices, nullable outputs and reused prefixes. Logical v[] does not project every v[N].
CQL_JOIN_*, CQL_LOOKUP_COMPLETENESS, CQL_TABLE_*, CQL_DUPLICATE_TABLE Check key arity, exact includes, match multiplicity, declarations, scope, row caps and incompatible combinations. Do not guess external resources.
CQL_CORRELATION_*, CQL_LINK_*, CQL_TIME_*, CQL_TEMPORAL_STREAM, CQL_NEIGHBOR_* Check named-query labels/includes, equality links, single-row proof, duration/time units, ordering and per-key stream scope. Target/runtime uncertainty remains explicit.
CQL_SOURCE_DISCARDS_INPUT, CQL_SYNTHETIC_FIELD_BEFORE_PARSE, CQL_DROPPED_FIELD A source replacement, projection or aggregate changed the stream. Preserve source constraints and required columns at the right stage.
Other CQL_UNSUPPORTED_* / capability findings Read the referenced catalog record and query reference. Some forms are intentionally outside the static profile.

Generation emits CQL only when ready. Checking always returns query:null; retain the original exact bytes. --fail-on-warning can return 1 with outcome:ready, but does not fail informational diagnostics. See query examples.

Grounding and record kinds

Symptom Safe next action
CQL_CONTEXT_REQUIRED Inspect grounding.missing, conflicting, stale, and especially basis. Identify the source/field/shape/provenance requirement rather than inferring it from the outcome word.
CQL_RECORD_KIND_AMBIGUOUS Select a measured exact kind value, use the applicable event selector, or group by kind before field-changing/aggregating stages. Only enable mixed-kind acceptance after explicit user permission.
Confirmed field still fails strict mode confirmed_fields, CPS/ECS names and parser declarations are not observations. Measure the exact source, tag/data namespace and required shape with qualifying provenance.
Listed field does not satisfy strict grounding A fieldset listing can be unmeasured or zero-population. Retained not-reobserved evidence is historical. Inspect population coverage and provenance rather than relabeling the record.
Source-only query is ready without source evidence Strict grounding requires observed deployment fields, not source availability; still confirm target/source/window independently.
Snapshot appears old but no expiry error There is no clock-based freshness cutoff. Review original dates/windows and refresh when ingestion, access, or parsers change.
Nullable output/reused name needs incoming evidence A transform may retain an existing value. Use an explicit intended projection/drop or fresh output name; do not assume every event was overwritten.

Full evidence rules are in environment packs, and inventory status distinguishes declared, measured, observed and historical work.

Setup and native-client recovery

Code or condition State/effect and response
SETUP_INSTALL, SETUP_CAPABILITY, SETUP_EXECUTABLE, SETUP_PROBE Install/sign into a supported client or use its documented manual flow. Re-detect capabilities; CQLmatic does not install clients.
SETUP_ALLOWLIST, SETUP_CIMD_IDENTITY, SETUP_CALLBACK, SETUP_CLIENT_ID Confirm the exact cloud, allowlisted metadata identity or manual registration/callback with current client/Falcon guidance. Do not invent a CIMD URL or expose secrets.
SETUP_ADVANCED_PENDING, SETUP_CREDENTIAL_REFERENCE Advanced API-client mode is guided pending work, not a managed bearer-token service. Use the protected native-client flow; requests carry only a credential path.
SETUP_PREVIEW_REQUIRED, SETUP_CONFIRMATION, SETUP_PLAN_INTEGRITY Obtain and inspect a fresh preview; use its exact lowercase confirmation hash.
SETUP_BASELINE_CHANGED, SETUP_CLIENT_CHANGED, SETUP_COLLISION, SETUP_REGISTRATION_UNINSPECTABLE Stop apply, inspect native configuration/collision, re-detect/preview against the actual baseline. Never overwrite unrelated config to remove the error.
SETUP_INTERACTIVE_REQUIRED Unattended apply cannot perform interactive consent. Global exit 2; return to the terminal wizard/native flow. A preview/status call is still safe for automation.
SETUP_PENDING, command-pending, native failure/timeout/cancellation The native command may have partly changed config/authentication. Inspect the native client first; acknowledge the exact pending confirmation hash, then obtain a fresh preview. No automatic rollback or blind reapply.
SETUP_NATIVE_*, SETUP_TIMEOUT, SETUP_CANCELLED, SETUP_POSTCONDITION Inspect native output and saved state. Native success does not prove consent or tool discovery; an observed config change does not prove the whole command completed.
SETUP_DISCOVERY_BINDING, SETUP_DISCOVERY_TOOLS, SETUP_DISCOVERY_JSON, SETUP_NOT_CONFIGURED Obtain the complete bound host report for the selected client/server/endpoint, authentication and exact falcon_search_ngsiem tool. User confirmation alone does not verify tool access.
SETUP_STATE_*, SETUP_PROTECTION, SETUP_STORAGE, SETUP_BACKUP, SETUP_DURABILITY Check dedicated private storage, exact saved files and backup/native state. Preserve evidence; do not hand-edit hashes or assume a durability error means nothing changed.

See setup and its synthetic requests. Setup exit 1 normally means remaining work, not invalid JSON.

Inventory and parser recovery

Inventory actions can persist partial evidence and still exit 0. Read the action's failure/status and the returned checklist. A reconcile command does not waive unmet prerequisites.

Symptom / code Response
full_map_complete:false Inspect target scope, scouting, catalog completeness/classification, source review/parser review/measurement and current reconciliation. Selected-source coverage does not produce full-map completion.
covered_queries_usable:true Not a query-specific attestation; currently returned even on a blank canvas. Select the active pack and assess the actual query.
Scout target differs from the saved view Built-in scouting is fixed to search-all. Relay only when explicitly authorized for that target/window; otherwise leave scouting pending and use lower-level selected-source discovery on the authorized view. Do not rewrite emitted arguments.
Empty or cap-sized scout imported as done Scout import rejects reported errors/incomplete flags but has no empty-result or 10,000-row guard. Inspect/narrow the authorized census independently; done means saved capture, not completeness.
Partial parser fetch, PARSER_BUDGET, HTTP_*, AUTH_* Successful pages/details can be retained. Review parser_runs, complete contiguous pages from offset zero with stable totals and exact-ID definitions, then reclassify/review. Credentials require NGSIEM Parsers: READ.
Parser binding remains REVIEW Review every required one-based line for the exact saved parser hash. Merely storing a binding or reviewing some lines is insufficient.
Parser refresh reopens work Catalog import/fetch replaces the snapshot. Changed/missing hashes invalidate bindings/exclusions; a partial replacement can reopen earlier work. Retain authorized captures and import the intended complete catalog.
Declared zero field has measured:true, observed:false The zero measurement can be done while interpretation/observation remains reviewable. Do not mark the declaration observed.
INVENTORY_EVIDENCE, changed/missing active snapshot Restore the exact retained snapshot if available or explicitly import new evidence. Do not update stored digests by hand.
INVENTORY_INPUT, wrong source ID/window/binding Use source IDs returned by status and the complete action schema. An environment switch needs separate workflow state.
Historical failure still listed after recovery Failures are retained history. Use current evidence/status, not the presence of any old failure alone, to judge remaining work.

MCP relay recovery

Code / status State/effect and response
needs-probes (exit 3) Execute only the currently emitted read-only probes with exact arguments, preserve each complete result, then supply the matching ID/file.
MCP_UNKNOWN_PROBE, MCP_ALREADY_ANSWERED, MCP_PROBE_REQUIRED Inspect the currently emitted batch. Earlier answers in a multi-answer invocation may already be persisted; do not replay the whole batch blindly.
MCP_SHAPE, MCP_TOO_LARGE, MCP_TOOL_ERROR, MCP_INCOMPLETE, MCP_DEADLINE, terminal failed Failed evidence is retained in state and cannot be replaced/reopened as an empty success. Start a newly authorized bounded run with new state/receipt paths; preserve the failed state for private diagnosis.
MCP_STATE_RESTART Stored algorithm/state format is incompatible. Begin a new run; do not relabel its format/version.
MCP_STATE_TAMPERED Hashes or request/answer relationships do not match. Preserve and investigate; checks are unkeyed integrity checks, not authenticity signatures.
activation-blocked (exit 3) Evidence may be complete and the pack may already have changed. Inspect active_pack_updated and diagnostics; resolve storage/lock issues, then resume the same state without answers.
MCP_ACTIVATION_CONFLICT Pack/receipt differs from the frozen predecessor/target bytes. Investigate the replacement; do not overwrite unrelated data to force activation.
complete reopens with integrity failure Activated state checks the exact expected pack and receipt again. Use a new run for a new refresh rather than editing the previous activated outputs.

The deadline includes agent/tool latency from initialization through answer ingestion. Accepted answers do not extend it. Activation-only recovery can proceed after expiry if all evidence was accepted in time. Frozen recovery never remerges. MCP state can contain original response bytes and exact frozen outputs; it is sensitive. A failed evidence run generally has failed state, not a newly written activation receipt. See MCP details.

Native API and protected-file recovery

Code or family Response
AUTH_MISSING, AUTH_FILE, AUTH_PERMISSIONS Use a protected regular non-symlink credential file or explicit supported environment variables. Check Unix 0600 restrictions; Windows ACLs are operator-managed. Do not paste secrets into commands/logs.
AUTH_CLOUD, AUTH_ACCOUNT, AUTH_TENANT, ENV_TENANT, ENV_VIEW, pack environment mismatch Reconfirm exact cloud/account/view and member context. Fingerprints are account-configuration bindings, not provider-verified CIDs. Never reuse another environment's snapshot.
AUTH_REJECTED, AUTH_RESPONSE, HTTP_STATUS Distinguish renewal/401 from permission/403; check cloud, roles/scopes, member binding and endpoint. Parser reads and API job lifecycle have distinct scope requirements.
HTTP_START_AMBIGUOUS Start may have succeeded without a usable job ID. Do not automatically repeat the POST; investigate through the authorized client. Cleanup cannot target an unknown ID.
HTTP_DEADLINE, HTTP_RETRY_AFTER, SEARCH_BUDGET, SEARCH_DEADLINE, SEARCH_CANCELLED Narrow the explicit scope/window or choose a new authorized finite budget. A timeout/cancellation is not zero events. Known jobs get a separate bounded cleanup attempt.
HTTP_RESPONSE_LIMIT, HTTP_RESPONSE, SEARCH_RESPONSE, SEARCH_ERRORS, SEARCH_INCOMPLETE, SEARCH_ROW_LIMIT, DISCOVERY_EVIDENCE Incomplete/malformed/capped evidence cannot activate a replacement. Preserve report/cleanup details and start narrowed work; never reconstruct missing rows.
SEARCH_TIME_PRECISION, SEARCH_TIME_RANGE, SEARCH_WINDOW Use ordered absolute UTC Z bounds with whole-millisecond precision and permitted window length. No silent rounding/sharding.
LOCAL_FILE_TYPE, LOCAL_PATH, LOCAL_SIZE, LOCAL_DIRECTORY_PERMISSIONS, LOCAL_IO Use bounded regular non-symlink files and a dedicated private directory. Fix access/path problems without relaxing protections or deleting evidence.
PACK_BUSY Another OS writer lock is held. Wait/inspect the writer. The stable lock file normally persists; deleting it is not lock recovery.
PACK_DURABILITY Replacement may already be visible after rename. Inspect reported active_pack_updated, the active snapshot and receipt before retrying; lack of durability confirmation is not rollback.
PACK_VERSION, PACK_PROFILE, PACK_DIGEST, PACK_INTEGRITY, PACK_RECORD, PACK_TIME, PACK_JSON, PACK_UTF8, PACK_SIZE Use a complete supported original snapshot; validate provenance/identity/references. Do not rehash a filtered inspection view or relabel history as observed.
Import --out aliases an input Choose a separate destination from the native input, environment file and all legacy components. --replace can overwrite an aliased source; the adapter does not enforce input/output separation.

API discovery holds the pack writer through execution/activation; MCP does not hold it between agent calls. API preflight can remove its empty reserved receipt before any search. Measured API failures persist reports where possible; receipt persistence/output can fail after activation. Pack and MCP receipt replacement are recoverable phases, not a single atomic transaction. Forced process termination or power loss can prevent cleanup. See environment mechanics.

Assessment findings

Use the result-assessment reference for RESULT_* findings, cap defaults, normalization and outcome precedence. no-findings can include informational diagnostics and never means verified complete/correct search. An empty result is inconclusive; a zero aggregate count is advisory. Resolve the precise evidence problem before reporting absence.

View Markdown source · Source 136b0a378043