Observability Without Breaking Determinism
Observability is opt-in evidence, not entropy. Everything recorded must keep hashed artifacts stable.
Logging rules
- Scope every log/metric with
run_idandpolicy_profile; includebackendand determinism class (D0/D1) for parity checks. - Generate
event_idfrom a deterministic counter or content hash, never from wall-clock time or randomness. - No timestamps, hostnames, or PIDs in any content that is hashed into bundles. If timestamps are needed, keep them in non-hashed debug channels.
- Redact secrets and user identifiers before writing support bundles; prefer structured JSON logs with explicit field allowlists.
- When debug mode is enabled (
HELIX_DEBUG=1or Studio “Debug traces”), automatically downgrade trust/taint todebugand mark outputs as non-deterministic.
Metrics
- Use counters/gauges keyed by
run_idandcomponent; avoid high-cardinality labels derived from inputs. - Emit metrics to sidecar files excluded from manifests (
observability/*.json) or to stdout; never embed metrics inside hashed artifacts. - Record duration via logical steps (e.g., event counts) instead of wall-clock timers when determinism matters; if timers are needed, keep them out of the hash boundary.
Tracing
- Deterministic traces: ordered events with stable IDs; omit timestamps and thread IDs.
- Debug traces: may include timestamps/stack traces but must be fenced off (debug taint) and excluded from artifact hashing.
- LiveGraph/streaming: attach monotonic sequence numbers; enforce backpressure by pausing emitters before spilling to disk.
Storage and export
- Support bundles must include: run config, manifest, policy profile, deterministic event log, and redaction log. Optional debug traces live under
debug/and are clearly marked non-deterministic. - Exporters must ensure manifest hashes exclude
observability/anddebug/payloads.
Failure handling
- If logging or metrics sinks block, fail the run with a clear
observability_backpressureerror rather than dropping events. - If a tool tries to record a forbidden field (timestamp/hostname), surface a validation error and keep the bundle unchanged.