Traces
A trace is the path of a request across services, modelled as a tree of
spans. avuru obs ingests spans from two sources — the eBPF sensor and OTLP
exporters — and stores them together in ClickHouse.
Model
- Span — one operation, with start/end time, status and attributes.
- Parent/child — nesting that forms the waterfall.
- Span links — relate spans across traces (e.g. batch/fan-in).
Explore traces
The trace explorer is live today. From the Traces screen you can:
- Search & filter by service, operation, status, tags (span attributes,
e.g.
http.status_code=500), duration range, and ordering (newest, oldest, slowest). Auxiliary traffic (health checks,/actuator/*, metrics scrapes) is hidden by default. - Latency heatmap — a latency × time histogram; click a band to filter.
- Per-operation overview — RED metrics (rate, errors, P50/P95/P99) per
(service, operation). - Open a trace in a full-window viewer with six views — timeline (waterfall), spans table, flamegraph, statistics, trace graph and JSON — plus a span-detail side panel.
- Compare two traces with a structural diff that highlights added, removed, faster and slower spans.
Did it work? Three answers, not two
A span's outcome is derived, not taken at face value: many auto-instrumentations
leave the OpenTelemetry status Unset even on a failing call, so Avuru Obs
reads the HTTP status alongside it.
| The span says | Outcome |
|---|---|
status Error | error — an explicit verdict always wins |
status Ok | ok — a developer set it, and that is final |
| 5xx | error, whichever side reported it |
| 4xx on a client span | error — the caller is the one that failed |
| 4xx on a server span | refused |
| 3xx, 2xx, no code | ok |
Refused is the request a server turned away: a WAF block, an authorization denial, a rejected payload. It is neither a failure of the service nor a success, and it gets its own amber class rather than being folded into either.
That separation is deliberate. Counting server 4xx as errors would put every
401 auth challenge and every 404 from a crawler into the error rate that
RED metrics, the service map's
health ring and alerting are built on — so refusals
stay out of all three. They are instead visible where you go looking for them:
a Refused column beside Errors in the per-operation overview and the trace
table, the status code shown on each row of the trace list, and a
Refused (4xx) option in the search filter.
If your gateway or WAF is doing its job, that column is where you will see it.
Sampling
Head sampling is configured at the SDK or gateway. Tail-based sampling at the gateway is on the roadmap.
Follow a slow request
Start from the service map or filter Traces by service and duration. Open a trace, select the longest relevant child span and inspect its target, status and attributes. Nested spans overlap in time: adding their durations does not give total request time.
Compare another trace with the same operation before assuming the selected request is typical. Follow its logs when they carry matching trace context. The slow checkout exercise provides a synthetic dataset and expected results for each step.
Missing spans and sampling
Avuru Obs does not fill gaps with invented operations. Check propagation, collector health, project and time window when a trace looks incomplete. Sampling is opt-in; configure it deliberately in your collection pipeline. Spans omitted before ingestion cannot be recovered from the storage backend. See OTLP adoption before changing an existing pipeline.