Skip to main content

Errors

Errors are already in your telemetry — buried in span exception events, error spans and ERROR/FATAL logs. The error-tracking module surfaces them as first-class issues: the same exception, seen 4 000 times since the 14:02 deploy, with a stack trace and a link to a trace that hit it.

Issues​

An issue is a group of occurrences that share a fingerprint — service

  • exception type + the normalized top stack frames (falling back to the message when there's no stack).

Normalization is what makes the group hold. Before hashing, avuru obs collapses the parts of the text that change on every occurrence: timestamps, UUIDs, hex identifiers such as trace and span ids, memory addresses, and any remaining run of digits — line numbers among them. So the fingerprint stays stable across deploys, and an error that carries the request's own trace id in its message still groups as one issue instead of one issue per request. The issue's title keeps the raw line, ids included, so it remains something you can search your logs for.

Each issue carries:

  • A stack trace — the crash site on top, following OTel exception.* semantic conventions.
  • An occurrence timeline — first seen, last seen, total count, and a histogram of occurrences over the window.
  • A link to the originating trace — pivot from the exception straight to the request that produced it (and from there to its logs and spans).
  • A source — whether the occurrence came from a span event, an error span, a log, or a Sentry SDK.

Two ways in​

Neither path needs an application change.

  1. Derived, in-database (default). ClickHouse materialized views turn the OTLP you already send — span exception events, error spans, and ERROR/ FATAL logs — into issues at insert time. Zero instrumentation: if a service already reaches avuru obs, its exceptions already group into issues.
  2. Sentry-SDK ingest (opt-in). The gateway runs a Sentry-protocol receiver on :4319. Point any Sentry SDK at avuru obs by changing its DSN — no rewrite, no new agent. This is the one signal eBPF can't reach: browser JavaScript errors. See the Sentry SDK integration.

Sentry events are stored as OTel log records, so the same logs materialized view that derives issues from your ERROR logs derives them from Sentry events too — and they also show up in the Logs explorer.

Reading the list​

Above the list sits a stats band that answers what the filtered set amounts to, over the same window and filters as the rows beneath it:

  • Issues — how many match right now.
  • New in window — of those, how many were first seen inside the window rather than merely seen again. This is the number that catches a bad deploy.
  • Regressed — how many were resolved and have recorded an occurrence since.
  • Events — total occurrences those issues produced in the window, with a histogram showing when they happened.
  • Top services — the busiest services by occurrences. Click one to filter the list to it; click it again to clear the filter.

The band and the list read the same issue set through the same query, so they cannot disagree: resolve an issue and it leaves both at once; a recurrence brings it back to both, flagged as a regression.

The list loads up to 200 issues and scrolls in place, with the tabs, filters and band staying put above it. If a filter matches more than 200, the list says so and names the real total — narrow by service, status or search text to see the rest.

Triage lifecycle​

Every issue has a status:

  • Unresolved — the default; something to look at.
  • Resolved — fixed; drops out of the active list.
  • Ignored — known and muted; won't resurface.

Regression detection is computed at read time: a resolved issue that records an occurrence after it was resolved is flagged as a regression — the bug you closed last week is back.

Use cases​

  • Catch a bad deploy. One new fingerprint whose first-seen lines up with the 14:02 rollout is the answer — no log spelunking. The Catch a bad deploy guide walks it end to end on the bundled sandbox.
  • Browser errors without new tooling. Your frontend already runs a Sentry SDK? Change its DSN and browser exceptions land next to the backend traces that caused them. See the Sentry SDK integration.
  • Regression watch. Resolve an issue and forget it — if the bug ships again, the issue flags itself as a regression instead of drowning in a new ticket.
  • Triage services you never instrumented. Issues derive from the telemetry eBPF and OTLP already deliver, so even SDK-less services get grouped, triageable errors.

Configuration​

  • modules.errorTracking.enabled — default on. Derivation is free value from data you already send, so it's enabled out of the box. Turn it off and the tables, the /errors API and the sidebar entry all disappear.
  • gateway.sentry.enabled — default off. The Sentry ingest port opens a network surface (:4319), so it's opt-in. It also needs the logs module (modules.logs.enabled), because Sentry events are stored as log records. Browser clients additionally need :4319 exposed and CORS configured for their origins.
  • AVURUOBS_RETENTION_ERRORS_DAYS — how long occurrences are kept, default 30 days.

v1 limitations​

  • An error may appear twice. One logical error that is both logged and recorded as a span exception event produces two issues — different sources, different fingerprints. Read-time trace-correlation dedup is future work.
  • No backfill. Derivation runs on inserts after the migration; errors from before the module was enabled aren't reconstructed.
  • Dangling trace links. Errors are retained longer than traces by default, so an old issue can link to a trace that has already expired — the UI renders a "trace expired" state.

:::note This page is expanding Alerting on new and spiking issues, source-map upload (deminified JS frames) and per-project DSN keys are on the roadmap. See the Roadmap and Feature status. :::