Skip to main content

Service health

Forty green tiles don't answer the 3 a.m. question: is anything my users care about actually broken? The Service Health board does. It rolls each service's RED health up into consolidated group status, lays the groups out by criticality tier, and enforces one rule dashboards usually leave to the reader: a service can't read green while a critical dependency is red.

It is derived from the traces you already send: no new storage, and nothing to instrument. On its own, the hub originates no traffic to your services at all — endpoint checks are the one deliberate exception, and only when you declare them.

From RED to status​

Each service gets one of four states, computed from its RED data over the selected window:

  • healthy — inside its error budget and latency objective.
  • degraded — over the warning error rate, or past its p95 objective.
  • down — over the critical error rate.
  • idle — not enough traffic to judge. A traffic gate runs first, so a quiet service reads idle, never down — no 3 a.m. page because a batch job went to sleep.

Every status carries a plain-language reason, not just a color: error rate 4.2% ≥ 1% budget, or p95 780ms ≥ 500ms objective. Thresholds resolve by precedence — service overrides > tier overrides > defaults — so you can hold T0 to 0.5% while the long tail lives with the built-ins.

Groups and tiers​

Grouping is hybrid, so the board is useful before you configure anything:

  • Named groups come from config: a selector matches services by name or by Kubernetes namespace, and assigns a criticality tier (T0/T1/T2).
  • Everything else auto-groups by namespace at the default tier. Services with no namespace label pool into an (unlabeled) group until config claims them.

The /health board lays groups out in tier lanes, most critical first — so the on-call glance starts where the blast radius is largest.

Dependency propagation​

Your checkout service can be perfectly healthy and completely useless, because payments is down. Service Health encodes that:

  • Dependencies come from the service map you already have — no extra declaration.
  • An edge is critical when its target is a T0 service (or when you force it with criticalEdges).
  • A critical dependency going down drags its dependents to at least degraded. Each member reports both its base status and its effective status, so the detail panel can say exactly that: healthy on its own, degraded because payments is down.

Propagation is one hop, cycle-safe, and down-only — deliberate v1 boundaries (see limitations below).

Use cases​

  • The on-call glance. One screen, tier lanes, worst first. If the T0 lane is green, go back to bed — and if it isn't, the reason string tells you why before you've opened a single dashboard.
  • Encode the golden rule instead of re-deriving it. "Checkout depends on payments" lives in the topology, not in tribal knowledge. New team members see the dependency chain in the member detail panel.
  • Re-tier mid-incident. Promote a suddenly-critical service to T0 in Settings → Groups and the next health read uses it — no restart, no redeploy, no PR. GitOps installs can keep doing the same from chart values.
  • Drive a status page. GET /api/v1/health/groups returns the whole board as JSON — overall status, per-group rollups, reasons. Feed it to an external status page or a wallboard.

Configuration​

The module is on by default (modules.serviceHealth.enabled) — it's free value from data you already collect. Turning it off removes the API routes and the sidebar entry, nothing else, because it owns no storage.

Groups, from the UI​

Settings → Groups is where most people should define groups: a name, a criticality tier and the namespaces or services it covers. Saving applies to the next health read. Writes need the admin role; anyone who can see the board can see the definitions behind it.

Services no group claims still fall into their namespace at defaultTier, so an install that configures nothing keeps the view it has today.

Groups, from chart values​

Grouping also lives in the serviceGroups Helm block, rendered to a ConfigMap the hub reads (mounted at AVURUOBS_GROUPS_CONFIG) and hot-reloads within ~15 s of a kubectl edit — the right primitive for a GitOps install:

serviceGroups:
  defaultTier: T2            # tier for services no group claims
  groups:
    - name: payments
      tier: T0
      selector:
        namespaces: [payments]
        services: []
  # SLO-lite thresholds. Precedence: services > tiers > defaults > built-in.
  thresholds:
    defaults:
      errorRateWarn: 0.01
      errorRateCrit: 0.05
      latencyP95ObjectiveMs: 500
      minSampleCount: 5
    tiers:
      T0: { errorRateWarn: 0.005, errorRateCrit: 0.02, latencyP95ObjectiveMs: 300 }
    services:
      payments: { latencyP95ObjectiveMs: 800 }
  # Force specific edges critical even when the target isn't T0 (rare).
  criticalEdges: []
  #  - { from: web, to: legacy-billing }

Validation is fail-loud: a bad tier or an empty selector is rejected at startup (and a bad tier already fails helm template against the chart's values schema). The UI is held to the same rules, so nothing you can save there will fail the hub's next boot.

Thresholds, criticalEdges and defaultTier are chart-only for now — they sit in the same object but are a sharper edge than group membership.

When both define the same name​

The chart wins. An install that manages its groups in Git must not have them overridden from a browser, so chart-declared groups render read-only and an attempt to create or edit one is refused rather than stored — the conflict shows up when you try to save it, not at the next helm upgrade. If chart values later claim a name you had already authored, that group is flagged as overridden in the editor instead of quietly ceasing to group anything, so you can move it or remove it.

Both sources are merged in a single resolver shared by the health API and the alert evaluator, so the grouping alerts fire on is always the grouping the board shows.

API​

  • GET /api/v1/health/groups?start&end — the whole board: overall status, per-group status with rollup reason and aggregate RED, and members with base vs. effective status and their dependencies.
  • GET /api/v1/health/groups/{name} — one group, for drill-downs.

See the API reference.

v1 limitations​

  • Propagation is one hop and down-only. A degraded dependency doesn't propagate, and transitive chains (A→B→C) don't cascade past the first hop.
  • Group latency is the worst member's p95, not a true group-level quantile.
  • Unlabeled services pool together in (unlabeled) until a group names them.
  • Groups are instance-wide, not per-project.

:::note This page is expanding Service health is where alerting gets its signal — a rule can page you when a group, service, or tier goes bad. See the changelog entry, the Roadmap and Feature status. :::

Endpoint checks: health when nothing is calling​

Everything above is derived from requests other people made. That leaves one question it structurally cannot answer: a group with no traffic in the window is either idle or dead, and no amount of observed telemetry tells you which. At 3 a.m. that is the only question that matters.

An endpoint check is a scheduled HTTP probe attached to a group — the request we make, so that silence stops being ambiguous:

serviceGroups:
  groups:
    - name: core
      tier: T0
      selector:
        namespaces: [storefront]
      checks:
        - id: core-login
          url: https://app.example.com/api/health
          interval: 60s
          expect:
            status: 200
            maxLatency: 800ms
  • A silent group whose probe passes reads healthy — not idle. Something is serving; we asked.
  • A silent group whose probe fails reads down. Nobody else is calling it, and the one thing that did was not served.
  • A group with live traffic and a failing probe is at least degraded: the traffic says something works, the probe says something does not.

Two failures, never one​

A check does not move a group until it has failed twice in a row. A single failed probe is a rolling restart, a lost lease or a dropped packet — and a board that reacts to one is a board people stop reading. The first failure is still visible on the group's card, marked apart from a check that is actually failing.

A check is traffic, not a side channel​

Each probe emits a span of its own. That means a check appears in your RED metrics, on the service map and in the trace explorer exactly like any other client — and a failing check on this board links straight to the trace of the request that failed, instead of leaving you to correlate by hand.

Check spans are classified as auxiliary traffic by the same mechanism that already keeps health-check spans out of user-facing RED, so your numbers do not move because you started probing.

The hub sends those spans to the gateway as an ordinary OTLP client, past the same receiver and the same ingest-key enforcement as everything else. It never writes to storage directly.

:::info Zero checks, zero cost Declare none and nothing runs: no scheduler, no rows, no behaviour change. :::

Reading them​

GET /api/v1/checks lists what is declared; GET /api/v1/checks/{id}/results returns one check's recent outcomes with the trace id of each probe.

What checks are not​

  • Not a replacement for Kubernetes probes. Liveness and readiness serve the orchestrator; these serve humans, and they run from outside the pod.
  • Not scripted journeys. Single requests only — login-then-navigate flows are a larger question.
  • Not guarded against internal targets, on purpose. A check exists to probe your own services, on exactly the private networks an SSRF guard would block. The URLs are configuration you write, never user input; what is enforced is a hard timeout and a refusal to follow a redirect to another host.