Exporting metrics to Grafana and OpenTelemetry

Metrics export pushes your organisation’s ScoutSentinel metrics to your own observability stack, so journey health sits next to the rest of your telemetry. It speaks OTLP over HTTP, which Grafana Cloud, Grafana Alloy, the OpenTelemetry Collector, Datadog, New Relic and Honeycomb all accept.

Metrics export is part of the Enterprise plan. On other plans the card under Integrations, Observability is locked, and POST /v1/integrations answers 422 with the problem code entitlement_exceeded.

How it works

Every minute ScoutSentinel checks which exports are due. For each one it computes a snapshot of your organisation’s metrics, encodes it as an OTLP ExportMetricsServiceRequest and sends one POST to your endpoint. Only your organisation’s data is ever in the payload.

• Interval: 60 seconds by default, from 30 to 3600.
• Timeout: 10 seconds. Redirects are not followed.
• Compression: gzip (Content-Encoding: gzip) unless you turn it off.
• Retries: on 429, 408 and 5xx answers and on network errors the next attempt backs off exponentially, up to one hour, and honours a Retry-After header. Other 4xx answers back off the same way, because they usually mean a credential or URL needs fixing.
• Status: the integration card shows the last export, its HTTP status, the number of data points and a short excerpt of the endpoint’s answer. Partial success answers (rejectedDataPoints) are reported.
• Destinations: HTTPS only, public hosts only. Private, loopback and link-local addresses are refused.

Export failures never affect probes, Events or notifications.

Settings

Temporality

Gauges are last values. Sums and the duration histogram are sent in one of two ways:

• Cumulative (default): each series counts from 00:00 UTC of the current day, with that instant as its start time. The start time is stable all day; the new day is an ordinary counter reset, which rate() and increase() handle.
• Delta: each export covers the time since the last successful export. Datadog’s OTLP metrics intake accepts delta only. New Relic recommends delta for sums and histograms.

Grafana Cloud

1. Sign in to the Grafana Cloud Portal and open your stack.
2. On the OpenTelemetry tile choose Configure. The page shows the OTLP endpoint for your stack, for example https://otlp-gateway-prod-eu-west-0.grafana.net/otlp, and your instance ID. Use the endpoint shown there: the host depends on your region.
3. On the same page generate an access policy token. It needs the metrics:write scope.
4. In ScoutSentinel open Integrations, Observability, Metrics export and choose Grafana Cloud. Paste the endpoint, enter the instance ID and the token, and choose Use as Authorization header. ScoutSentinel builds Authorization: Basic <base64 of instanceId:token> for you.
5. Connect, then choose Send test. The test pushes one data point, scoutsentinel.export.connection_test, and shows the endpoint’s answer.

Grafana Cloud stores OTLP metrics in Prometheus form: dots become underscores, units are appended and monotonic sums end in _total. scoutsentinel.journey.run.duration in ms becomes the scoutsentinel_journey_run_duration_milliseconds histogram. Resource attributes, including your organisation slug, land on the target_info metric rather than on every series.

A first dashboard query, availability per journey:

scoutsentinel_journey_availability_ratio{environment="Production"}

Grafana Alloy or an OpenTelemetry Collector

Choose OpenTelemetry Collector or Grafana Alloy and enter the public HTTPS address of your receiver. ScoutSentinel pushes from the internet, so the receiver must be reachable over HTTPS and should require authentication.

Alloy, receiving on port 4318 with basic authentication and forwarding to any OTLP backend:

otelcol.auth.basic "scoutsentinel" {
  username = "scoutsentinel"
  password = sys.env("SCOUTSENTINEL_PUSH_PASSWORD")
}

otelcol.receiver.otlp "scoutsentinel" {
  http {
    endpoint = "0.0.0.0:4318"
    auth     = otelcol.auth.basic.scoutsentinel.handler
  }

  output {
    metrics = [otelcol.exporter.otlphttp.backend.input]
  }
}

otelcol.exporter.otlphttp "backend" {
  client {
    endpoint = "https://otlp.example.com"
  }
}

The same with the OpenTelemetry Collector:

extensions:
  basicauth/server:
    htpasswd:
      inline: |
        scoutsentinel:${env:SCOUTSENTINEL_PUSH_PASSWORD}

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: basicauth/server

exporters:
  otlphttp:
    endpoint: https://otlp.example.com

service:
  extensions: [basicauth/server]
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlphttp]

Terminate TLS in front of the receiver, or give it a certificate with its tls settings. In ScoutSentinel set the header Authorization to Basic followed by the base64 of scoutsentinel:<password>.

Datadog and New Relic

The presets fill in the header name, the protocol and the temporality.

Metrics

Names are shown with the default prefix. Every export carries the resource attributes service.name (scoutsentinel), scoutsentinel.organisation.slug and scoutsentinel.organisation.name, plus any you add, and the instrumentation scope scoutsentinel.metrics-export. Attributes are limited to the ones in the table, so the number of series stays bounded: at most 500 journeys and the 200 certificates that expire soonest.

Outcome and behaviour stay separate, as everywhere in ScoutSentinel: journey.status never folds in behaviour changes, and journey.behaviour.changed never affects the status.

Troubleshooting