Skip to main content

Observability stack

Infrahub ships an observability stack that runs alongside Infrahub:

  • Grafana Alloy — collects logs and metrics
  • Loki — log storage
  • Prometheus — metric storage
  • Tempo — distributed tracing
  • Grafana — dashboards and visualization
  • Prefect exporter — task-manager metrics, see Metrics

You can deploy it with Docker Compose or on Kubernetes with the infrahub-observability Helm chart. Grafana comes provisioned with pre-built dashboards and data sources for Infrahub, Neo4j, RabbitMQ, and Prefect.

infrahub-observability Helm charthttps://github.com/opsmill/infrahub-helm/tree/stable/charts/infrahub-observability

Docker Compose​

The Community and Enterprise Docker Compose deployments can both include the observability stack. Add the ?observability=true query parameter when you fetch the compose file:

curl "https://infrahub.opsmill.io/?observability=true" > docker-compose.yml
docker compose -p infrahub up -d

On Enterprise, it combines with other parameters such as a sizing preset:

curl "https://infrahub.opsmill.io/enterprise?size=small&observability=true" > docker-compose.yml

Once running, Grafana is available at http://localhost:3500 with the default credentials admin / admin.

Request tracing​

Infrahub exports OpenTelemetry traces, so you can follow one request or background task across the API server, the task workers and the database. The Compose file of the stack, docker-compose-observability-standalone.yml, turns tracing on for the infrahub-server and task-worker services and sends the traces to the bundled Tempo instance, which Grafana shows under the Tempo data source. Infrahub then records one new trace in ten; see Trace sampling to change that. Without the stack, tracing stays off in a Compose deployment. To turn it off with the stack, set INFRAHUB_TRACE_ENABLE=false in a .env file alongside the compose file.

To turn tracing on in a Compose deployment that does not include these settings, add the following to a .env file alongside the compose file:

INFRAHUB_TRACE_ENABLE=true
INFRAHUB_TRACE_EXPORTER_TYPE=otlp
INFRAHUB_TRACE_EXPORTER_PROTOCOL=grpc
INFRAHUB_TRACE_EXPORTER_ENDPOINT=http://infrahub-tempo:4317
INFRAHUB_TRACE_INSECURE=true
INFRAHUB_TRACE_SAMPLE_RATIO=0.1

INFRAHUB_TRACE_INSECURE=true sends spans over a plaintext connection, which is what the bundled Tempo instance expects. Set it to false when your collector terminates TLS.

Exporting to a collector that uses TLS​

When the collector presents a certificate issued by a private or internal certificate authority, the default trust store cannot verify it and every export fails with StatusCode.UNAVAILABLE (gRPC) or a certificate verification error (HTTP). Point Infrahub at the issuing CA instead:

INFRAHUB_TRACE_EXPORTER_ENDPOINT=otel-collector.example.com:4317
INFRAHUB_TRACE_TLS_CA_BUNDLE=/etc/ssl/certs/internal-ca.pem

The bundle is a PEM file that must be readable by the Infrahub server and task workers, so mount it into both containers. With the grpc protocol, setting it also switches the connection to TLS, so INFRAHUB_TRACE_INSECURE can be left at its default. With http/protobuf the endpoint URL scheme is what selects TLS, so the endpoint must start with https://. Infrahub checks the bundle when configuration loads and refuses to start if it cannot be loaded, rather than starting and silently dropping every span.

To skip certificate verification entirely in a test environment, set INFRAHUB_TRACE_TLS_INSECURE=true. This requires INFRAHUB_TRACE_EXPORTER_PROTOCOL=http/protobuf and an https:// endpoint, because the gRPC client cannot skip verification; use a CA bundle for gRPC collectors. Never enable it in production.

For the full Compose install, see the Community or Enterprise install guide.

Kubernetes with Helm​

On Kubernetes the stack is the infrahub-observability Helm chart, which wraps the upstream Grafana and Prometheus community charts. Deploy it bundled with your Infrahub release or as a standalone release.

Prerequisites​

  • A Kubernetes cluster (version 1.24 or later)
  • Helm (version 3 or later) installed on your system
  • A persistent volume provisioner in the cluster. Loki, Prometheus, Tempo, and Grafana enable persistence by default

The stack installs and runs on its own. To collect Infrahub's own metrics and Prefect task data, install it in the same namespace as an Infrahub or Infrahub Enterprise release (or set global.infrahubReleaseName when the release name differs). Without a reachable Infrahub, the stack still starts, but the Prefect exporter stays unhealthy and the Infrahub scrape targets report no data.

Deploy​

Deploy the stack either bundled with your Infrahub release or as its own release.

Bundled with Infrahub​

The infrahub and infrahub-enterprise charts package the observability stack as a subchart, gated by infrahub-observability.enabled (default false). Enable it in your Infrahub values to deploy the stack as part of the same release and namespace — no separate install.

For the infrahub (Community) chart:

# infrahub values
infrahub-observability:
  enabled: true

For the infrahub-enterprise chart, nest the key under infrahub:

# infrahub-enterprise values
infrahub:
  infrahub-observability:
    enabled: true

Apply the change with helm upgrade on your Infrahub release. Enabling the bundled stack also turns tracing on automatically: Infrahub emits traces to the bundled Tempo at <release>-tempo:4317 (where <release> is your Infrahub release name) with no further configuration. The chart does not set the sampling ratio, so Infrahub records one new trace in ten, its default; see Trace sampling.

Standalone release​

To manage the stack on its own release lifecycle — for example, to add it next to an existing Infrahub release without changing that release — install the chart directly. The --create-namespace flag creates the target namespace if it does not already exist; drop it when installing into the namespace where Infrahub already runs:

helm install obs oci://registry.opsmill.io/opsmill/chart/infrahub-observability --version 0.1.0 -n infrahub --create-namespace

The release name (obs above) prefixes the service names referenced throughout this guide.

A standalone release does not wire Infrahub's tracing for you. To send traces to its Tempo, set global.tracing on the Infrahub release:

# infrahub values
global:
  tracing:
    enabled: true
    endpoint: "obs-tempo:4317"
    protocol: grpc
    insecure: true

Services and access​

Every component is exposed through a ClusterIP service, reachable only from inside the cluster:

ServicePurpose
<release>-grafanaGrafana UI
<release>-lokiLog storage (Alloy pushes logs here)
<release>-prometheus-serverMetric storage (Alloy remote-writes here)
<release>-tempoTrace storage (OTLP receiver on port 4317)
<release>-prometheus-node-exporterHost metrics
<release>-infrahub-observability-prefect-exporterPrefect metrics (scraped by Alloy on port 8000)

<release> is your Infrahub release name when bundled, or the observability release name (obs above) when standalone. Run kubectl get svc -n infrahub to list the exact service names for your release.

The log, metric, and trace services only need to be reachable from inside the cluster, so keep them as ClusterIP. Grafana is the only component intended for people to browse.

To open Grafana, forward its service to your machine (replace obs with your Infrahub release name if you bundled the stack):

kubectl port-forward svc/obs-grafana 3000:80 -n infrahub

Then browse to http://localhost:3000 and sign in with the default credentials admin / admin.

To reach Grafana without port-forwarding, set grafana.service.type to LoadBalancer or NodePort, or enable an ingress with grafana.ingress.enabled.

warning

Change the default Grafana credentials before exposing it outside the cluster. Set grafana.admin.existingSecret to a secret you manage rather than relying on the default admin / admin.

Configure the stack​

The component values below set the size, retention, and exposure of each part of the stack:

# Turn off components you don't need
tempo:
  enabled: false        # disable tracing

# Persistence and retention
loki:
  singleBinary:
    persistence:
      size: 20Gi
prometheus:
  server:
    retention: 7d
    persistentVolume:
      size: 50Gi

# Expose Grafana through an ingress and use a managed admin secret
grafana:
  ingress:
    enabled: true
  admin:
    existingSecret: grafana-admin

Where you place these values depends on how you deploy:

  • Standalone — set them at the top level of the observability values.yml.
  • Bundled — nest the same blocks under infrahub-observability: (or infrahub.infrahub-observability: for Enterprise) in your Infrahub values:
# infrahub values (bundled)
infrahub-observability:
  tempo:
    enabled: false
  grafana:
    ingress:
      enabled: true
global values are shared, not nested

global.* values are Helm global values, shared across Infrahub and the bundled subchart, so they always stay at the top level of your values — never nest them under infrahub-observability:. The same applies to the Infrahub chart's global.tracing. When bundled, the stack resolves the Infrahub release name and namespace from the release itself, so global.infrahubReleaseName and global.infrahubNamespace are only needed for a standalone release that points at a separately named or located Infrahub.

OptionDefaultDescription
<component>.enabledtrueToggle any of alloy, loki, tempo, grafana, prometheus, prometheus-node-exporter, prefectExporter
component persistence sizeLoki 10Gi, Tempo 10Gi, Prometheus 20Gi, Grafana 5GiPersistent volume size per component
loki.loki.limits_config.retention_period24hLog retention
tempo.tempo.retention, prometheus.server.retention96hTrace and metric retention
grafana.service.type, grafana.ingress.enabledClusterIP, falseHow Grafana is exposed
grafana.adminPassword, grafana.admin.existingSecretadminGrafana credentials
alloy.cadvisor.enabledtrueScrape per-container metrics from the kubelet cAdvisor endpoint. Disable where cluster policy forbids nodes/proxy access
tempo.tempo.metricsGenerator.enabledfalseGenerate request metrics from spans. Requires tempo.tempo.metricsGenerator.remoteWriteUrl
prefectExporter.enabledtrueDeploy the Prefect metrics exporter
global.infrahubReleaseName, global.infrahubNamespaceinfrahub, release namespaceStandalone only (top-level, never nested): point the release at a separately named or located Infrahub. Auto-resolved when bundled

For the complete list of values, see the chart's values.yaml.

The Docker Compose stack keeps traces and logs for the same time as the chart: 96 hours of traces, set by block_retention in the Tempo configuration, and 24 hours of logs, set by retention_period in the Loki configuration. Both files are embedded in the Compose file of the stack.

infrahub-observability values.yamlhttps://github.com/opsmill/infrahub-helm/blob/stable/charts/infrahub-observability/values.yaml

Verify the deployment​

Check that the stack's pods are running:

kubectl get pods -n infrahub

You should see an Alloy pod on each node (deployed as a DaemonSet), single-instance Loki, Tempo, Prometheus, and Grafana pods, a node exporter on each node, and the Prefect exporter. The Prefect exporter only becomes healthy once it can reach the Infrahub task manager, so deploy it alongside an Infrahub release. Once Grafana is reachable, open it and confirm that the Infrahub dashboards render and that the Prometheus and Loki data sources connect successfully.

To upgrade the stack later, see Upgrade the observability stack.

Trace sampling​

When tracing is on, Infrahub records a share of the new traces, set by INFRAHUB_TRACE_SAMPLE_RATIO on the API server and on every task worker:

ValueEffect
0.1 (default)Records one new trace in ten
1Records every new trace
0Records no new trace

Infrahub recorded every trace before this setting existed. Each recorded trace adds work to the API server and the task workers and uses storage in Tempo, so a higher ratio costs more. Infrahub reads the value when a process starts. To record more traces for a limited time without a restart, see Record more traces during an incident.

For a request or a flow run that arrives with a trace context, Infrahub keeps the decision of its caller: it records the request when the caller recorded its own span, and does not record it otherwise. A flow run started by an API request therefore belongs to the same trace as that request, and a trace is either complete or absent.

Record more traces during an incident​

A slow component is easier to find when every request is traced, and a restart can remove the slow behavior you want to observe. An account with the super_admin permission can therefore replace the ratio on every API server process and task worker for 1 second to 24 hours, without a restart. Send a PUT request to /api/troubleshooting/trace-sample-ratio with the ratio and the number of seconds it applies:

curl -X PUT "https://infrahub.example.com/api/troubleshooting/trace-sample-ratio" \
  -H "X-INFRAHUB-KEY: <api-token>" \
  -H "Content-Type: application/json" \
  -d '{"ratio": 1, "duration_seconds": 1800}'

The response shows the sampling state of the API server process that handled the request:

{
  "enabled": true,
  "default_ratio": 0.1,
  "temporary_ratio": 1.0,
  "expires_at": "2026-10-21T14:30:00Z",
  "ratio_in_force": 1.0
}
  • A GET request to the same path returns this state without changing it.
  • A DELETE request to the same path clears the temporary ratio before the end time, so every process uses INFRAHUB_TRACE_SAMPLE_RATIO again.
  • A new PUT request replaces the ratio and the end time of the previous one.
  • At the end time, every process uses INFRAHUB_TRACE_SAMPLE_RATIO again, with no request needed.
  • A process that starts or restarts while the temporary ratio applies uses INFRAHUB_TRACE_SAMPLE_RATIO. Send the PUT request again to apply the temporary ratio to it.
  • When tracing is off, the PUT and DELETE requests fail with Tracing is not enabled on this instance; set INFRAHUB_TRACE_ENABLE, and the GET request returns "enabled": false.

For each change, Infrahub records an infrahub.trace.sample_ratio_updated event with the account that made the change, the ratio before and after it, and the end time. See the trace events reference.

Check that every process applied the change​

Each API server process and task worker writes one log line when it applies the change: Set the temporary trace sampling ratio, with the ratio and expires_at fields, or Cleared the temporary trace sampling ratio. In Grafana, search Loki for that text and check that you find one line per API server process and per task worker. A process without the line did not receive the change; send the request again.

Reduce the cost of tracing​

To stop recording new traces at once, send a PUT request with "ratio": 0 and a duration of up to 86400 seconds. At the next restart, lower INFRAHUB_TRACE_SAMPLE_RATIO, or set INFRAHUB_TRACE_ENABLE=false to turn tracing off.

Find the log lines of a trace​

Each log line that the API server or a task worker writes while a trace is recorded has a trace_id field: the trace ID as 32 lowercase hexadecimal characters, the form Tempo uses. The API server wrote it before as a 0x hexadecimal number without leading zeros; update the log queries and alert rules that read trace_id.

In Grafana, open a span in Tempo and select Logs for this span to view the log lines of its trace, from one hour before to one hour after the span. From a log line in Loki, open the line and select View trace in Tempo next to its trace_id field. The task logs that the Infrahub web interface shows for each task don't have a trace_id.

To set up the same links in your own Grafana, see Send Infrahub telemetry to your own monitoring.

Values that spans never contain​

Before a span leaves the API server or a task worker, Infrahub replaces these attribute values with [MASKED] and keeps the attribute name:

  • Client IP addresses
  • URL query strings, for example the code and state parameters of a single sign-on login. The scheme, the host name and the path of the URL stay readable.
  • Attributes whose name contains token, password, secret, email, user_name or username
  • The account_name attribute

Database queries and query names stay readable. Log lines are not masked.