Metrics
Infrahub exposes Prometheus metrics from the API server and from each task worker. Use them to track API latency and errors, database load and lock contention.
The task manager doesn't expose metrics through Infrahub. Each task worker reports how long the flow runs it starts waited and how many it runs now, per priority queue, as described in Monitor the task manager. For the state of the work queues and work pools, scrape the Prefect exporter included in the observability stack.
For the full list of metric names, types and labels, see the Prometheus metrics reference.
Where metrics are exposed​
| Source | Endpoint | What it reports |
|---|---|---|
| API server | /metrics on the HTTP port (8000) | HTTP requests, GraphQL queries, database queries and connection pool, locks, API admission control |
| Task worker | /metrics on INFRAHUB_METRICS_PORT (default 8000) | Database queries, locks, flow run start delay and flow runs in progress per priority queue, and the version of each worker |
| Prefect exporter | /metrics on port 8000 of the exporter container | Task manager: work queues, work pools, flow runs and their states |
| Infrahub Exporter | Its own endpoint | Metrics and service discovery built from the objects stored in Infrahub, not the health of Infrahub itself |
The /metrics endpoint of the API server has no metric for the task manager queue. The task workers report the start delay and the flow runs in progress, and the Prefect exporter reports the queues, as described in Monitor the task manager.
The API server runs several worker processes, four by default, and /metrics combines their values in one response. Gauges are combined in two ways, depending on what they measure:
- A gauge that describes one process, such as
infrahub_infoand the connection pool gauges, reports one series per process, with apidlabel. - A gauge for a count or a limit that adds up across processes, such as
infrahub_admission_in_flightandinfrahub_admission_max_concurrency, reports the sum over the running processes. - When a worker process stops, its values disappear from these two kinds of gauges. Its counters and histograms stay in the totals, and a restart of one worker process doesn't reset the values of the others.
Infrahub combines the values this way only when gunicorn starts with the Infrahub configuration file, --config backend/infrahub/serve/gunicorn_config.py, as the Docker Compose files and the Helm charts do. Without it, /metrics keeps reporting the gauge values of worker processes that stopped.
The /metrics endpoint doesn't require authentication. If the API port is reachable from networks you don't trust, restrict access to /metrics at your load balancer or ingress.
To turn off the task worker endpoint, set INFRAHUB_METRICS_PORT=0.
Scrape the metrics​
The observability stack scrapes the API server, the task workers and the Prefect exporter, plus Neo4j and RabbitMQ, and includes Grafana dashboards for them. If you deploy it, there is nothing to configure.
To scrape Infrahub from your own Prometheus, add a job per source. Scrape each task worker directly rather than through a shared address, so that every worker is collected:
scrape_configs:
- job_name: infrahub-server
static_configs:
- targets: ["infrahub-server:8000"]
- job_name: infrahub-worker
dns_sd_configs:
- names: ["task-worker"]
type: A
port: 8000
- job_name: task-manager-exporter
static_configs:
- targets: ["infrahub-task-manager-exporter:8000"]
The host names match the service names of the Docker Compose deployment. On Kubernetes, use the service names of your release.
Monitor the task manager​
Infrahub runs Generators, Transformations, checks, Git operations and other background work as flow runs in the task manager. When task workers are all busy, flow runs wait past their scheduled start time. The Prefect exporter reports this per work queue:
prefect_work_queues_late_runs_countis the number of flow runs in a queue that are in theLatestate: the task manager marks a run late once it is 15 seconds past its scheduled start. A value that stays above zero means flow runs are scheduled faster than the workers start them: add task workers, or look for long-running tasks in the task list.prefect_info_work_queuesandprefect_info_work_poolsare0when a queue or pool is paused. Thehealthyandstatuslabels report the task manager's own health check.prefect_deployment_failed_flow_runslists recent failed or crashed flow runs per deployment.
The task workers report two metrics of their own, labelled by flow name (flow) and priority queue (queue). They don't need the Prefect exporter:
infrahub_task_flow_run_start_delay_secondsis the time between the expected start time of a flow run and the moment a task worker started it. A rising delay on thehighqueue means users wait for work they started themselves, such as a branch creation.infrahub_task_flow_runs_in_progressis the number of flow runs that each task worker runs now. Sum it over the task workers to get the total per queue.
A flow run with no work queue name is labelled queue="unknown".
prefect_info_flow_runs counts flow runs by state, but only runs whose start time, or scheduled start time if they haven't started, is within the last three minutes. A run that waits longer than that drops out of the count, so it measures throughput, not backlog.
In Grafana, the Prefect / Platform Overview and Prefect / Flow Runs Overview dashboards show flow-run states and work-queue counts from the Prefect exporter. They don't chart prefect_work_queues_late_runs_count, so add a panel or an alert rule for it.
Duration buckets​
A duration histogram counts a value above its highest bucket only in the +Inf bucket, so a percentile computed from it can't be higher than that bucket. The highest bucket of each duration histogram is listed last:
| Metric | Bucket bounds, in seconds |
|---|---|
infrahub_request_duration_seconds | 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600, 1200 |
infrahub_graphql_duration_seconds | 0.01, 0.025, 0.05, 0.1, 0.5, 1, 5, 10, 60, 120, 300, 600, 1200 |
infrahub_db_query_execution_seconds | 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300 |
infrahub_db_connection_wait_seconds | 0.001, 0.01, 0.1, 0.5, 1, 5, 10, 30, 60 |
infrahub_lock_acquire_seconds | 0.0005, 0.25, 0.5, 1, 5, 30, 60, 120, 300 |
infrahub_lock_reserved_duration_seconds | 0.001, 0.5, 1, 5, 10, 30, 60, 120, 300 |
infrahub_task_flow_run_start_delay_seconds | 1, 5, 15, 30, 60, 120, 300, 600, 1800, 3600 |
The lower bounds are the same as in earlier releases, so queries and alert rules that select a bucket by its le label, such as le="0.5", still work.
Metrics to watch​
| To detect | Watch |
|---|---|
| Slow or failing API requests | infrahub_request_duration_seconds, infrahub_requests_total with a 5xx status_code |
| Expensive GraphQL queries | infrahub_graphql_duration_seconds, infrahub_graphql_query_objects, infrahub_graphql_response_size_bytes |
| Database connection pressure | infrahub_db_last_connection_pool_usage against infrahub_db_connection_pool_size, and infrahub_db_connection_wait_seconds. Use them to size INFRAHUB_DB_MAX_CONCURRENT_QUERIES in performance tuning |
| Database under stress | infrahub_db_reference_query_stress_ratio_median, where 1.0 means no stress |
| Requests rejected under load | infrahub_admission_rejected_total, infrahub_admission_waiters |
| Lock contention | infrahub_lock_acquire_seconds |
| Flow runs starting late | infrahub_task_flow_run_start_delay_seconds per queue, and prefect_work_queues_late_runs_count |
| Task workers running many flow runs at once | infrahub_task_flow_runs_in_progress per queue |
GraphQL metrics are labelled by branch and by query name, so their number of series grows with the number of branches and distinct queries.
Dashboard panels for load questions​
The Infrahub Monitoring dashboard of the observability stack has a panel for each of these questions:
| Question | Row and panel | Metric |
|---|---|---|
| Does a process use all the connections of its database pool? | Connection Pool: Highest Connection Pool Usage and Connection Pool Usage vs Size (per process) | infrahub_db_last_connection_pool_usage, infrahub_db_connection_pool_size |
| Do queries wait for a database connection? | Connection Pool: Connection Wait | infrahub_db_connection_wait_seconds |
| How long do flow runs wait before a task worker starts them, per priority queue? | Priority Queues: Flow Run Start Delay p95 by Queue | infrahub_task_flow_run_start_delay_seconds |
| How many flow runs do the task workers run now, per priority queue? | Priority Queues: Flow Runs in Progress by Queue | infrahub_task_flow_runs_in_progress, summed over the task workers |
| How long do slow requests, queries and locks last? | Details Requests and Details DB Query/GraphQL: the duration panels | The histograms in Duration buckets |
Infrahub reads the pool usage when a query starts, so the panel shows the last known value of each process. A query waits at most 60 seconds for a connection, the default of the Neo4j driver, and then fails. A connection wait close to 60 seconds therefore means queries are about to fail: increase INFRAHUB_DB_MAX_CONNECTION_POOL_SIZE, or look for queries that hold connections for a long time.
On the Priority Queues row, a start delay that rises on the high queue while the flow runs in progress stay flat means the task workers don't start more flow runs: add task workers, or look for long-running flow runs in the task list.