Prometheus metrics
Infrahub exposes the metrics below in the Prometheus text format. The API server serves them on /metrics on its HTTP port, and each task worker serves them on the port set by INFRAHUB_METRICS_PORT (default 8000). Task-manager metrics come from the Prefect exporter included in the observability stack.
To choose which metrics to watch and configure scraping, see Metrics.
HTTP requests​
Recorded by the API server for every request except /health. The path label contains the route template, such as /api/schema/{schema_kind}, not the literal URL.
| Metric | Type | Labels | Description |
|---|---|---|---|
infrahub_requests_total | Counter | method, path, status_code, app_name | Total HTTP requests |
infrahub_request_duration_seconds | Histogram | method, path, status_code, app_name | HTTP request duration, with buckets from 0.1 to 1200 seconds |
infrahub_requests_in_progress | Gauge | method, app_name | HTTP requests currently in progress |
Instance​
| Metric | Type | Labels | Description |
|---|---|---|---|
infrahub_info | Gauge | version, worker_id | Always 1. Identifies the Infrahub version and the process that reports it |
GraphQL​
Recorded by the API server for each GraphQL request.
| Metric | Type | Labels | Description |
|---|---|---|---|
infrahub_graphql_duration_seconds | Histogram | type, operation, branch, name, query_id | Query duration, with buckets from 0.01 to 1200 seconds |
infrahub_graphql_response_size_bytes | Histogram | type, operation, branch, name, query_id | Response size, in bytes |
infrahub_graphql_query_depth | Histogram | type, operation, branch, name, query_id | Query depth |
infrahub_graphql_query_height | Histogram | type, operation, branch, name, query_id | Query height |
infrahub_graphql_query_objects | Histogram | type, operation, branch, name, query_id | Number of objects in the query |
infrahub_graphql_top_level_queries | Histogram | type, operation, branch, name, query_id | Number of top-level queries |
infrahub_graphql_variables | Histogram | type, operation, branch, name, query_id | Number of variables |
infrahub_graphql_errors | Histogram | type, operation, branch, name, query_id | Number of errors in the response |
infrahub_graphql_generate_schema | Histogram | branch | Time to generate the GraphQL schema of a branch |
Database​
Recorded by every process that queries the database, so both the API server and the task workers report them.
| Metric | Type | Labels | Description |
|---|---|---|---|
infrahub_db_query_execution_seconds | Histogram | type, query, runtime, context1, context2 | Execution time of a database query, with buckets from 0.005 to 300 seconds |
infrahub_db_transaction_retries_total | Counter | name | Transactions retried after a transient error |
infrahub_db_last_connection_pool_usage | Gauge | address | Last known number of active connections in the driver pool of the process, read when a query starts. One series per process |
infrahub_db_connection_pool_size | Gauge | address | Maximum number of connections in the driver pool of the process (INFRAHUB_DB_MAX_CONNECTION_POOL_SIZE). One series per process |
infrahub_db_connection_wait_seconds | Histogram | Time a query waited to get a connection from the driver pool, with buckets from 0.001 to 60 seconds | |
infrahub_db_reference_query_floor_seconds | Gauge | All-time minimum execution time of the reference permission query | |
infrahub_db_reference_query_window_min_seconds | Gauge | Minimum execution time of the reference query over the recent sliding window | |
infrahub_db_reference_query_stress_ratio_median | Gauge | Recent median reference-query time divided by the all-time floor. 1.0 means the database is not under stress |
Locks​
| Metric | Type | Labels | Description |
|---|---|---|---|
infrahub_lock_acquire_seconds | Histogram | lock, type | Time to acquire a lock, with buckets from 0.0005 to 300 seconds |
infrahub_lock_reserved_duration_seconds | Histogram | lock, type | Time a lock is held by a client, with buckets from 0.001 to 300 seconds |
API admission control​
Recorded by the API server when priority-aware backpressure is enabled (INFRAHUB_API_BACKPRESSURE_ENABLED, default true, see the configuration reference). The priority label is high, medium or low, taken from the X-Priority request header.
| Metric | Type | Labels | Description |
|---|---|---|---|
infrahub_admission_offered_total | Counter | priority | Requests entering the admission layer |
infrahub_admission_admitted_total | Counter | priority | Requests admitted |
infrahub_admission_rejected_total | Counter | priority, reason | Requests rejected with 429 Too Many Requests |
infrahub_admission_in_flight | Gauge | priority | Admitted requests currently running, summed over the running API worker processes |
infrahub_admission_waiters | Gauge | priority | Requests queued for a slot, summed over the running API worker processes |
infrahub_admission_sojourn_seconds | Histogram | priority | Time a request waited for a slot |
infrahub_admission_max_concurrency | Gauge | Effective number of admission slots, summed over the live API workers | |
infrahub_admission_missing_priority_total | Counter | Requests with a missing or invalid X-Priority header | |
infrahub_admission_sustained_load_seconds | Gauge | Seconds the worker has continuously seen database stress at or above the significant-load ratio, 0 below it. One series per API worker process |
Task workers​
Recorded by each task worker for the flow runs it starts. The flow label is the flow name, never a flow run ID. The queue label is the priority queue of the flow run (high, medium or low), or unknown when the flow run has no work queue name.
| Metric | Type | Labels | Description |
|---|---|---|---|
infrahub_task_flow_run_start_delay_seconds | Histogram | flow, queue | Time between the expected start time of a flow run and the moment a task worker started it, with buckets from 1 to 3600 seconds. A flow run started before its expected start time records 0, and a flow run without an expected start time records nothing |
infrahub_task_flow_runs_in_progress | Gauge | flow, queue | Flow runs that the task worker is running now |
Task manager (Prefect exporter)​
The observability stack runs prometheus-prefect-exporter 3.3.0 against the task manager API. The metrics most useful for Infrahub are:
| Metric | Labels | Description |
|---|---|---|
prefect_work_queues_late_runs_count | work_queue_name, work_pool_name, priority, is_paused, status, healthy, type, health-check policy labels | Flow runs in the queue in the Late state, which the task manager sets 15 seconds after the scheduled start |
prefect_info_work_queues | same as above | Work queue state; 0 when paused, 1 otherwise |
prefect_info_work_pools | work_pool_name, type, is_paused, status | Work pool state; 0 when paused, 1 otherwise |
prefect_info_flow_runs | deployment_name, flow_name, state_name, work_queue_name | Number of flow runs per state, counting only runs whose start time, or scheduled start time if not started, is within the last OFFSET_MINUTES (default 3) |
prefect_deployment_failed_flow_runs | deployment_name, flow_name, last_failed_run_id, state_name | Last failed or crashed flow runs per deployment, within FAILED_RUNS_OFFSET_MINUTES (default 7 days) |
prefect_flow_runs_total_run_time | flow_name | Total run time of flow runs, in seconds |
prefect_flow_runs_total, prefect_flows_total, prefect_deployments_total, prefect_work_pools_total, prefect_work_queues_total | Object counts |
Dependencies​
The observability stack also scrapes Neo4j on port 2004 and RabbitMQ on port 15692. Their metrics are documented by each project: Neo4j metrics and RabbitMQ Prometheus plugin.