Subqueries

PromQL's subqueries and pipeline operators enable advanced data processing and nested query logic, essential for complex observability scenarios. Subqueries allow you to embed one query within another, while pipeline operators transform time series data by grouping, joining, or filtering metrics. These features are critical for building dynamic, reusable metrics pipelines in Prometheus.

Subqueries: Nesting Metrics Queries¶

Subqueries let you execute one query as part of another, enabling hierarchical data processing. They are defined using parentheses () and are often used to compute intermediate results that drive subsequent calculations.

Syntax¶

<metric> <operator> ( <subquery> )

Example: Threshold Detection¶

http_requests_latency_seconds{job="api"} > (avg_over_time(http_requests_latency_seconds{job="api"}[5m]) * 2)

This query identifies latency spikes exceeding twice the average over the last 5 minutes.

Use Cases¶

Derived metrics: Compute a baseline metric (e.g., average) and compare against it.
Aggregation pipelines: Use subquery results as inputs for further filtering or grouping.

Note: Subqueries are evaluated first, and their results are treated as static values in the outer query.

Pipeline Operators: Transforming Time Series Data¶

Pipeline operators reshape time series data by grouping, joining, or filtering metrics. They are applied in sequence using the | symbol and are fundamental for correlating metrics across labels.

Core Operators¶

Operator	Description
`by`	Groups time series by specified labels.
`without`	Groups time series by all labels except specified ones.
`group_left`	Joins metrics using left outer join based on matching labels.
`group_right`	Joins metrics using right outer join based on matching labels.

Example: Joining Metrics¶

http_requests_total{job="api"} 
  group_left() 
  http_response_time_seconds{job="api"}

This query joins request counts with response times using the job label, preserving all time series from the left side.

Example: Aggregation Pipeline¶

http_errors_total{job="api"} 
  by (job, status_code) 
  | sum_over_time()

Groups errors by job and status code, then sums them over time.

Tip: Pipeline operators are applied in the order they appear, enabling complex data transformations.

Advanced Use Cases¶

Subquery + Pipeline Operators¶

Combine subqueries with pipeline operators for multi-stage processing:

(
  avg_over_time(http_requests_latency_seconds{job="api"}[5m])
  by (job)
) 
| increase() 
| topk(5)

1. Compute average latency per job. 2. Track increases over time. 3. Identify top 5 jobs with rising latency.

Dynamic Label Filtering¶

Use by and without to dynamically filter metrics:

http_requests_total{job="api"} 
  by (method, status_code) 
  | avg_over_time()

Aggregates requests by HTTP method and status code.

Key Takeaways¶

Subqueries enable nested metrics processing, ideal for deriving thresholds or baselines.
Pipeline operators reshape time series data via grouping, joining, and filtering.
Combine subqueries with operators to build multi-stage observability pipelines.
Use by/without for label-based aggregation and group_left/group_right for metric correlation.