Prometheus and Grafana: Metrics Monitoring and Dashboards

A practical guide to Prometheus and Grafana — the de facto open-source standard for collecting, querying, alerting on, and visualizing metrics — covering Prometheus's pull-based model, metric types, PromQL, alerting rules, Grafana dashboards, and .NET integration, completing this series' observability trio alongside Structured Logging and Distributed Tracing.

Table of Contents

1. Introduction
2. Why Prometheus's Pull Model Is Different
3. Metric Types
4. Exposing Metrics from .NET
5. Service Discovery and Scrape Configuration
6. PromQL: Querying Metrics
7. Recording Rules
8. Alerting
9. Grafana Dashboards
10. Cardinality: The Silent Cost Multiplier
11. Long-Term Storage and Federation
12. Prometheus/Grafana Within the Broader Observability Stack
13. Common Pitfalls
14. Quick Reference Table
15. Conclusion

Introduction

Prometheus is an open-source metrics collection and alerting system built around a distinctive pull-based model, and Grafana is the open-source visualization layer most commonly paired with it — together they form the de facto standard, vendor-neutral stack for metrics monitoring in cloud-native environments, especially Kubernetes (per this series' Kubernetes/Helm guide), where both originated from and remain most deeply integrated. This guide completes this series' observability trio: Structured Logging covers the logs pillar, Distributed Tracing (built on OpenTelemetry) covers the traces pillar, and this guide covers the metrics pillar in depth.

histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, route))

That single PromQL expression computes the p99 latency per API route over a rolling 5-minute window — the kind of question this guide builds toward answering fluently, along with how to get the underlying data into Prometheus in the first place and turn it into dashboards and alerts that actually help.

1. Why Prometheus's Pull Model Is Different

Pull, not push

Prometheus server → periodically SCRAPES → /metrics endpoint on each target

app.MapPrometheusScrapingEndpoint(); // exposes GET /metrics for Prometheus to scrape

Unlike many metrics systems (and unlike the OpenTelemetry Collector's typical push-based OTLP export, covered in this series' OpenTelemetry guide), Prometheus works by pulling — the Prometheus server itself periodically makes an HTTP request to a /metrics endpoint exposed by each monitored application, rather than applications pushing their metrics out to a central collector.

Why this design choice matters practically

• Prometheus can tell if a target is down — a failed scrape (connection refused, timeout) is itself a meaningful signal ("this target isn't reachable"), distinct from a target that's simply not emitting metrics; a push-based system generally can't distinguish "not sending data" from "not running" as easily.
• No agent needed on the application side for basic exposition — the application just needs to expose an HTTP endpoint; it doesn't need to know Prometheus's address, handle export retries, or manage a connection to a remote collector.
• Centralized control over scrape frequency and target list — operators configure what to scrape and how often from the Prometheus server's own configuration, rather than every application independently deciding its own export cadence.

The trade-off

Pull-based scraping requires Prometheus to have network access to every target it monitors — for genuinely short-lived jobs (a batch job that completes in seconds, potentially before a scrape interval elapses) or targets behind restrictive network boundaries, this model needs a workaround (Section 4's discussion of the Pushgateway) rather than working naturally out of the box the way a push-based system would for those specific scenarios.

2. Metric Types

Prometheus defines four core metric types, and choosing the right one for a given measurement is what makes later querying (Section 5) actually work correctly.

Counter: a value that only ever increases

var ordersPlacedCounter = meter.CreateCounter<int>("orders_placed_total");
ordersPlacedCounter.Add(1);

A counter represents a cumulative count that only goes up (or resets to zero on a restart) — total requests served, total orders placed, total errors encountered. Counters are never queried for their raw value directly in practice; they're almost always queried via rate() (Section 5) to get a meaningful per-second rate over time, since the raw cumulative total by itself ("14,382,910 total requests since the process started") is rarely the interesting number.

Gauge: a value that can go up or down

var activeConnectionsGauge = meter.CreateObservableGauge("active_connections", () => GetCurrentConnectionCount());

A gauge represents a value that can increase or decrease freely — current memory usage, active connection count, queue depth right now. Unlike a counter, a gauge's raw current value is directly meaningful and commonly graphed as-is.

Histogram: a distribution of observed values, bucketed

var requestDurationHistogram = meter.CreateHistogram<double>("http_server_request_duration_seconds");
requestDurationHistogram.Record(elapsedSeconds, new KeyValuePair<string, object?>("route", "/orders"));

A histogram samples observations (request durations, response sizes) into a configured set of buckets, and exposes both a total count and a running sum, alongside per-bucket cumulative counts — this is what enables the percentile calculations (histogram_quantile, Section 5) central to latency analysis, directly connecting to this series' Distributed Tracing guide's percentile-based latency discussion, but computed from aggregated metric data rather than derived from individual traces.

Summary: client-side-calculated percentiles (generally the less-preferred option)

A summary calculates percentiles directly on the client (application) side before exposing them, rather than exposing raw bucket counts for the server to calculate from — this avoids histogram's bucket-configuration considerations but has a significant limitation: summary percentiles cannot be meaningfully aggregated across multiple instances (you can't average or combine pre-calculated p99s from ten different pod replicas into a genuine overall p99), which is precisely why histograms are generally the recommended choice for anything that will run as multiple replicas — nearly every real production service.

3. Exposing Metrics from .NET

Via OpenTelemetry's Prometheus exporter (the recommended modern path)

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter("OrderApi.Metrics")
        .AddPrometheusExporter());

var app = builder.Build();
app.MapPrometheusScrapingEndpoint(); // exposes GET /metrics

As covered in this series' OpenTelemetry guide, the same Meter/Counter/Histogram API used for OTLP export can also be exposed in Prometheus's native text format via the OpenTelemetry Prometheus exporter — meaning a .NET application instrumented once with OpenTelemetry's metrics API can serve both an OTLP-based pipeline and a traditional Prometheus scrape target simultaneously, without maintaining two separate instrumentation approaches.

What a scraped /metrics endpoint actually looks like

# HELP http_server_request_duration_seconds Duration of HTTP requests
# TYPE http_server_request_duration_seconds histogram
http_server_request_duration_seconds_bucket{route="/orders",le="0.1"} 245
http_server_request_duration_seconds_bucket{route="/orders",le="0.5"} 480
http_server_request_duration_seconds_bucket{route="/orders",le="1.0"} 495
http_server_request_duration_seconds_bucket{route="/orders",le="+Inf"} 500
http_server_request_duration_seconds_sum{route="/orders"} 62.4
http_server_request_duration_seconds_count{route="/orders"} 500

This plain-text exposition format is genuinely simple — human-readable, easy to curl and inspect directly, and straightforward enough that writing a custom exporter for something not already covered by an existing instrumentation library is a modest undertaking, not a significant engineering project.

Custom application metrics

private static readonly Meter OrderMeter = new("OrderApi.Metrics");
private static readonly Counter<int> OrdersPlacedCounter = OrderMeter.CreateCounter<int>("orders_placed_total");
private static readonly Histogram<double> OrderProcessingDuration = OrderMeter.CreateHistogram<double>("order_processing_duration_seconds");

public async Task<Order> PlaceOrderAsync(CreateOrderRequest request)
{
    var stopwatch = Stopwatch.StartNew();
    var order = await _repository.CreateAsync(request);
    OrdersPlacedCounter.Add(1, new KeyValuePair<string, object?>("customer_tier", request.CustomerTier));
    OrderProcessingDuration.Record(stopwatch.Elapsed.TotalSeconds);
    return order;
}

Business-specific metrics (orders placed by tier, processing duration) follow exactly the same pattern covered in this series' OpenTelemetry guide — registered once via AddMeter, they flow through the same pipeline as framework-level metrics and become queryable in Prometheus alongside them.

4. Service Discovery and Scrape Configuration

Static configuration for a small, fixed set of targets

scrape_configs:
  - job_name: 'order-api'
    scrape_interval: 15s
    static_configs:
      - targets: ['order-api-1:8080', 'order-api-2:8080']

For a small number of known, stable targets, static configuration is simple and sufficient — but this doesn't scale to environments where instances come and go dynamically (autoscaling, rolling deployments), which is the normal case for anything covered in this series' Kubernetes/Helm and cloud compute guides.

Kubernetes service discovery

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

# On the pod itself, in the Deployment manifest
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"

Prometheus's Kubernetes service discovery integration automatically discovers pods (or services, endpoints, nodes) matching configured criteria — commonly, annotations on the pod itself (as shown above) opt it into scraping — meaning as pods are created and destroyed by deployments, rollouts, and autoscaling (per this series' Kubernetes/Helm guide), Prometheus's scrape target list stays automatically current without manual configuration updates for every new instance.

The Pushgateway: the workaround for short-lived jobs

echo "batch_job_duration_seconds 45.2" | curl --data-binary @- http://pushgateway:9091/metrics/job/nightly-cleanup

For genuinely short-lived batch jobs (per this series' Background Services guide's discussion of scheduled jobs) that might complete and exit before Prometheus's next scheduled scrape, the Pushgateway provides an intermediary a job can push its final metrics to, which Prometheus then scrapes from instead of the job itself — an explicit, deliberate exception to the pull model, reserved specifically for this narrow use case rather than a general-purpose push mechanism for anything that finds pull inconvenient.

5. PromQL: Querying Metrics

Instant vectors and range vectors

http_server_request_duration_seconds_count                    # instant vector: current value of every matching series
http_server_request_duration_seconds_count[5m]                 # range vector: every sample over the last 5 minutes

An instant vector returns the most recent value of a metric (per unique label combination); a range vector returns every sample within a specified time window — most useful PromQL functions (like rate()) operate on range vectors to compute something meaningful over time, rather than working with a single instant snapshot.

rate(): turning a counter into a meaningful per-second value

rate(http_requests_total[5m])

Since a counter (Section 2) only ever increases, rate() computes the per-second average rate of increase over the specified window — this is almost always how a counter is actually queried in practice; the raw cumulative counter value on its own is rarely the interesting number.

Aggregation: sum, avg, grouping with by

sum(rate(http_requests_total[5m])) by (route, method)

Aggregation operators combine multiple time series (one per unique label combination) into fewer, more meaningful series — sum(...) by (route, method) computes total request rate per unique route/method combination, collapsing away other labels (like the specific pod instance) that aren't relevant to this particular question.

histogram_quantile(): computing percentiles from histogram buckets

histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, route))

This is the PromQL idiom for the percentile-based latency analysis covered in this series' Distributed Tracing guide, but computed from aggregated histogram data across every instance of a service rather than from individual sampled traces — histogram_quantile interpolates a percentile value from the cumulative bucket counts, and grouping by (le, route) (keeping the histogram's bucket boundary label while aggregating away instance-specific labels) gives a genuine, combined p99 across every replica of a service for a specific route.

Alert-style boolean expressions

rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05

This expression computes the error rate (5xx responses divided by total requests) over a 5-minute window and evaluates whether it exceeds 5% — exactly the kind of expression that becomes an alerting rule (Section 7), turning a metric query into an actionable, automatically-evaluated condition.

increase() for counting events over a window

increase(orders_placed_total[1h])

increase() computes the total increase in a counter over the specified window (accounting for counter resets, e.g., from a process restart) — useful for "how many orders were placed in the last hour," as distinct from rate()'s per-second average.

6. Recording Rules

Pre-computing expensive queries

groups:
  - name: order-api-recording-rules
    interval: 30s
    rules:
      - record: order_api:request_duration_p99
        expr: histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, route))

A recording rule pre-computes a PromQL expression on a schedule and saves the result as a new, permanently-stored time series — rather than recalculating an expensive aggregation (like the p99 example from Section 5) every time a dashboard panel or alert needs it, the recording rule computes it once, on Prometheus's own schedule, and every downstream consumer just reads the pre-computed result.

Why this matters at scale

For a genuinely high-cardinality metric (many distinct label combinations, Section 9) queried frequently by multiple dashboards and alerts, repeatedly recalculating the same expensive aggregation on every dashboard refresh and every alert evaluation cycle is wasteful — recording rules compute it once and let everything else read the cheap, pre-aggregated result, a meaningful performance and cost optimization once a Prometheus deployment reaches real production scale.

7. Alerting

Alertmanager: Prometheus's companion alerting component

groups:
  - name: order-api-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for {{ $labels.route }}"
          description: "Current error rate: {{ $value | humanizePercentage }}"

Prometheus itself evaluates alerting rules (PromQL expressions with a threshold) on a schedule, and when a rule's condition is true continuously for the duration specified by for (avoiding alerting on a single, momentary blip), it fires an alert to Alertmanager — a separate component responsible for deduplication, grouping related alerts together, silencing, and routing to the actual notification channels (PagerDuty, Slack, email).

Why for matters: avoiding alert flapping

for: 5m  # the condition must be true continuously for 5 minutes before actually firing

Without a for duration, a metric that briefly crosses a threshold for a single evaluation cycle (a momentary spike that resolves itself immediately) would fire and immediately resolve an alert — for requires the condition to hold continuously across multiple evaluation cycles before actually notifying anyone, filtering out exactly the kind of noisy, self-resolving blips that erode trust in alerting (the same "flaky test" trust-erosion problem covered in this series' CI/CD Pipelines guide, applied to alerts instead of test results).

Alertmanager routing and grouping

route:
  group_by: ['alertname', 'route']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-oncall'

receivers:
  - name: 'slack-oncall'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/...'
        channel: '#oncall-alerts'

Grouping related alerts (multiple routes all showing elevated error rates simultaneously, likely from the same underlying cause) into a single notification, rather than paging on-call separately for each one, is what keeps alerting genuinely actionable at scale — an on-call engineer receiving 40 separate notifications for what's actually one incident is a well-documented path to alert fatigue and, eventually, ignored pages.

Designing alerts around symptoms, not causes

The general, widely-adopted guidance: alert on symptoms users would actually notice (elevated error rate, high latency, a failed health check) rather than on every possible underlying cause independently (CPU usage, memory usage, a specific internal queue depth) — a single well-designed symptom-based alert, investigated using the distributed tracing and structured logging techniques covered in this series' companion guides, is generally more actionable and less noisy than dozens of narrowly-scoped, cause-based alerts that may or may not actually correspond to a real user-facing problem.

8. Grafana Dashboards

Connecting Grafana to Prometheus

# Grafana data source configuration
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

Grafana connects to one or more data sources (Prometheus being the most common for metrics, but also Loki for logs and Tempo for traces, forming the LGTM stack referenced in this series' OpenTelemetry guide) and builds dashboards from queries against them — Grafana itself stores no metric data; it's purely a query and visualization layer.

Building a dashboard panel

{
  "title": "P99 Request Latency by Route",
  "targets": [
    { "expr": "histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, route))" }
  ],
  "type": "timeseries"
}

A Grafana panel is, at its core, a PromQL query (or several) paired with a visualization type (time series graph, gauge, heatmap, table) — the PromQL expertise from Section 5 transfers directly into building genuinely useful dashboard panels, rather than being a separate skill.

Dashboards as code

// A dashboard JSON model, version-controlled and provisioned automatically

apiVersion: 1
providers:
  - name: 'default'
    folder: 'Order API'
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards

Storing dashboard definitions as version-controlled JSON files (exported from Grafana's UI, or authored directly) and provisioning them automatically on Grafana startup — rather than manually clicking together dashboards through the UI, which tends to drift and doesn't survive a Grafana redeployment — extends the same "infrastructure and configuration as code" discipline covered throughout this series (Terraform/Bicep, GitOps, CI/CD Pipelines) to dashboards themselves.

Variables for reusable, parameterized dashboards

$environment  → dropdown: production, staging, development
$service       → dropdown: dynamically populated from label_values(up, job)

histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket{environment="$environment", job="$service"}[5m])) by (le))

Grafana template variables let one dashboard definition serve many contexts — the same latency panel, filterable by a dropdown to any service or environment — avoiding the need to maintain nearly-identical, hand-duplicated dashboards per service, which (like the copy-pasted pipeline YAML problem covered in this series' GitHub Actions and Azure DevOps guides) tends to drift out of sync as one copy gets updated and others don't.

9. Cardinality: The Silent Cost Multiplier

What cardinality actually means here

http_requests_total{route="/orders", method="POST", status="200", customer_id="42"}

Every unique combination of a metric name and its label values is a distinct time series that Prometheus must store and index independently — including customer_id as a label above means Prometheus is now storing a separate time series per individual customer, not one aggregated series for the /orders route.

Why high-cardinality labels are a genuine operational risk, not just a style preference

10 routes × 5 methods × 10 status codes = 500 time series      ← entirely manageable
10 routes × 5 methods × 10 status codes × 100,000 customers = 50,000,000 time series  ← a serious problem

Adding a high-cardinality label (a customer ID, a user ID, a raw request ID, anything with effectively unbounded distinct values) to a metric multiplies the number of stored time series by that label's cardinality — this is one of the most common, most damaging Prometheus operational mistakes, capable of degrading query performance and dramatically increasing memory/storage usage for the entire Prometheus deployment, not just for the one metric that introduced it.

Where high-cardinality data actually belongs instead

Metrics (Prometheus): aggregate counts and rates — "how many orders per minute," not "which specific customer"
Traces (per this series' Distributed Tracing guide): individual request detail, including customer ID as a SPAN attribute
Logs (per this series' Structured Logging guide): individual event detail, including customer ID as a structured property

This is a direct, practical consequence of the "three pillars answer different questions" principle from this series' OpenTelemetry guide — genuinely per-entity, high-cardinality detail (which specific customer, which specific order ID) belongs in traces and logs, which are architecturally designed to handle high-cardinality, per-event data; metrics and Prometheus specifically are optimized for aggregate, bounded-cardinality dimensions, and forcing high-cardinality data into a metric label is using the wrong pillar for the job.

10. Long-Term Storage and Federation

Prometheus's default local storage limitation

By default, a single Prometheus server stores data locally on disk with a configured retention period (commonly 15 days to a few months) — appropriate for recent operational monitoring and alerting, but not designed as a long-term historical data warehouse, and a single Prometheus instance doesn't natively scale horizontally for very high metric volume on its own.

Remote write to long-term storage backends

remote_write:
  - url: "http://mimir:9009/api/v1/push"

Prometheus supports remote write — continuously streaming scraped samples to an external, horizontally-scalable long-term storage system (Grafana Mimir, Thanos, Cortex, or a managed cloud equivalent) — decoupling "how long can we retain and efficiently query this data" from Prometheus's own local storage and retention configuration.

Federation and Thanos/Mimir for genuinely large-scale deployments

For organizations running many Prometheus instances (one per Kubernetes cluster, per region, per team), Thanos or Grafana Mimir provide a global query layer aggregating data across all of them, plus long-term, cost-efficient object storage (S3/Azure Blob-backed) for historical retention — the practical solution once a single Prometheus server's local storage and single-instance query scope genuinely becomes a limiting factor, mirroring the "start simple, add the distributed/scaled version once genuine scale demands it" pattern covered throughout this series' database and infrastructure guides.

11. Prometheus/Grafana Within the Broader Observability Stack

Completing this series' observability picture

Metrics (Prometheus)  → aggregate, dashboards, alerting — THIS GUIDE
Traces (OpenTelemetry + a tracing backend) → per-request, causal chain across services — Distributed Tracing guide
Logs (structured, centralized) → per-event detail, correlated with traces → Structured Logging guide

With this guide, this series' observability trio is complete — metrics for the aggregate "how is the system behaving" question and alerting, traces for the per-request "what actually happened, across which services" question, and logs for the detailed "what exactly occurred at this specific point" question, all correlated together via the shared trace/span IDs and consistent labeling conventions covered across these three guides.

Prometheus and OpenTelemetry: complementary, not competing

As covered in Section 3, a modern .NET application can instrument once with OpenTelemetry's metrics API and export to both an OTLP pipeline and a native Prometheus scrape endpoint simultaneously — Prometheus's pull-based scraping and PromQL query language remain genuinely valuable and widely adopted specifically for the metrics pillar, even as OpenTelemetry has become the standard for the instrumentation layer sitting above it.

Grafana as the unifying visualization layer

Because Grafana can query Prometheus (metrics), Loki (logs), and Tempo (traces) from within the same dashboard — and increasingly supports jumping directly from a metric panel to a correlated trace or log query — it's commonly the single pane of glass tying together every guide in this series' observability trio into one coherent operational view, rather than three separate tools an engineer has to manually context-switch between during an investigation.

12. Common Pitfalls

Quick Reference Table

Conclusion

Prometheus and Grafana together form the metrics half of the observability picture this series has built out across its Structured Logging, Distributed Tracing, and OpenTelemetry guides — Prometheus's pull-based model and purpose-built metric types (especially histograms, and the histogram_quantile percentile analysis they enable) give the aggregate, "how is the system behaving" view that traces and logs individually can't provide efficiently, while Grafana turns that data into dashboards and, via Alertmanager, into genuinely actionable alerts.

The disciplines that make this stack valuable rather than noisy echo the same themes across this series' observability guides: keep high-cardinality detail out of metrics and in traces/logs where it belongs, alert on user-facing symptoms rather than every possible internal cause, and treat dashboards and alerting rules as version-controlled configuration rather than manually-maintained UI state. Done well, Prometheus and Grafana complete a genuinely coherent observability stack — one where a metric-driven alert leads naturally into a specific trace, which leads naturally into the exact correlated log lines that explain precisely what happened.

Found this useful? Feel free to star the repo, open an issue with corrections, or share the cardinality explosion that taught you to keep customer IDs out of metric labels.