Dagster doesn't expose a /metrics endpoint out of the box. The officially documented way to get Dagster metrics into Prometheus is the dagster-prometheus resource, which pushes metrics to a Pushgateway from inside a run.
That works, but it has a structural blind spot: the push only happens if code inside the run gets to call it.
- A run that's OOM-killed, or crashes before reaching the push call, never reports anything. The failure is silent from Prometheus's point of view.
- A run sitting in
QUEUEDbecause of a run-queue concurrency limit hasn't started user code yet, so it can't push either — you can't see queue backlog forming. - Prometheus's own docs are explicit that Pushgateway isn't meant to be a general substitute for pull-based scraping, only for short-lived batch jobs that genuinely can't be scraped.
So I built dagster-prometheus-exporter: a small standalone Go binary that polls Dagster's GraphQL API on an interval and derives metrics from whatever state Dagster itself already has — including runs that never got a chance to push anything.
Architecture
Dagster (GraphQL) <--poll-- exporter --scrape--> Prometheus --> Grafana
The exporter keeps in-memory state, not an external store. Scraping (writing state) and serving /metrics (reading state) are decoupled, so if a GraphQL call is slow or fails, /metrics still serves the last-known state instead of breaking. Completed runs are fetched incrementally (watermark-based, not a full rescan every time), so the cost doesn't grow with total run history.
What it currently covers
| Metric | What it answers |
|---|---|
dagster_active_runs |
How many runs are queued/starting/started, per job |
dagster_active_run_duration_seconds |
How long the oldest active run in a job has been stuck there — useful for spotting stalls |
dagster_completed_runs_total / dagster_last_run_info / dagster_last_run_duration_seconds
|
Success/failure counts and timing for completed runs |
dagster_run_queue_concurrency_key_backlog |
Queue backlog per dagster/concurrency_key tag — this is the one that needed the most digging, since instance.concurrencyLimits in the GraphQL schema looks like the answer but is actually a separate op-pool concurrency mechanism and reports 0 regardless of run-queue backlog. Ended up reading each queued run's own tags instead. |
dagster_code_location_load_error |
Whether a code location is currently failing to load (e.g. broken import) — independent of job-level metrics, since a broken location can't be inferred from run counts |
dagster_schedule_status / dagster_schedule_last_tick_status
|
Whether a schedule is running, and its last tick outcome |
dagster_sensor_status / dagster_sensor_last_tick_status
|
Same, for sensors |
dagster_exporter_build_info |
Which version/commit is actually running — handy once you have more than one exporter pod |
Full label reference is in the README.
Trying it out
git clone https://github.com/HirofumiTsuda/dagster-prometheus-exporter.git
cd dagster-prometheus-exporter
docker compose up --build
brings up Dagster + the exporter + Prometheus + a pre-provisioned Grafana dashboard together. For an existing Dagster deployment, there's a published image and a Helm chart:
docker run -p 9101:9101 -e DAGSTER_GRAPHQL_ENDPOINT=http://dagster:3000/graphql \
ghcr.io/hirofumitsuda/dagster-prometheus-exporter:0.2.0
helm install my-dagster-exporter oci://ghcr.io/hirofumitsuda/charts/dagster-prometheus-exporter \
--version 0.1.3 \
--set env.DAGSTER_GRAPHQL_ENDPOINT=http://dagster-webserver.dagster.svc.cluster.local/graphql
What's not covered yet
Asset materialization status. I've looked into the GraphQL shape for it (assetNodes + assetsLatestInfo) and the same kind of gotcha as the concurrency backlog shows up: AssetNode.assetMaterializations only records successful materializations, so detecting a failed one means cross-referencing assetsLatestInfo.latestRun.status instead. Tracked in #56 if anyone wants to compare notes.
Repo: https://github.com/HirofumiTsuda/dagster-prometheus-exporter — issues and PRs welcome.
