Quick context if you didn't see the earlier post: opti-pipe compares your pipeline's config against its actual run metrics and tells you what's over/under-provisioned — executor memory, shuffle partitions, cluster size — with the reasoning shown, not a black-box score. Rule-based, not ML, on purpose: I can point at the exact threshold that fired.
dbt had a real integration from day one (reads your actual target/run_results.json). Spark didn't — the rules existed, fully tested, but ran against hand-typed fixtures because there was no ingestion path. That's fixed now.
Why an event log, not a live REST API reader
Two real options existed: a live reader against Spark's REST API / History Server, or an uploaded event log (the file Spark writes when spark.eventLog.enabled=true). The live path is more precise but needs a reachable driver or History Server — friction most solo setups don't want. The event log is self-serve, same shape as dbt's upload, ships without needing cluster connectivity. Picked the second one; the live reader stays on the roadmap.
The parsing itself
parse_spark_event_log walks the real listener event stream — SparkListenerApplicationStart/End, SparkListenerEnvironmentUpdate, SparkListenerStageSubmitted, SparkListenerTaskEnd with its nested Task Metrics/Task End Reason — line by line (it's JSON-lines, one event per line, not a single document). From it: task runtime, GC pause time, record counts, OOM detection via matching Task End Reason against OOM error signatures, queue delay from stage-submission-vs-task-launch timestamps, and heap usage as a percentage of configured executor memory.
A bug worth mentioning
Spark's event log has zero concept of CPU utilization — no listener emits it, it's an infra-level metric. My first pass defaulted that field to 0.0. That's a real bug: 0% CPU reads to the "overprovisioned cluster" rule as genuinely idle, so it would've spuriously recommended shrinking your cluster on every single real import regardless of actual load. Fixed with an explicit CPU_UTIL_UNAVAILABLE sentinel that the aggregation and rule logic both treat as "no data," not "0%." Only surfaced because I stopped testing against fixtures I'd hand-written with plausible non-zero values and pointed it at something real.
Try it against your own Spark job (upload guide included, walks through finding the event log on EMR/Databricks/Kubernetes/standalone): opti-pipe.onrender.com. Feedback — especially "this recommendation is wrong" — is genuinely the most useful thing anyone can send me right now.
