I spent the last few months tuning a production pipeline on AWS Glue and PySpark: Oracle to S3 ingestion, then packaging, encryption, and transfer to a downstream analytics platform. The goal was to cut runtime without touching business logic, and across four workstreams it came to roughly a 56% reduction end to end.
Here are the four changes that moved the needle, then two investigations that never became wins but taught me more. All numbers are from my own runs, checked against logs, with anything environment-specific genericized.
Ingestion: crossing a concurrency wave boundary
The 63-table ingestion ran through a Step Functions Map at concurrency 30. With 63 items that forces three waves, and the wall clock is the last item to finish, not the slowest job. The run sat at 13:55.
Raising Map concurrency to 50 dropped it to two waves and about 8 minutes. Waves are a cliff, so the payoff is in crossing a boundary, not nudging the number. Largest-first packing helps too (start the heavy tables in wave one so none is stuck in the final wave).
Publish: stop rewriting the same bytes
The publish layer wrote output, then rewrote it: a rename pass, a newline pass, and a standardize pass, each re-reading the whole thing. It was redoing on the driver what the executors had already produced. Writing the final bytes once, straight from the executors, and parallelizing the reconciliation instead of running it serially, took that stage from about 25 minutes to 7. No logic changed, just how many times the data got touched.
Outbound packaging and encryption
The outbound job zips files, GPG-encrypts each ZIP, and uploads with SSE-KMS. It ran on Glue Spark with 6 DPUs on G.2X, and it was slow for a dumb reason: everything ran serially on the driver, with Spark only used to build one logging DataFrame at the end. Six DPUs of executors sat idle while the driver zipped one file at a time.
Compression was the CPU-bound phase, so I threaded just that and dropped DEFLATE to level 1:
with ThreadPoolExecutor(max_workers=N) as pool:
pool.map(lambda part: build_zip(part, compresslevel=1), parts)
Compression dropped from about 9 minutes to 4. Level 1 does make the zips slightly larger, but that was an easy trade here since runtime was the constraint, not archive size. It's worth knowing which side you're optimizing for: if a downstream had tight storage or bandwidth limits, I'd push the level back up and accept the slower compression. For this pipeline the faster, slightly bigger output won. Giving each batch its own temp key handled the rest.
Encryption had two traps. GPG compresses before encrypting by default, but the input was already zipped, so it burned CPU re-compressing incompressible data and the output actually grew (one batch went 5246 MB to 5312 MB). --compress-algo none fixes it. Then parallelizing encryption the way I did compression OOM-killed the driver, because each multi-GB ZIP was read fully into memory alongside an armored copy. I kept encryption serial as the safe landing. The proper fix, staged separately, streams file-to-file with gpg.encrypt_file so no whole ZIP is in RAM, gives each call its own GNUPGHOME, and writes unarmored binary (about 33% smaller). The blocker was never concurrency, it was the memory model.
SFTP transfer: when Spark is the wrong tool
A separate job pushed large files to an external SFTP endpoint on Glue Spark (6 DPU, G.2X) and did zero distributed work. It was bandwidth-bound at about 11 MB/s on roughly 5 GB files, and raising connector concurrency did nothing, because the ceiling is throughput, not session overhead. Moving it to a Glue Python Shell job at 1 DPU kept the same throughput and cut annual cost from about $63 to $11 at $0.44/DPU-hour. A single-stream, IO-bound transfer is a script, not distributed compute.
Two investigations that never became wins
A JDBC partition that made every connection full-scan the table. The ingestion runtime made no sense: 8.9M rows took 8 minutes, but 25M in prod ran 2h45m and climbing. I assumed skew, but the bucket-distribution query came back near-uniform, which killed that theory. The real cause was that the partition column is a computed expression Oracle can't index, so every connection full-scans the whole window just to find its slice, and at prod scale those concurrent scans blow the buffer cache and stall. It stayed a diagnosis: the fix is to partition on a prunable column, which I haven't implemented and measured yet.
A hang that fixed itself on rerun. The same job hung 2h42m one run, then finished in 14 minutes on an identical rerun. The cause was a JDBC socket read with no timeout: one connection died silently and blocked its task forever while the others finished, and Spark can't close a stage until every task returns. Layered timeouts (queryTimeout, oracle.jdbc.ReadTimeout, oracle.net.CONNECT_TIMEOUT) plus speculation break and reschedule a dead read. One trap: oracle.net.CONNECT_TIMEOUT is milliseconds as a connection property but seconds bare in the URL.
The through-line
The wins were all mechanism, not logic: cross a concurrency boundary, stop touching the same bytes twice, parallelize the phase that fits the resource you have headroom in, and right-size the runtime to the work. The investigations were the other half of the lesson, that the bottleneck is almost never where you first guess. Skew looked obvious until one query killed it, and half the real story lived in the source database where the Spark UI can't see it. Mostly it's proving what the bottleneck actually is before changing anything.
