Why your Python script is slow: two lines took it 3.71 s to 1.20 s

One script, one million rows, one question: where does the time go? This is the write-up of the video above. Every number comes from the same measurement files the video used, all on one Mac mini (M4 Pro, 48 GB) with Python 3.14.6. The workload is synthetic: a deterministic generator (seed 42) writes a sales CSV so anyone can reproduce the run.

The result first

report.py (baseline): 3.71 s — single run, timed inside the script
report_cached.py (two lines added): 1.20 s — single run, timed inside the script
cmp report.txt report2.txt → identical, byte for byte

About 3.1× faster, same output, two lines changed. The interesting part is how we found the two lines, and where the trick stops working.

What the script does

report.py reads a CSV of 1,000,000 sales rows, parses each row's date string, totals revenue by month and region, and writes a text report. We measure wall time, and the report must stay identical.

$ python3 report.py sales.csv report.txt
wrote report.txt: 1,000,000 rows in 3.71 s

Not slow enough to notice once. Slow enough to hate inside a loop.

Ask Python where the time goes

$ python3 -m cProfile -s tottime report.py sales.csv report.txt | head -12
         32016593 function calls (32016422 primitive calls) in 8.440 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1000000    3.045    0.000    5.556    0.000 _strptime.py:516(_strptime)
  1000001    0.859    0.000    1.411    0.000 csv.py:173(__next__)
  1000000    0.465    0.000    6.021    0.000 _strptime.py:812(_strptime_datetime_datetime)
        1    0.451    0.451    6.947    6.947 report.py:12(summarize)

The top line is the date parser, strptime: one million calls, 3.045 s of self-time, 36% of the profiled run. Two cautions that the profiler itself teaches: self-time excludes subcalls, and profiling changes the timing (8.44 s here versus 3.71 s unprofiled). Read the share, not the seconds. The share points at parsing.

A profile says which function, not why

So look at the input.

$ python3 count_distinct.py sales.csv
1,000,000 rows, 365 distinct dates

One million calls, 365 distinct strings. The same date string gets parsed about 2,740 times.

The two-line change

 from collections import defaultdict
+from functools import lru_cache

+@lru_cache(maxsize=None)
 def parse_date(s):
     return datetime.strptime(s, "%Y-%m-%d %H:%M:%S")

Prediction before running it: strptime executes 365 times instead of 1,000,000, and the report does not change.

Did the prediction hold?

$ python3 report_cached.py sales.csv report2.txt
wrote report2.txt: 1,000,000 rows in 1.20 s
$ cmp report.txt report2.txt && echo identical
identical
$ python3 cache_check.py sales.csv
CacheInfo(hits=999635, misses=365, maxsize=None, currsize=365)

365 misses, 999,635 hits. The cache holds exactly 365 entries.

Why it works

strptime is expensive; a dictionary lookup is cheap. With 365 distinct inputs the cache misses 365 times and hits everywhere else. Which raises the real test: what if every date is different?

The limit: when inputs don't repeat

Three data sets, five runs per version, medians. These are whole-process timings (startup and shutdown included), which is why they sit above the single in-script runs at the top. Compare plain and cached within a row, not across the two timers.

$ python3 bench.py --table
distinct dates    plain   cached  speedup  same output
           365    3.96s    1.43s    2.77x  yes
        20,000    3.70s    1.49s    2.48x  yes
     1,000,000    3.76s    3.98s    0.95x  yes
1,000,000 rows · 5 runs each, median · Python 3.14.6

Here, caching paid when inputs repeated. With every row unique, the cache cost about 6% and returned nothing.

When to use it

Try caching an expensive pure function when its inputs repeat. Count the inputs before you trust the speedup.

Use when: the same strings hit a parser thousands of times; the function is pure (same input, same output); you counted, and distinct inputs ≪ calls.
Doesn't fix: unique inputs (here, about 6% slower); loading — a separate benchmark took about 0.9 s just to read the CSV and build the rows; memory — the cache keeps every distinct key.

Start by profiling your script and inspecting its most expensive functions. Check their inputs, then benchmark any change without the profiler:

python3 -m cProfile -s tottime your_script.py

Reproduce it

Code, data generator and every capture: in the pinned comment under the video

python3 make_data.py --rows 1000000 --unique 365 --out sales.csv
python3 report.py sales.csv report.txt
python3 -m cProfile -s tottime report.py sales.csv report.txt | head -12
python3 count_distinct.py sales.csv
python3 report_cached.py sales.csv report2.txt && cmp report.txt report2.txt && echo identical
python3 cache_check.py sales.csv
python3 bench.py --table

Two timing scopes appear above on purpose. The 3.71 s and 1.20 s are single measurements timed inside the script. The table reports five-run median whole-process times. They are not the same timer, and the ratios are what transfer.