Kill the Full Refresh: Implementing Delta Lake Change Data Feed for Silver Pipelines

The "Bronze-to-Silver full-refresh" is the industry’s most expensive lie. Every time I see a pipeline re-processing ten terabytes of historical patient records just to capture the five hundred updates from the last hour, I see money literally burning in the cloud provider’s furnace.

Why I chose this topic: In my last three roles, I’ve had to fix "silver" layers that collapsed under their own weight because they relied on full-table overwrites. Change Data Feed (CDF) is the only way to build pipelines that actually scale, and I'm tired of seeing engineers treat it like it’s too "complex" to touch.

Your current pipeline probably looks like this: A massive batch job kicks off at 2 AM, scans the entire Bronze storage, executes a deduplication window function over the last 90 days of partition data, and writes a massive OVERWRITE to the Silver table. It works fine when your data is small. But once you hit the 10-terabyte mark, your Spark driver starts throwing OOM errors, and your finance department starts asking why the Databricks bill for the "ETL Sandbox" is higher than the marketing budget.

You’re not doing data engineering; you’re doing data laundering. You are taking raw data, washing it in a massive compute cycle, and outputting the same data you already had, just slightly cleaner. It is a loop of technical debt that compounds every time a new source system is ingested.

The real problem

The problem isn't your code; it's the paradigm of batch-oriented thinking. We treat tables like immutable blobs that must be replaced to be updated. When you run a full refresh, you are essentially telling the cloud provider, "Please ignore the fact that 99% of this data hasn't changed, and charge me for the privilege of re-calculating it."

Delta Lake’s Change Data Feed (CDF) changes the contract. Instead of asking "What is the state of the world?", you ask "What happened since the last time I checked?" By enabling CDF, Delta writes a sidecar log of row-level changes—inserts, updates, and deletes. You can consume these changes incrementally, transforming only the delta, and merging them into your downstream tables. This is how you go from an 8-hour batch job to a 5-minute incremental micro-batch.

Photo by Winston Chen on Unsplash

Step 1: Enable the change feed on your bronze tables

Before you can do anything, you need to turn the feature on. This isn't retroactive, so do it today. If you have a legacy table, you’ll need to set the property and then perform a one-time rewrite (the last full-refresh you'll ever do).

ALTER TABLE bronze_patient_data 
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

-- If the table is already massive, run this to populate the change log
-- effectively from this point forward.
INSERT OVERWRITE TABLE bronze_patient_data 
SELECT * FROM bronze_patient_data;

Once this property is set, Delta starts tracking the operations in the _change_data folder hidden within your table path. Note that this increases storage slightly—usually by about 5–10%—but the cost of that storage is pennies compared to the cost of your redundant compute.

Step 2: Read the changes using Spark structured streaming

Now, instead of reading the whole table, you point your reader at the change log. You don't need to manually calculate watermarks or manage state if you stick to the readChangeFeed API. This is where the magic happens. You treat the changes as a stream, applying your logic—data type casting, PII masking, and deduplication—only to the records that actually changed.

# Read the feed as a stream
df_changes = spark.readStream \
    .format("delta") \
    .option("readChangeFeed", "true") \
    .option("startingVersion", "150") \
    .table("bronze_patient_data")

# Perform your business logic here
# Note: Since this is an incremental stream, 
# your transformations must be idempotent.
silver_df = df_changes.select("id", "patient_name", "status", "_change_type")

# Write to your silver table
silver_df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/mnt/delta/checkpoints/silver_patient") \
    .table("silver_patient_data")

The _change_type column is your best friend here. It will tell you if the row was an insert, update_preimage, update_postimage, or delete. You don't need to guess why a row appeared; the audit trail is built-in.

Step 3: Implement the upsert logic in silver

Your Silver layer shouldn't just be an append-only log; it should be a mirror of the latest state of your entities. Since you are reading changes, you need to merge those changes into your final Silver table. This is where MERGE INTO becomes the backbone of your architecture.

def upsert_to_silver(microBatchDF, batchId):
    microBatchDF.createOrReplaceTempView("updates")

    # Merge the micro-batch into the target table
    microBatchDF._jdf.sparkSession().sql("""
        MERGE INTO silver_patient_data target
        USING updates source
        ON target.id = source.id
        WHEN MATCHED AND source._change_type IN ('update_postimage', 'insert') THEN
            UPDATE SET *
        WHEN NOT MATCHED THEN
            INSERT *
        WHEN MATCHED AND source._change_type = 'delete' THEN
            DELETE
    """)

# Execute using foreachBatch
silver_df.writeStream \
    .foreachBatch(upsert_to_silver) \
    .option("checkpointLocation", "/checkpoints/silver_patient") \
    .start()

By using foreachBatch, you gain full control over the MERGE logic. You handle the update_postimage for existing IDs and handle the delete events cleanly. This keeps your Silver table perfectly in sync with the Bronze source without ever scanning the entire dataset.

Photo by August Phlieger on Unsplash

Lessons learned from production

I’ve seen this setup fail in predictable ways. Here is how you avoid the midnight pager duty:

• The Checkpoint Trap: Never, ever delete your checkpoint directory. If you do, you lose the state of your stream. You will be forced to reprocess the entire table from version zero, which is exactly what we are trying to avoid. Keep your checkpoints in a highly available, versioned S3 bucket with lifecycle policies that prevent accidental deletion.
• The Deletion Lag: If your source systems perform hard deletes and you don't have a reliable way to capture those deletes in the Bronze layer, CDF won't help you with the Silver layer. Ensure your upstream integration (e.g., Debezium, Fivetran) is configured to propagate DELETE events. If you only see INSERTs, your Silver layer will become a "hall of mirrors" for deleted records.
• Schema Evolution: CDF is sensitive to schema changes. If you add a column to your Bronze table, your MERGE statement in the Silver writer might fail if you are using UPDATE SET *. Be explicit in your MERGE column mappings if you expect your Bronze schema to evolve frequently.
• The "Too Many Files" Problem: Because you are streaming, you might end up with millions of tiny files in your Silver table. Run OPTIMIZE and VACUUM on a schedule (e.g., weekly), but do it outside of your streaming window. If you try to run OPTIMIZE while a stream is writing to the same partition, you will hit lock contention errors.

Conclusion

Change Data Feed isn't a "nice to have" feature for high-scale platforms; it's a survival requirement. Once you stop treating your data lake like a collection of static files and start treating it like a stream of events, your infra costs will drop, and your pipeline reliability will skyrocket. It requires more upfront design than a simple INSERT OVERWRITE, but that's the difference between a junior engineer and a senior architect.

Try it: Enable delta.enableChangeDataFeed on your smallest, most annoying production table today. Use spark.read.format("delta").option("readChangeFeed", "true") in a local notebook to see exactly what those updates look like. Once you see the audit log, you’ll never go back to full-table refreshes again.

Tags: #datalake #spark #delta #engineering

Cover photo by Maria Ionova on Unsplash.