Every cross-system integration I have debugged in the last two years failed the same way, and it was never the way the logs suggested.
The logs said success. The API returned 200. The retry counter was zero. And the field in the target system still held the old value.
This is not an edge case. It is the default behaviour of every major CRM and ticketing API, and if your pipeline treats a 2xx as proof of a write, you have a silent data-loss bug you have not found yet.
A 2xx means "accepted", not "stored"
That distinction sounds pedantic until it costs you a quarter of pipeline data. Here are the five mechanisms I keep hitting, roughly in order of how often they bite.
1. Partial success inside a successful response. Bulk and composite endpoints are the worst offenders. Salesforce's composite API with allOrNone: false returns 200 OK with a body containing per-record failures. If you check the HTTP status and move on, you have just discarded the error.
# This is the bug.
r = requests.post(composite_url, json=payload, headers=h)
r.raise_for_status() # 200. Looks fine.
log.info("synced %d records", len(payload["records"]))
# The failures were in here all along.
for result in r.json()["results"]:
if not result["success"]:
log.error("record %s: %s", result.get("id"), result["errors"])
2. Downstream automation overwrites you milliseconds later. Your write lands. Then a workflow rule, a Flow, or a HubSpot workflow fires on that same record change and sets the field back, or to something else entirely. Your write succeeded and was immediately undone. Nothing in your logs will show it, because from your side nothing went wrong.
3. Duplicate and merge rules move the record out from under you. You write to record A. A dedupe rule merges A into B. Your value is now either on a record you were not targeting or gone. The API still told you 200, because at the moment it answered, it was true.
4. Field-level permissions drop fields silently. The integration user lacks edit access on one field in a twelve-field update. Depending on the endpoint, you get a 200 and eleven fields written. The twelfth is simply absent from the result, not reported as an error.
5. Eventual consistency means read-your-write is not guaranteed. Read back too fast and you get the pre-write value, which sends you chasing a bug that does not exist. This one produces false alarms rather than silent failures, which is why it is the least dangerous and the most annoying.
The fix is boring: read it back
def write_and_verify(client, record_id, fields, *, attempts=3, delay=0.5):
"""Write, then confirm the target system actually holds the values."""
client.update(record_id, fields)
for attempt in range(attempts):
time.sleep(delay * (2 ** attempt)) # back off for #5
actual = client.get(record_id, fields.keys())
drift = {k: (v, actual.get(k)) for k, v in fields.items()
if actual.get(k) != v}
if not drift:
return Verified(record_id, fields)
# Do NOT retry the write. Something is actively rejecting or
# overwriting it, and hammering the endpoint will not change that.
raise VerificationFailed(record_id, drift)
Three things matter more than the code.
Verify the fields you care about, not all of them. A full record comparison will drift constantly on system-managed fields like LastModifiedDate and produce alerts nobody reads. Pick the fields whose wrongness would actually cost something.
Never auto-retry a failed verification. A failed verify means something is rejecting or overwriting your write. Retrying makes that happen again, faster. Escalate to a human with the diff: expected, actual, record, timestamp. A three-line diff is a five-minute fix. A retry loop is a Tuesday.
Log the diff, not the outcome. verification failed tells you nothing at 2am. close_date expected 2026-03-31, actual 2026-06-30, record 0061x…, 340ms after write tells you a Flow is overwriting close dates, which is a completely different problem than a permissions error.
Why the verify step is the one that gets skipped
Because it is the only stage with no visible output when it works. Trigger, join, transform and write all produce something you can point at in a demo. Verification produces silence, and silence does not demo well.
It is also the stage that separates an automation from a system. A zap moves a record. A system proves the record arrived and tells you when it did not. That gap is where most "our data is a mess" complaints actually originate, long after anyone remembers which integration caused it.
If you want the wider version of this argument, the discipline of building revenue workflows this way now has a name and a job market attached to it: GTM engineering. The write-verify pattern above is stage five of five in how those workflows get built, and it is the stage almost every tutorial leaves out.
I build these systems at Mindlyft. Half of what we ship is not the automation. It is the proof that the automation did what it said.
What is the worst silent-write failure you have found? I am collecting the mechanisms, and I suspect five is not the complete list.
