A payment service can pass every unit test it has and still lose money. The failures that matter in production are not about one request being wrong: a provider webhook delivered twice, two workers picking up the same payout row, a ledger that drifts by a cent until somebody closes the books and notices. They are about two correct requests colliding, or one correct request arriving again.
We have built payment processing, virtual account ledgering, and reconciliation flows across fintech and crypto for years, and one pattern holds everywhere: money bugs are found by accountants, not by monitoring. That is what makes them expensive. A 500 appears on a dashboard within a minute. A double payout appears in a bank statement, after the money has left. Rolling back the deploy restores the code, not the transfer.
So the money path gets a different test standard than the rest of the service. Four tests, all of them against a real database. This is what they look like, and what our review gate rejects when they are missing.
No mocked database on a money path
Money correctness does not live in application code. It lives in constraints, transaction boundaries, and lock semantics. A unique index is what actually makes an idempotency key idempotent. FOR UPDATE SKIP LOCKED is what actually makes a queue worker safe. A mocked session tests our belief about Postgres, and our belief is exactly the thing that is wrong when this class of bug ships.
So the suite starts a real database on the same major version as production. Everything below assumes that fixture.
import os
import pytest
from sqlalchemy import create_engine, text
@pytest.fixture(scope='session')
def engine():
return create_engine(os.environ['TEST_DATABASE_URL'], future=True)
@pytest.fixture
def db(engine):
with engine.begin() as conn:
conn.execute(text(
'TRUNCATE ledger_entries, transfers, payout_jobs RESTART IDENTITY CASCADE'
))
with engine.connect() as conn:
yield conn
Test 1: the ledger cannot drift
Every movement of money is written as balanced entries, and no code path is allowed to write a single-sided one. The test is not a test at all in the usual sense; it is an assertion that runs after every test in the suite, so any new feature that touches the ledger inherits it for free.
@pytest.fixture(autouse=True)
def ledger_stays_balanced(db):
yield
unbalanced = db.execute(text('''
SELECT transfer_id, currency, SUM(amount) AS delta
FROM ledger_entries
GROUP BY transfer_id, currency
HAVING SUM(amount) <> 0
''')).all()
assert not unbalanced, f'unbalanced transfers: {unbalanced}'
Two details carry most of the value. The grouping includes currency, because the first cross-currency feature is where a naive invariant quietly stops meaning anything. And amount is an exact numeric type, never a float. A float column is an automatic rejection at review; the invariant above would fail on rounding noise long before a human noticed the balances were wrong, which is the good case, and would pass while being subtly wrong in the bad one.
This fixture catches a specific and common mistake: a refund, chargeback, or fee handler that credits one account and forgets the counter-entry because the happy-path test only checked the user's visible balance.
Test 2: the same event twice
Providers retry. Queues redeliver. Operators click twice. Any endpoint that moves money has to be safe on the second delivery, and the only correct way to enforce that is a unique constraint on the external event identity plus a handler that treats the violation as success.
def test_webhook_is_idempotent(client, db):
payload = provider_callback(external_id='pay_1', amount='100.00', status='captured')
first = client.post('/webhooks/provider', json=payload)
second = client.post('/webhooks/provider', json=payload)
assert first.status_code == 200
assert second.status_code == 200
assert entry_count(db, external_id='pay_1') == 2 # one balanced pair, not two
What the gate rejects here is the pre-check: SELECT to see whether the event was already processed, then INSERT if not. It passes this test and fails in production, because the two deliveries arrive concurrently and both read an empty table. The constraint has to be in the schema, and the handler has to catch the integrity error rather than avoid it. A test that only sends the duplicate sequentially does not prove that, which is why the next test exists.
Test 3: two workers, one payout
Outbound money is a queue, and the interesting question is what happens when the queue is consumed twice at once, whether because of a scaling event, a redeploy overlap, or a stuck job being requeued while the original worker is still alive.
from concurrent.futures import ThreadPoolExecutor
def claim_payout(engine, payout_id):
with engine.begin() as conn:
claimed = conn.execute(text('''
SELECT id FROM payout_jobs
WHERE id = :id AND state = 'pending'
FOR UPDATE SKIP LOCKED
'''), {'id': payout_id}).first()
if claimed is None:
return False
conn.execute(text('''
UPDATE payout_jobs SET state = 'sent' WHERE id = :id
'''), {'id': payout_id})
return True
def test_only_one_worker_claims_a_payout(engine, pending_payout):
with ThreadPoolExecutor(max_workers=2) as pool:
results = list(pool.map(lambda _: claim_payout(engine, pending_payout.id), range(2)))
assert sorted(results) == [False, True]
The test needs two real connections, because that is the whole point: a single-session test cannot observe a lock. SKIP LOCKED rather than a plain FOR UPDATE is deliberate. With a plain lock the second worker waits, then reads the row after the state change and has to re-check the state anyway; with SKIP LOCKED the loser is told immediately and moves to the next job, which is what keeps a payout batch flowing when one row is slow.
Test 4: reconciliation classifies, it does not heal
The fourth test is the one teams skip, and it is the one that pays for itself. Every money system eventually disagrees with its counterparty: a deposit arrives with a reference that matches no open invoice, an amount comes back short by the provider's fee, a capture is reported that we have no record of requesting. The reconciliation job's job is to name the disagreement, not to fix it.
@pytest.mark.parametrize('statement_row, expected_kind', [
(statement(external_id='pay_1', amount='100.00'), 'matched'),
(statement(external_id='pay_1', amount='99.00'), 'amount_mismatch'),
(statement(external_id='pay_unknown', amount='50.00'), 'unattributed'),
(statement(external_id='pay_1', amount='100.00', duplicate=True), 'duplicate'),
])
def test_reconciliation_classifies_instead_of_healing(db, captured_payment, statement_row, expected_kind):
before = entry_count(db)
report = reconcile(db, rows=[statement_row])
assert report.single().kind == expected_kind
assert entry_count(db) == before
The last assertion is the important one. Reconciliation writes discrepancy records and nothing else; it never posts a correcting entry on its own. A self-healing reconciler is the worst possible component to own a money path, because it turns a visible mismatch into an invisible adjustment, and the next mismatch is then reconciled against an already-adjusted state. When a case needs correcting, a human decides and the correction goes through the same ledger code as every other movement, with the same invariant applied.
The unattributed case deserves its own fixture rather than being folded into the mismatch bucket. Attribution by payment requisites is where inbound money actually goes wrong in production: the sender edits the reference, or pays from a different account than the one on file. Classifying that separately is what lets a compliance or finance operator work a queue instead of reading raw statements.
What it costs to run
Honestly: it is slower than a mocked suite. The CI job starts a database container, migrations run before the session, and the concurrency tests spend real time waiting on real locks. Money-path tests run in minutes rather than seconds, and we parallelise them with a schema per worker so the truncation fixture stays isolated.
One rule holds the whole thing together: we never quarantine a flaky money test. A nondeterministic result in this suite is not test infrastructure being annoying, it is the race we were trying to find, surfacing on the cheap side of production. Every time we have chased one instead of retrying it, there was something real underneath.
The gate
A change that touches the money path does not pass review with a float column for an amount, a mocked database, an idempotency pre-check instead of a constraint, a single-session concurrency test, or a reconciler that writes ledger entries. None of these are exotic engineering. They are the four questions we have learned to ask before the code goes anywhere near a real account, because it is the only stage where the answers are still cheap.
Originally published on shipmindlabs.com — where we write about payment systems, infrastructure and marketplace backends.
