Six weeks of integration work with a new settlement provider, a sandbox we could hit as often as we liked, a certification checklist we passed in full, and a go-live at nine on a Monday that rejected a hundred percent of our requests for thirty-five minutes.
Three differences, none of them in any document we had been given.
Their production requires a merchant reference header on every request. The sandbox treats it as optional, and their API reference marks it optional too, because the reference describes the protocol and the requirement lives in the acquiring rules for our region. We had never sent it. Production answered with a generic validation error and an error code that resolved, in their public documentation, to "invalid request".
Their production enforces twenty-five requests per second per merchant. The sandbox enforces nothing. Our cut-over began with a backfill of the weekend's transactions at roughly two hundred a second, which meant that once we fixed the header we spent the next twenty minutes being rate limited instead.
And their production checks the client certificate against a list of source addresses. We had registered our office egress address during integration and never the production NAT gateways, which nobody noticed because the sandbox does not check.
What I would do differently is not about any of those three specifically. We treated passing certification as evidence that the integration worked, when certification is a test of the parts the provider thought to model. Before the next go-live we asked for something nobody had offered: a written list of behavioural differences between their environments. They had one internally. It ran to two pages and would have caught two of our three.
Then we changed the cut-over itself. New integrations now run a pilot of one percent of real production traffic for a week behind a flag, with the old path still authoritative and a comparison of both results logged. Rejection rate in the first hour of any new integration is alerted on, and the flag rolls itself back at five percent. And once we have real production responses, we record them and replay them as contract tests, which is the only test data that has ever been honest about their rules.
A sandbox is a model of an integration built by the people who wrote it. It is faithful about everything they remembered.
– Sergey Shinder
