Our payment provider tells us about refunds and chargebacks by webhook. In June we moved every inbound callback from our main API host to a dedicated hooks host behind its own gateway. We left the old path answering with a 301 to the new one, as we would for a web page, and put asking partners to update their configuration on a list for later.
The provider's HTTP client follows redirects. When it receives a 301 or a 302 in reply to a POST, it does what a lot of clients have done for a very long time and the specification tolerates: it repeats the request as a GET, without the body. So the hooks host received a GET on the refund path, with no payload and no signature header. Our framework routed GET on that path to the verification handler the provider calls once, when an endpoint is registered, and which answers 200 with an echo of the challenge. The provider saw 200 and marked every event delivered.
That went on for twelve days and one thousand three hundred and forty events. Nothing errored. Our logs showed a verification request every few minutes, which reads as nothing. Month end reconciliation found it, when finance compared the provider's settlement report with our ledger and the refunds did not match.
Updating the URL in the provider's dashboard took five minutes, twelve days late. We replayed the missing events through their events API, which is the part that made this recoverable. Then the structural changes. The old path proxies requests instead of redirecting them, logs which caller used it, and will return 410 after ninety days, by which time the log should be empty. The webhook handler accepts only POST with a valid signature, and the verification handler lives on its own path, so no request can end up in the wrong one. A daily job pulls the provider's event list and compares it with what we received, alerting on anything missing for more than an hour. And for every partner that calls us we now record which URLs they have configured.
A redirect is an instruction for a browser, and browsers agree on what to do with it. A machine caller does whatever its HTTP library settled on long ago. Moving an endpoint other systems call means changing their configuration, not leaving a signpost at the old address.
– Sergey Shinder
