Everyone building a trading bot pours their energy into the strategy. The entry logic, the indicators, the pretty backtest curve. Then they wire it to a live broker API and it quietly falls apart — not because the strategy was wrong, but because the layer underneath it lied about what actually happened in the market.
I've spent more time on that layer than on any strategy, so here are the four places it breaks, and how I stop it from breaking. Examples are on the Interactive Brokers API, but the failure modes are the same on any broker.
The four ways it goes wrong
1. The duplicate callback. The API sends you the same execution twice. Naive code adds the fill twice, and now your bot thinks it owns 200 shares when it owns 100.
2. The reconnect replay. Your socket drops for four seconds. On reconnect, the broker helpfully replays recent fills. If you treat them as new, you double-count everything that happened around the disconnect — which is exactly when you can least afford to be wrong about your position.
3. The out-of-order partial fill. A large order fills in pieces. The messages don't always arrive in order. If you just overwrite state with the latest message, a late-arriving early fill can rewind your average price or your filled quantity.
4. The stale status that rewinds a finished order. You get a Filled. Then, a beat later, a delayed Submitted for the same order shows up. Trust it and you've just un-filled a completed trade in your own books.
Notice what none of these do: crash. No exception, no stack trace, no red text. They silently corrupt state — phantom positions, double orders, wrong P&L — and you find out when real money is already on the wrong side. That's what makes them the expensive kind of bug.
The fix is boring on purpose
You don't beat these with clever code. You beat them with a few strict rules and a design that lets you prove they hold.
Make executions idempotent. Every fill has a unique id (execId on IB). Key on it. If you've seen it, it's a no-op. Replays and duplicates just do nothing.
Take truth from executions, not from status. Filled quantity is the sum of real executions you've actually seen — never a number you trust blindly from a single orderStatus. And state only moves forward: once an order is terminal, no late message drags it backward.
Aggregate partial fills properly. Accumulate the pieces and carry a quantity-weighted average price, so order matters less and a late fragment can't distort your basis.
Reconcile — don't overwrite. Periodically pull the broker's own open-orders and positions snapshot and compare it to your local truth. If they disagree, report the drift. Don't silently overwrite one with the other — the disagreement is the signal, and hiding it is how small errors become big ones.
Keep the fragile logic pure. The part that does dedup, aggregation and reconciliation has no broker I/O in it at all. That's deliberate: it means you can run the nasty scenarios deterministically in CI instead of hoping they behave on a live line.
Proof, not vibes
I put all of this in a small, tested engine. The offline suite fires every failure mode above at it and asserts the state stays sane — 8/8 checks passing: duplicate execution ignored, reconnect replay idempotent, partial fills aggregated to a quantity-weighted average, stale and stray statuses rejected, and reconciliation flagging any order/position drift.
Then I ran it against a live IBKR paper account over the TWS API — strictly read-only, it observes and reconciles, it never places orders. I placed a market order by hand in TWS and let the engine reconstruct the truth on its own: it tracked every transition (Submitted then Filled), summed the fill, derived the position, and matched the broker's own snapshot exactly — reconciliation clean.
The point
A strategy decides what to do. The execution layer decides whether what you think happened matches what actually happened. Get the second part wrong and the first part doesn't matter — you're trading on a fiction.
The whole thing is open. Read it instead of taking my word for it:
https://github.com/lungucristian1980-hub/ibkr-order-state-engine
I build execution reliability like this for a living — reconnect-safe order handling, kill-switches, reconciliation, clean handovers — at Quiet Machines. If your bot does something it shouldn't when the connection drops, that's the work I do. You bring the strategy; I make sure the plumbing tells the truth.
— Cristian Lungu · https://contra.com/lungu_cristian_d7nm4wbh
