Fail-Closed Is the Only Safe Default in Banking Event Systems

Your payment service processes 400 transactions per second through an event pipeline. A consumer crashes on a malformed message. The queue starts backing up. What does your system do next?

Most event-driven architectures skip the bad message, log it, and keep processing. In web applications, that default makes sense. A dropped analytics event or a failed notification is an annoyance, not a danger.

In banking, that skipped message might be the fraud check that was supposed to screen the next 10,000 transactions, or the deduplication step that should have caught a payment that already settled.

Skip-and-continue in a payment pipeline preserves the data but loses the check that was supposed to run before money moved.

In July 2012, Nationwide Building Society processed the same batch of Visa debit card transactions twice. Someone sent a file that had already run through the system again. 704,426 accounts were double-debited in a single day.

The deduplication step that should have caught the replayed batch didn't halt the pipeline. The system continued, and nearly two million duplicate transactions posted across those accounts.

Five failure modes with dollar consequences

The poison message

A malformed event hits a consumer and crashes it. Maybe a schema change shipped without updating the consumer, maybe a payload was corrupted in transit. On restart, the consumer reads from the last committed offset, hits the same message, and crashes again. The entire partition stalls.

The common fix is a dead letter queue, which routes the bad message somewhere else so processing can continue. But "somewhere else" means a payment event is now sitting in a side channel that nobody monitors in real time. If that event was a debit instruction, the credit side may have already posted. If it was a fraud alert, the transaction it flagged is still processing unchecked.

The fail-closed fix is harder. Halt the consumer, fire an alert, and don't resume until an engineer confirms the message can be reprocessed or safely skipped without financial impact. Every skipped payment event gets a reconciliation entry.

Consumer lag and stale balances

When a consumer falls behind its topic, downstream systems read data that hasn't caught up. A balance-check service queries the ledger and sees $2,400. But the consumer processing deposits is 30 seconds behind, and three deposits totaling $1,800 haven't posted yet. The real balance is $4,200.

That lag is invisible to the balance check. A withdrawal request for $3,000 gets rejected because the system thinks the account is short. Or worse, if the lag runs the other direction: a withdrawal succeeds against a balance that includes a deposit that hasn't actually settled.

A post office that delivers mail in order but sometimes runs an hour late doesn't just delay the letters. It means every decision you made based on the last letter you received used old information. If you paid a bill from a statement that was already superseded by a correction still in transit, you paid the wrong amount.

In production, consumer lag above a threshold should block any balance-dependent operation. The system should return an explicit "stale" response instead of a number when it knows the answer might be wrong.

Out-of-order events

A payment is initiated, producing an event. A fraud screening service consumes it and publishes a "cleared" event. A settlement service picks up both, but they arrive in the wrong order. The settlement service sees the payment first, checks for fraud clearance, finds none, and either waits (fail-closed) or proceeds without the check (fail-open).

Idempotency keys create a subtler problem. Two events reference the same payment but carry different states. If the older state arrives second, a naive consumer overwrites the newer state with the stale one. A payment that already settled gets marked as pending again.

Strict ordering within a partition handles some of this, but only when every service in the chain publishes to the same partition key. The moment a fraud service publishes clearance events to a different topic with different partitioning, ordering guarantees vanish.

The rebalance window

Kafka consumer groups rebalance when a consumer joins or leaves the group. During rebalancing, partition ownership shifts. There's a window where the outgoing consumer has already read a message but hasn't committed the offset, and the incoming consumer starts reading from the last committed position.

Both consumers process the same payment event. Both submit the same debit instruction to the ledger. If the ledger doesn't enforce idempotency at the write level, the account is debited twice.

The window is short, usually a few seconds. But a pipeline handling 400 transactions per second generates roughly 1,200 events in that time. Even a small fraction of those being duplicated creates a reconciliation problem that takes hours to untangle.

Fail-closed at the rebalance boundary means the ledger rejects any write that doesn't carry a unique, verified idempotency key. The consumer treats a rejected write as a signal to re-read the event and verify state before retrying.

The silent dead letter queue

Every DLQ starts as a safety valve and becomes a graveyard. Messages land there because of transient errors, schema mismatches, or downstream timeouts. Each one represents a payment event that didn't complete its normal path.

In September 2025, a payments platform discovered duplicate card settlements appearing for the same authorization. The incident ran for seven days before the duplicates were detected and offset with corrective entries. Seven days of customers seeing wrong charges, and a manual cleanup that required adding rows to undo each duplicate.

A DLQ with 200 unprocessed payment events is 200 reconciliation breaks waiting to surface. If the alerting threshold is set too high, or if there isn't one, those breaks compound silently until end-of-day reconciliation fails and an operations team starts investigating by hand.

Fail-closed means the DLQ has a hard ceiling. When it reaches a threshold, say 10 messages for a payment-critical topic, the pipeline halts until the backlog is reviewed. The ceiling is low because each message in a financial DLQ has a dollar value attached to it.

What fail-closed looks like in production

Fail-closed in practice looks like circuit breakers at each pipeline stage, not a kill switch for the entire payment system.

A balance-check service that detects consumer lag above 5 seconds returns an explicit "stale" response instead of a number. The caller decides whether to wait or reject the transaction. A settlement service that can't find a fraud clearance event within its timeout window declines the settlement rather than proceeding without the check.

The ledger is the final fail-closed boundary. Every write carries an idempotency key, and every debit has a corresponding credit. If the entries don't balance, nothing posts.

That constraint, built into the data model rather than the application logic, catches duplicates, missed events, and out-of-order state changes that the event pipeline didn't prevent.

This is why the three axioms matter at the infrastructure layer, not just the application layer. The balance must be derived from ledger entries, not cached independently. Every state transition must be durable before the offset commits. And every event from an external service must be verified before it can change the ledger.

The batch systems that event-driven architectures replaced were accidentally fail-closed. When a tape failed in a mainframe batch run, nothing processed until someone fixed it. The entire run stopped. That was a throughput problem, but it was never a correctness problem.

Event-driven systems gave us throughput and quietly traded away the one guarantee that mattered: when something breaks, nothing moves until it's fixed.