The Three Axioms of Fintech Engineering

Your database says the user has $10,247. The bank's API says $9,812. Which number do you show?

The bank's. Always. But that resolves the symptom, not the cause. A $435 gap appeared with no corresponding transaction on either side, and fourteen other accounts have the same kind of drift, ranging from $12 to $2,100.

This kind of bug, a number that's quietly wrong, traces back to one of three architectural mistakes.

No invented data

Every number in your system must trace back to an authoritative source. If you can't point to where a number came from, you've invented it.

The balance drift from the opening happened because the system computed balances from its own transaction log instead of treating the bank's API as the source of record. Most of the time, the two agreed. When a webhook arrived late, or a transaction posted on the bank side without triggering an event in the app, the numbers diverged.

Nobody noticed for three weeks. When the operations team finally caught it, they reconciled three weeks of balance history across fourteen accounts against the bank's records, line by line. It took four days.

A grocery store that stops checking invoices and starts pricing items by counting what's on the shelves has the same problem. If a delivery shows up overnight and nobody scans it in, the shelf count looks right. The prices won't be.

Fee amounts should be computed from the agreed rate times the base at the time of the transaction, not read from a pre-stored column that can diverge when rates change. The same holds for account balances, which should be derived from the sum of ledger entries rather than stored independently.

This is the central argument for treating a balance as a view, not a field.

Pick any number your system displays to a user and trace it back to its source. Every link in the chain should resolve to an authoritative external record or a computation from authoritative inputs. If any link comes from your own running total or a separately maintained cache, you've invented data.

No lost data

Every state change writes to a durable log before it's considered complete. If a process crashes between step 3 and step 4, the system picks up at step 4.

A payment processor sends a webhook: a $2,000 transfer just settled. Your service picks it up, updates the transaction status, and starts adjusting the user's balance. Between the status update and the balance adjustment, the pod restarts.

The transaction now shows "settled" but the balance hasn't moved. The webhook provider saw a 200 response and won't retry. You have a settled transaction with no corresponding balance entry, and nobody knows until the next reconciliation run catches it.

If you have one.

Construction sites keep daily logs for every task completed. Not because foremen have bad memories, but because shifts change, people call in sick, and six months later an insurance adjuster needs to reconstruct what happened on a specific Tuesday. If the work isn't in the log, the adjuster treats it as if it didn't happen.

Financial systems carry the same requirement at higher stakes. Banks in the US file roughly 4 million Suspicious Activity Reports per year. Each one requires reconstructing the exact sequence of events that triggered it.

SOX-governed systems must retain audit trails for at least 7 years. A gap in your log on a Tuesday in March becomes a question you can't answer when a regulator asks about it.

So you use persisted state machines with explicit transitions rather than in-memory workflows that vanish on restart. Reconciliation jobs run every 15 to 60 minutes to catch anything that fell through, whether a webhook that never arrived or a status transition stuck mid-flight.

The log is your proof that something happened, or didn't, when a dispute lands on your desk 90 days later.

No trust

Every boundary crossing is a trust boundary. Data from outside your system gets the same treatment as user input: validate first, then act.

A webhook arrives claiming a $50,000 transfer just settled. Your system updates the ledger. An hour later, your reconciliation job pulls the actual status from the provider's API: the transfer is still pending. You credited $50,000 to a user's available balance based on a claim you never confirmed.

Maybe the webhook was legitimate and the API was slow to update. Maybe it was a replay. You acted on unverified input.

A restaurant kitchen doesn't fire a dish because someone walks in and says "table 12 ordered the lobster." The cook checks the ticket, not the voice.

When a webhook claims a settlement, call the provider's GET endpoint for that transaction ID and confirm the status before making any state change. Proceed only when the status matches. Any mismatch gets logged for reconciliation.

The same discipline applies to API responses (validate the schema before storing values) and authentication tokens (verify the signature cryptographically rather than trusting a header).

Most of the time, trust works. The webhook is real and the response is well-formed. Trust violations surface at the edges, which is why they survive testing.

A staging environment leaking test events to your production webhook URL, or a provider deploying a response format change without notice. Low-probability events that become certainties at sufficient transaction volume.

They fail together

These axioms compound. A system that invents data usually has gaps in its logs too, because the same engineering shortcuts that produce one produce the other. Weak trust boundaries let bad external input through on the way in, which is how the invented data gets created in the first place.

Nothing crashes or throws an error. The numbers drift, quietly, for weeks, until a reconciliation report catches the gap or a regulator asks a question the system can't answer.