Distributed Systems · Chapter 08

Charge the card exactly once

The subscription-payment service — where getting it wrong means double-charging a customer or losing a sale. This chapter traces a card payment through six intermediaries you don't control, then builds a backend on Saga (no locks across banks), event sourcing (an immutable timeline you can replay), and idempotency (click purchase ten times, get charged once).

🧭

How to read this

First the domain: what actually happens between clicking "purchase" and the money moving — six players, two stages (authenticate, then transfer funds). Then the backend you build, layered: Saga because you can't lock a bank's database; event sourcing because payments need a replayable timeline; Kafka partitioning because event order matters; and idempotency because networks retry. Anything marked [+] is context I added; everything else is rewritten from the five teaching lessons.

The online card payment flow

Before designing anything, understand what happens end-to-end. A card payment has two stages — user authentication, then funds transfer — passing through a chain of players, none of which (past your own platform) you control.

Player	Role
Consumer	The user. Clicks purchase → request routes to your platform's payment backend.
Platform (Merchant)	You. Your payment module asks the payment processor to process the transaction (order id, amount, …).
Payment Processor	Stripe, PayPal, Square, Razorpay. A single point of contact to many banks/financial services; screens for fraud/duplicates, and picks the best acquiring-bank gateway in real time to maximize success rate.
Payment Gateway	Provided by the acquiring bank (where you feed card details). The acquiring bank collects funds on the merchant's behalf; the merchant holds an account (usually current) there.
Card Network	VISA, MasterCard, RuPay, Amex — connects issuing and acquiring banks (the logo on your card).
Issuing Bank	Holds the user's account. Authenticates the user (the OTP page), then debits if funds suffice.

Figure 1 — The card-payment chain: authenticate (OTP at the issuing bank), then transfer funds; the response flows back to the merchant. Original diagram after Figures 8.0–8.1.

🏦

The money doesn't move immediately — settlement & capture

After the issuing bank debits and notifies the user, the response flows back and the user sees "order successful." But the acquiring bank isn't paid instantly — it submits the day's transactions to the issuing bank in a batch settlement. Until settlement, the card is in the captured state: the debit is authorized and ready, not yet settled.

Why Saga, not two-phase commit

Which database for payments?

Payments need strong consistency and ACID transactions — so a relational DB (Postgres/MySQL) is the obvious choice. (Subscription info reuses the existing User table, assuming one subscription per user at a time.) Recall the earlier lesson: for strong consistency there's no difference between SQL and NoSQL — both must lock nodes, and locking a NoSQL setup kills the availability and performance that made it attractive.

🔒

You can't lock a bank's database

A subscription purchase spans multiple systems — your platform is just one. You cannot acquire a lock across services (your platform can't lock a bank server or the processor's infrastructure), so 2PC is out. The answer is the Saga pattern: a sequence of local transactions, each with a compensating transaction triggered on failure. If anything fails at any stage, the whole thing rolls back from the point of failure. Rollbacks are vital in financial transactions.

You only control the merchant side; everything else is third-party. You interact with the payment processor purely through its API, against which your entire payment backend is written. Every transaction spawned from the merchant carries a unique transaction reference number.

Event sourcing and the payment architecture

Event sourcing tracks every event in the system in an immutable, sequential log — rather than storing only the final application state. Why bother?

Debug by time-travel If a customer is over-charged, the log lets you pinpoint the rogue event(s), fix the bug, and see exactly what happened when. Final-state-only storage can't tell you that.

Replay to fix data Fixing the bug isn't enough — you must repair the bad data for every affected customer. The log lets you replay the whole purchase flow from where it broke.

Analytics The event stream reveals customer purchase behavior — why a plan isn't selling, purchases by hour — feeding ML for better UX and retention.

The architecture

User clicks purchase → request hits the Payments API, which creates an order-purchase event and pushes it to a Kafka queue — the source of events for the whole service.
The Payment Module pulls events, applies business logic, and persists order details to the relational Order table (the application state) with OrderStatus: IN PROGRESS. The User table has a one-to-many with Order (a user places many orders — subscriptions plus one-off content: podcasts, articles, merch).
The Event Handler pulls events in parallel and appends them to the event store, building the timeline.
The Payment Module forwards the request to the payment processor — and every component pushes its event details back to the queue so the event store keeps a complete sequential log.

Figure 2 — The event-sourced payment service: the Kafka queue is the event backbone; the Payment Module owns app state, the Event Handler owns the immutable log. Original diagram after Figures 8.4–8.9.

🔗

The whole flow is a Saga

From the moment the API receives the request, the payment flow is a Saga: passing the event to the queue, writing to the Order table, writing to the event store, and forwarding to the processor are each separate local transactions with compensating transactions. Any one fails → the entire transaction rolls back.

The event store

An immutable log of events. You can back it with a relational, document, or wide-column store (Cassandra), or a purpose-built event database (EventStoreDB). Whatever you pick must support sequential writes and querying events by id; if you append multiple events at once, it needs transactions too. The author prefers a relational DB (if not a native event store) and, when choosing products, favors maturity — time in market, community, docs — over a shiny new option, to avoid production firefighting.

Event ordering with Kafka partitions

A customer's actions must be logged in the order they occurred. Kafka guarantees order within a partition — events in a partition are persisted and consumed in occurrence order. With one payments topic, which partition an event lands in is decided by hashing the event key:

partition = hashCode(key) % N   // N = number of partitions

🔑

Partition by customer id

To keep one customer's events in order, partition by customer id so they all land in the same partition. Without a key, events distribute round-robin across partitions — and a consumer might read a cancel-order event before the purchase event that it cancels, creating chaos. (Event ordering in distributed systems is tricky and needs heavy testing; corner cases remain even with keyed partitioning.)

Idempotency and CQRS

The duplicate-events problem

Message queues offer delivery guarantees, and neither is free:

Strategy	Problem for payments
At least once	Guarantees delivery — but may deliver more than once. If a consumer-delivery timeout occurs, Kafka can't tell if it arrived, so it retries → duplicate events → the customer charged multiple times.
Only once	May drop some messages — also unacceptable when strong consistency is the order of the day.

🎯

The fix: a unique id as primary key

Downstream systems (payment module, event handler) own duplicate-handling. For every subscription-purchase request, generate a unique transaction id / reference number — it can combine customer id + transaction id + timestamp. Use it as the primary key in your tables. Now a duplicate can't be persisted twice — the DB throws a duplicate-key exception, weeding out duplicates. The customer can click purchase n times; all events are idempotent and they're charged once. (The payment processor always receives a unique transaction request.)

CQRS with the event-store data

Recall CQRS (Command Query Responsibility Segregation) from the sharding chapter: besides writing to the primary database, an async process writes to a dedicated query store built for specific reads. The payment service does the same shape — the Order table holds application state (the command side), while the event store serves insights into user activity (the query side). External services read the event store for batch validations, and its data can stream into ELK for analytics.

🧾

Why batch operations matter

Payment/banking systems are eventually consistent by their distributed nature. Companies run daily batch operations over event data to validate the day's transactions — catching a customer erroneously over-charged, an inconsistent DB entry, etc., and fixing them. Batch reconciliation is a vital part of any payment system, not an afterthought.

Recap

Payment chain Consumer → Platform → Processor → Gateway (acquiring bank) → Card Network → Issuing Bank. Authenticate (OTP), then transfer funds; batch settlement, captured state.

Saga, not 2PC Payments need ACID → relational DB. But you can't lock a bank's database, so Saga with compensating transactions handles the multi-system flow.

Event sourcing Immutable sequential log enables debugging by time-travel, replay to fix data, and analytics. Order table = state; event store = timeline.

Kafka ordering Order guaranteed within a partition; partition by customer id (hashCode(key) % N) so a cancel never precedes its purchase.

Idempotency At-least-once → duplicates; unique txn id as primary key throws duplicate-key on retries — click n times, charged once.

CQRS + batch Order table (command) vs. event store (query); feeds batch validations and ELK analytics. Payments are eventually consistent, so reconcile daily.

💡

The single idea underneath it all

A payment is the hardest kind of distributed transaction because the stakes are money and you own almost none of the systems involved. You can't lock what you don't control, so you give up on atomic locking entirely — Saga lets each step fail and compensate. You can't trust the network to deliver exactly once, so you make every step safe to repeat — a unique id turns retries into no-ops. And because money must be auditable and eventually reconciled, you don't just store the final balance — you store the whole story as an immutable log you can replay. Design for failure, repetition, and scrutiny, because in payments all three are guaranteed.