Payment Gateway
Design a payment gateway with idempotent APIs, a double-entry ledger, exactly-once processing, PSP integration, and reconciliation under strong consistency.
By Ratish Jain · Updated
Problem
A payment gateway moves money between a customer and a merchant by orchestrating external Payment Service Providers (PSPs) and card networks, while keeping an authoritative record of every cent. Money must never be created or destroyed by a bug, a retry, or a crash, so the system demands strong consistency, idempotency, and exactly-once effects on the ledger.
The design centers on a double-entry ledger as the source of truth, idempotent APIs so retries are safe, an orchestrator that drives each payment through a state machine, and a reconciliation process that continuously proves internal records match the PSP.
Functional Requirements
- Authorize, capture, and refund payments through one or more PSPs.
- Guarantee idempotent charges so retries never double-charge.
- Maintain an auditable double-entry ledger of all money movements.
- Handle async PSP webhooks and status callbacks.
- Reconcile internal records against PSP settlement reports.
Non-Functional Requirements
- Strong consistency and durability for all financial records.
- Exactly-once effect on the ledger despite retries and failures.
- High availability with no lost or orphaned transactions.
- Auditability — every state change is immutable and traceable.
- PCI-compliant handling; no raw card data stored by the gateway.
Capacity Estimation
- Transactions/day
- 50M (~600/sec avg, 5K/sec peak)
- Ledger entries
- ≥2 per txn (debit+credit) ≈ 100M/day
- Avg record size
- ~1 KB per ledger entry
- Storage growth
- ~100 GB/day, retained for years
- PSP latency
- 500ms–2s per authorization
- Reconciliation
- daily batch over full day's ledger
APIs
- POST /payments
- charge(idempotencyKey, amount, method) -> {paymentId, status}
- POST /payments/{id}/capture
- capture(paymentId, amount) -> {status}
- POST /payments/{id}/refund
- refund(paymentId, amount, idempotencyKey) -> {status}
- POST /webhooks/psp
- onEvent(pspEvent) -> {ack}
Components
- Payment API: enforces idempotency keys and validates requests.
- Orchestrator: state machine (created → authorized → captured → settled/refunded) driving each payment.
- Ledger service: append-only double-entry records; the financial source of truth.
- PSP adapters: integrate with card networks/banks; handle auth, capture, refund.
- Outbox / event bus: reliably publishes state changes using the outbox pattern.
- Reconciler: batch job comparing ledger against PSP settlement files.
- Idempotency store: maps keys to prior responses.
Request Flow
- 1Merchant calls POST /payments with an idempotency key; API returns the cached result if the key was seen.
- 2The ledger records a pending/reserved entry in a transaction.
- 3The orchestrator advances the state machine and calls the PSP to authorize/capture.
- 4The PSP responds synchronously or later via webhook.
- 5On success, the ledger posts balanced debit/credit entries and the payment is marked captured.
- 6The reconciler later verifies these entries against the PSP settlement report.
Flow Diagram
Data / Storage Design
The ledger is append-only and uses double-entry bookkeeping: every movement writes balanced debit and credit entries so the books always sum to zero — the strongest invariant for correctness. It lives in a strongly consistent relational store (Postgres or a distributed SQL DB) with ACID transactions.
Idempotency is enforced by storing each idempotencyKey with its first response; duplicates return the stored result instead of re-executing. To publish events reliably without dual-write bugs, state changes and their outbound events are written in the same transaction via the outbox pattern, then relayed asynchronously. Raw card data is tokenized by the PSP and never persisted here.
Data Model
Scaling Strategy
Financial correctness outranks raw throughput, so the ledger stays strongly consistent — shard by merchantId or accountId to scale writes while keeping each account's entries in one partition for transactional integrity. Reads (statements, dashboards) go to replicas.
The orchestrator is stateless and scales horizontally; payment state persists in the DB, so any instance can resume a payment. PSP calls are the slow, failure-prone hop, so they run through a saga-style compensating flow: if capture fails after authorization, a compensating void/refund unwinds it.
Bottlenecks & Failure Scenarios
- Double-charge from retries without idempotency keys.
- Dual-write inconsistency between DB and event bus — solved by the outbox pattern.
- PSP timeout leaving a payment in an unknown state; needs status polling/webhooks.
- Ledger hot partition for a high-volume merchant.
- Reconciliation mismatches from missed or duplicated webhook events.
Design Tradeoffs
- Double-entry ledger gives a provably balanced, auditable record.
- Idempotency keys make retries safe and enable exactly-once effects.
- Outbox pattern removes dual-write inconsistency.
- Strong consistency limits horizontal write scalability.
- Saga compensations add complexity versus a single distributed transaction.
- Dependence on external PSP latency and reliability.
Common Interview Questions
Clients pass an idempotency key per logical payment. The gateway stores the key with the first outcome in the same transaction; any retry with that key returns the stored response instead of calling the PSP again, giving exactly-once effect.
Every money movement writes equal debits and credits, so total balances always reconcile to zero. This makes bugs and inconsistencies detectable, provides a full audit trail, and is the accounting-standard way to represent transfers.
Use the outbox pattern: write the state change and an outbox event row in one ACID transaction, then a relay publishes the event. This avoids the dual-write problem where the DB commits but the message publish fails (or vice versa).
- →Double-entry, append-only ledger is the strongly consistent source of truth.
- →Idempotency keys make retries safe and give exactly-once effects.
- →Orchestrator drives a payment state machine; PSP calls are the risky hop.
- →Outbox pattern publishes events without dual-write inconsistency.
- →Batch reconciliation proves internal records match PSP settlements.
Related
Everything on Archprimer is connected — follow the graph.