Archprimer
ArchitectureAdvanced 8 min read

Payment Gateway

Design a payment gateway with idempotent APIs, a double-entry ledger, exactly-once processing, PSP integration, and reconciliation under strong consistency.

idempotencyledgerexactly-onceconsistencyreconciliation

By Ratish Jain · Updated

Diagram with 6 components: Merchant, Payment API, Ledger DB, Orchestrator, PSP, Reconciler. Connections: Merchant → Payment API (1 · charge); Payment API → Ledger DB (2 · reserve); Payment API → Orchestrator (3 · process); Orchestrator → PSP (4 · capture); Orchestrator → Ledger DB (5 · post); Reconciler → Ledger DB (verify).
ClientGatewayDatabaseServiceExternal
Payment flow: idempotent API writes ledger, calls PSP, records outcome.

Problem

A payment gateway moves money between a customer and a merchant by orchestrating external Payment Service Providers (PSPs) and card networks, while keeping an authoritative record of every cent. Money must never be created or destroyed by a bug, a retry, or a crash, so the system demands strong consistency, idempotency, and exactly-once effects on the ledger.

The design centers on a double-entry ledger as the source of truth, idempotent APIs so retries are safe, an orchestrator that drives each payment through a state machine, and a reconciliation process that continuously proves internal records match the PSP.

Functional Requirements

  • Authorize, capture, and refund payments through one or more PSPs.
  • Guarantee idempotent charges so retries never double-charge.
  • Maintain an auditable double-entry ledger of all money movements.
  • Handle async PSP webhooks and status callbacks.
  • Reconcile internal records against PSP settlement reports.

Non-Functional Requirements

  • Strong consistency and durability for all financial records.
  • Exactly-once effect on the ledger despite retries and failures.
  • High availability with no lost or orphaned transactions.
  • Auditability — every state change is immutable and traceable.
  • PCI-compliant handling; no raw card data stored by the gateway.

Capacity Estimation

Transactions/day
50M (~600/sec avg, 5K/sec peak)
Ledger entries
≥2 per txn (debit+credit) ≈ 100M/day
Avg record size
~1 KB per ledger entry
Storage growth
~100 GB/day, retained for years
PSP latency
500ms–2s per authorization
Reconciliation
daily batch over full day's ledger

APIs

POST /payments
charge(idempotencyKey, amount, method) -> {paymentId, status}
POST /payments/{id}/capture
capture(paymentId, amount) -> {status}
POST /payments/{id}/refund
refund(paymentId, amount, idempotencyKey) -> {status}
POST /webhooks/psp
onEvent(pspEvent) -> {ack}

Components

  • Payment API: enforces idempotency keys and validates requests.
  • Orchestrator: state machine (created → authorized → captured → settled/refunded) driving each payment.
  • Ledger service: append-only double-entry records; the financial source of truth.
  • PSP adapters: integrate with card networks/banks; handle auth, capture, refund.
  • Outbox / event bus: reliably publishes state changes using the outbox pattern.
  • Reconciler: batch job comparing ledger against PSP settlement files.
  • Idempotency store: maps keys to prior responses.

Request Flow

  1. 1Merchant calls POST /payments with an idempotency key; API returns the cached result if the key was seen.
  2. 2The ledger records a pending/reserved entry in a transaction.
  3. 3The orchestrator advances the state machine and calls the PSP to authorize/capture.
  4. 4The PSP responds synchronously or later via webhook.
  5. 5On success, the ledger posts balanced debit/credit entries and the payment is marked captured.
  6. 6The reconciler later verifies these entries against the PSP settlement report.

Flow Diagram

Diagram with 5 components: Merchant, Payment API, Ledger, PSP / Acquirer, Post & Notify. Connections: Merchant → Payment API (1 · charge); Payment API → Ledger (2 · reserve); Payment API → PSP / Acquirer (3 · authorize); PSP / Acquirer → Post & Notify (4 · capture); Ledger → Post & Notify (5 · post).
ClientGatewayDatabaseExternalService
A charge reserves funds, authorizes with the PSP, then posts and notifies.

Data / Storage Design

The ledger is append-only and uses double-entry bookkeeping: every movement writes balanced debit and credit entries so the books always sum to zero — the strongest invariant for correctness. It lives in a strongly consistent relational store (Postgres or a distributed SQL DB) with ACID transactions.

Idempotency is enforced by storing each idempotencyKey with its first response; duplicates return the stored result instead of re-executing. To publish events reliably without dual-write bugs, state changes and their outbound events are written in the same transaction via the outbox pattern, then relayed asynchronously. Raw card data is tokenized by the PSP and never persisted here.

Data Model

Diagram with 5 components: Payment API, Idempotency, Ledger DB, Outbox, PSP. Connections: Payment API → Idempotency (1 · check); Payment API → Ledger DB (2 · entries); Ledger DB → Outbox (3 · event); Outbox → PSP (4 · settle).
GatewayCacheDatabaseQueueExternal
Idempotency keys dedupe retries; ledger + outbox commit atomically.

Scaling Strategy

Financial correctness outranks raw throughput, so the ledger stays strongly consistent — shard by merchantId or accountId to scale writes while keeping each account's entries in one partition for transactional integrity. Reads (statements, dashboards) go to replicas.

The orchestrator is stateless and scales horizontally; payment state persists in the DB, so any instance can resume a payment. PSP calls are the slow, failure-prone hop, so they run through a saga-style compensating flow: if capture fails after authorization, a compensating void/refund unwinds it.

Bottlenecks & Failure Scenarios

  • Double-charge from retries without idempotency keys.
  • Dual-write inconsistency between DB and event bus — solved by the outbox pattern.
  • PSP timeout leaving a payment in an unknown state; needs status polling/webhooks.
  • Ledger hot partition for a high-volume merchant.
  • Reconciliation mismatches from missed or duplicated webhook events.

Design Tradeoffs

Pros
  • Double-entry ledger gives a provably balanced, auditable record.
  • Idempotency keys make retries safe and enable exactly-once effects.
  • Outbox pattern removes dual-write inconsistency.
Cons
  • Strong consistency limits horizontal write scalability.
  • Saga compensations add complexity versus a single distributed transaction.
  • Dependence on external PSP latency and reliability.

Common Interview Questions

How do you prevent double-charging on retries?

Clients pass an idempotency key per logical payment. The gateway stores the key with the first outcome in the same transaction; any retry with that key returns the stored response instead of calling the PSP again, giving exactly-once effect.

Why a double-entry ledger?

Every money movement writes equal debits and credits, so total balances always reconcile to zero. This makes bugs and inconsistencies detectable, provides a full audit trail, and is the accounting-standard way to represent transfers.

How do you keep the DB and events consistent?

Use the outbox pattern: write the state change and an outbox event row in one ACID transaction, then a relay publishes the event. This avoids the dual-write problem where the DB commits but the message publish fails (or vice versa).

60-Second Revision
  • →Double-entry, append-only ledger is the strongly consistent source of truth.
  • →Idempotency keys make retries safe and give exactly-once effects.
  • →Orchestrator drives a payment state machine; PSP calls are the risky hop.
  • →Outbox pattern publishes events without dual-write inconsistency.
  • →Batch reconciliation proves internal records match PSP settlements.

Related

Everything on Archprimer is connected — follow the graph.