Archprimer
ConceptBeginner 7 min read

Caching

Storing hot data in a faster, closer layer to cut latency and load, governed by hit ratio, eviction policies like LRU, and the hard problem of invalidation.

performancelatencymemoryevictionhit-ratio

By Ratish Jain · Updated

Diagram with 3 components: App Server, Cache, Database. Connections: App Server → Cache (1 · lookup); App Server → Database (2 · miss: read); Database → Cache (3 · fill).
ServiceCacheDatabase
App checks cache first; a miss falls through to the database, then fills cache.

Definition

Caching is keeping a copy of frequently accessed data in a faster or closer storage layer so future reads avoid the slower source. The goal is to serve hot data quickly and shield the backing store from repeated work.

Why It Exists

Backing stores (disks, databases, remote services) are orders of magnitude slower than memory or a nearby cache. Most workloads are read-heavy and follow a skewed distribution — a small fraction of data gets most requests. Caching exploits that skew to slash latency and multiply effective capacity without scaling the origin.

How It Works

On a read, the app checks the cache: a hit returns immediately; a miss falls through to the source, then populates the cache for next time. The hit ratio (hits ÷ total) measures effectiveness. Because caches are small, an eviction policy decides what to drop when full — LRU (least recently used) is most common, alongside LFU and TTL-based expiry. The hard part is invalidation: when the source changes, stale cache entries must be updated or evicted, or you serve wrong data. Caches exist at many layers: client/browser, CDN edge, application (Redis/Memcached), and inside the database itself.

What It's Used For

  • Speeding up expensive DB queries and computed results
  • Session and token storage in memory
  • Rate-limiter counters and leaderboards
  • CDN edge caching of static assets
  • Deduplicating calls to slow external APIs

Tradeoffs

Pros
  • Large latency reduction for hot reads
  • Offloads and protects the backing store
  • Cheap way to add read capacity
  • Layered — apply at client, edge, app, and DB
Cons
  • Invalidation is genuinely hard (stale data bugs)
  • Adds a consistency window between cache and source
  • Cold caches and stampedes can overload the origin
  • Extra memory cost and another moving part

When to Use / Avoid

Use when
Reads dominate, data is reused, and slight staleness is acceptable for a big latency win.
Avoid when
Data must be strongly consistent and changes constantly, or access is uniformly random with no hot set.
Real-world
Nearly every large system caches — Facebook's memcached tier and Twitter's Redis clusters serve billions of reads.

Interview Questions

Why is cache invalidation considered hard?

The source of truth and the cache are separate stores that can drift. Knowing precisely which entries a write affects, and evicting them everywhere before a stale read, is tricky — TTLs bound staleness but don't eliminate it.

What is a cache stampede and how do you prevent it?

When a popular key expires, many requests miss simultaneously and hammer the origin at once. Mitigations include request coalescing (single-flight), staggered/jittered TTLs, and serving stale-while-revalidate.

60-Second Revision
  • →Store hot data in a faster/closer layer to cut latency.
  • →Hit ratio measures effectiveness; misses fall through to source.
  • →Eviction (LRU/LFU/TTL) decides what to drop when full.
  • →Invalidation is the hard problem — beware stale data.
  • →Layers: client, CDN, application, database.

Related

Everything on Archprimer is connected — follow the graph.