Design a distributed cache

Clarifying questions

• Which reads can be stale and which reads must reflect writes immediately?
• Are keys small objects, large blobs, or precomputed query results?
• Do partners require stronger consistency than public clients?

Functional requirements

• Cache product records and common product-list queries.
• Invalidate or refresh entries after product updates.
• Expose cache behavior to operators through dashboards.

Nonfunctional requirements

• Keep p95 read latency below 50 ms for cached objects.
• Avoid a cache stampede after popular keys expire.
• Survive one cache node failure without full cold-start behavior.

Scale assumptions

• 500 million reads per day.
• 10 million active products.
• Top 10,000 keys produce a large fraction of traffic.

API sketch

• GET /v1/products/{id} reads through cache.
• POST /internal/cache/invalidate { key, version } for write-path invalidation.

Data model

• Cache key: product:{productId}:v{version}.
• Invalidation event: entity_type, entity_id, old_version, new_version, emitted_at.

Architecture components

• Application services use read-through caching for product records.
• Write service emits versioned invalidation events after database commit.
• Hot-key protection uses request coalescing and stale-while-revalidate.

Bottlenecks

• Popular list queries can cache too many combinations of filters.
• Invalidation fanout can lag behind writes during import jobs.

Failure modes

• Cache node failure: client library rehashes keys and falls back to database with rate limits.
• Invalidation loss: TTL bounds maximum stale duration.
• Cold start: warm top keys from access logs before promotion.

Observability

• Hit rate by route and object type, not only global hit rate.
• Origin database QPS, coalesced request count, stale response count.

Security / privacy

• Never cache per-user authorization decisions under shared keys.
• Encrypt sensitive values or keep them out of shared cache entirely.

Cost considerations

• Memory cost follows object size times replication factor times hot set size.
• Over-caching low-hit keys wastes memory and reduces eviction quality.

Tradeoffs

• Versioned keys make invalidation safer but increase orphaned entries until TTL expiry.
• Write-through cache improves freshness but couples writes to cache availability.

Rubric