What is CDC?
Change data capture records inserts, updates, and deletes from a source system so downstream systems can stay in sync.
- Reads transaction log or events
- Supports near-real-time sync
- Needs ordering and deduplication
Quick recall
374 cards
What is late arriving data?
Late arriving data shows up after the expected processing window and can break simple incremental assumptions.
- Common in event systems
- Needs watermark or backfill strategy
- Affects correctness of incremental models
What is schema evolution?
Schema evolution is how a system handles added, removed, or changed fields over time without breaking downstream consumers.
- Plan backward compatibility
- Track versions
- Validate critical contracts
Why is idempotency important in data pipelines?
Idempotent jobs can rerun safely without duplicating or corrupting data.
- Critical for retries
- Use stable keys or merge logic
- Aim for deterministic outputs
CAP theorem basics
In a network partition, a distributed system must choose between strong consistency and availability.
- Partition tolerance is unavoidable in distributed systems
- Choice matters only during partition
- Many systems pick different tradeoffs by workflow
Database indexing in system design
Indexes speed up common queries, but they cost extra storage and slower writes.
- Optimize real query patterns
- Too many indexes hurt writes
- Composite indexes matter
Database sharding basics
Sharding splits data across multiple databases so one machine does not hold or serve everything.
- Scale write and storage capacity
- Need shard key
- Cross-shard queries get harder
How do message queues help system design?
Queues decouple producers from consumers so work can be retried, smoothed, and processed asynchronously.
- Absorbs traffic spikes
- Improves reliability
- Requires idempotent consumers
Primary-replica database setup
A primary handles writes while replicas serve copied data, often improving read scale and availability.
- Writes usually go to primary
- Replicas can lag
- Read/write split adds complexity
Rate limiting basics
Rate limiting controls how many requests a client can make in a window to protect system stability and fairness.
- Protects against abuse
- Token bucket is a common algorithm
- Can be per user, IP, or API key
Scalability vs latency
Scalability is how well a system handles growth, while latency is how quickly one request gets a response.
- Scale is capacity
- Latency is response time
- Optimizing one can hurt the other
What is a CDN?
A CDN caches content closer to users so static or cacheable responses arrive with lower latency.
- Edge locations near users
- Reduces origin load
- Great for static assets and media