Tag: Sharding

What Is Sharding in System Design?

As applications grow, data grows.
And when data grows too much, a single database can’t handle it efficiently.

This is where sharding comes in.

The Problem: One Database, Too Much Load

Imagine you have:

  • Millions of users
  • Billions of records
  • Thousands of requests per second

If all data lives in one database:

  • Queries become slow
  • CPU and memory max out
  • Scaling becomes expensive
  • One failure can bring everything down

👉 Vertical scaling (adding more RAM/CPU) has limits.

The Solution: Sharding

Sharding means:

Splitting a large database into smaller, independent pieces called shards and storing them across multiple servers.

Each shard holds only a portion of the data.

Real-World Analogy: Supermarket Billing Counters

Imagine a supermarket with:

  • One billing counter
  • Hundreds of customers

❌ Chaos.

Now introduce multiple counters:

  • Counter 1 → Customers A–D
  • Counter 2 → Customers E–J
  • Counter 3 → Customers K–Z

✔ Faster checkout
✔ Less load per counter
✔ Easy to add new counters

👉 That’s sharding.

How Sharding Works (Conceptually)

Instead of:

Single Database
└── All Users Data

We have:

Shard 1 → User IDs 1–1,000,000
Shard 2 → User IDs 1,000,001–2,000,000
Shard 3 → User IDs 2,000,001–3,000,000

Each shard is:

  • Independent
  • Handles only its own data
  • Can scale separately

Shard Key (Most Important Concept)

A shard key decides:

Which data goes to which shard

Common shard keys:

  • User ID
  • Customer ID
  • Region
  • Date

Example:

userId % 3
userIdShard
1Shard 1
2Shard 2
3Shard 3
4Shard 1

Types of Sharding

1️⃣ Horizontal Sharding (Most Common)

Split rows across shards.

Users 1–1000 → Shard A
Users 1001–2000 → Shard B

✔ Used in most real systems

2️⃣ Vertical Sharding

Split columns.

User Profile → Shard A
User Orders → Shard B

✔ Useful when some data is accessed more frequently

3️⃣ Directory-Based Sharding

A lookup table tells:

“Which shard has this data?”

✔ Flexible
❌ Extra lookup cost

Real-World Use Cases

CompanyHow They Use Sharding
FacebookUser-based sharding
InstagramMedia & user shards
AmazonCustomer & order shards
NetflixRegional sharding
Banking AppsAccount number shards

Challenges in Sharding

Sharding is powerful, but not free.

❌ Cross-Shard Queries

  • Joining data across shards is expensive

❌ Re-sharding

  • Changing shard strategy later is painful

❌ Uneven Load

  • Bad shard key = hotspot shard

Sharding vs Replication (Very Important)

ShardingReplication
Splits dataCopies data
Improves write scalabilityImproves read availability
Each shard has different dataAll replicas have same data

👉 Large systems often use both together.

When Should You Use Sharding?

Use sharding when:

  • Data size is huge
  • Write traffic is high
  • Single DB can’t scale vertically
  • Latency is increasing

❌ Don’t shard too early — it adds complexity.

Simple Rule to Remember

Replication is for availability.
Sharding is for scalability.

Final Thoughts

Sharding is not a database feature —
it’s a system design decision.

Understanding sharding means:

  • You think about scale
  • You think about failure
  • You think like a backend engineer

Architectural Concepts that Every Software Developer should know.

  1. Load Balancer

Distributes incoming traffic across multiple servers to ensure no single server gets overwhelmed.

  1. Caching

    Stores frequently accessed data in memory to reduce latency.

    1. Content Delivery Network (CDN)

    Stores static assets across globally distributed edge locations or servers so users download the content from the nearest location

    1. Message Queue

    Decouples components by letting producers enqueue messages that consumers process asynchronously.

    1. Publish-Subscribe

    Enables multiple consumers to receive messages from a topic

    1. API Gateway

    Acts as a single entry point for client requests, handling routing, authentication, rate limiting and protocol translation

    1. Circuit Breaker

    Prevents a system from repeatedly calling a failing service by temporarily stopping requests until the service recovers.

    1. Service Discovery

    Automatically tracks available service instances, enabling components to discover and communicate with each other dynamically.

    1. Sharding

    Sharding splits large datasets across multiple nodes using a shard key to improve scalability and performance.

    1. Rate Limiting

    Rate limiting controls how many requests a client can make within a time window to prevent service overload.

    1. Consistent Hashing

    Consistent hashing is a data distribution technique that minimizes data movement when nodes join or leave the system.

    1. Auto Scaling

    Auto scaling automatically adds or removes compute resources based on demand or predefined metrics.