Twitter or X
Users post short tweets, follow others, and read a home timeline of tweets from people they follow, newest first. Some accounts have 100M followers.
In plain words
You follow 300 people. When you open the app, the newest tweets from all of them must appear in under a second. Reading 300 lists and sorting everything every time would be far too slow at 600,000 timeline reads a second.
So the work is done ahead of time: when your friend tweets, the tweet's id is dropped into your timeline list right away, like a newspaper boy putting a paper in every subscriber's box. Except for accounts with 100 million followers: their tweets aren't delivered to 100 million boxes. When you open the app, the system grabs their latest few and mixes them in.
Examples
- Twitter keeps about 800 tweet ids per user's timeline in Redis.
- Tweet ids are snowflakes: the time comes first, so sorting by id is sorting by time.
- A celebrity tweet costs one write instead of 100 million.
- Like counts on a viral tweet go to several counters that are added up, so no single row melts.
1. Requirements
Functional
- Post short tweets with media.
- Follow and unfollow people.
- Home timeline: tweets from people you follow, newest first.
- Like and reply.
Non functional
- Timeline loads in under 200 ms at 600k reads per second.
- A tweet may take a few seconds to reach timelines.
- Accounts with 100M followers must not melt the system.
- Ids must sort by time without a central counter.
Questions to ask
- Chronological or ranked timeline?
- How many daily users, and reads per write?
- Search, trends and DMs in scope?
- How fresh must the timeline be?
2. Estimates
| tweets | 500M a day, about 6k per second, peaks 3x |
| timeline reads | about 100 reads per tweet: 600k per second |
| storage | 500M x 300 bytes is about 150 GB a day of text, media separate |
3. API
POST /tweets { text, mediaIds? } { id }
GET /timeline/home?cursor=... [tweets], nextCursor
GET /users/{id}/tweets?cursor=... [tweets], nextCursor
POST /follow/{userId} 204
4. Data model
tweets(id, author_id, text, created_at) id is a snowflake: sortable by time
follows(follower_id, followee_id) both directions indexed
timeline:{user} Redis list of tweet ids, newest first, about 800
media: object storage + CDN
5. High level design
Follow one request at a time, top to bottom.
Post a tweet
- ClientPOST /tweets
- Tweet servicemakes a snowflake id, saves the tweet
- Fan out queueskipped for huge accounts
- Fan out workerspush the id into each follower's timeline list
- Redis timelinesabout 800 ids per user
Read the timeline
- ClientGET /timeline/home
- Timeline servicereads the user's id list
- Celebrity mergeadds recent tweets from huge accounts you follow
- Tweet cacheids to full tweets in one multi get
- CDNimages and video
6. Deep dives
Push or pull for timelines?
Push for most authors: a new tweet id is written into each follower's cached timeline. Pull for huge accounts: their tweets are merged in when you read. A merge of a few sorted lists is cheap.
How are ids made at 6k per second?
Snowflake style: milliseconds, machine number, counter. No coordination per tweet, and sorting by id sorts by time, which makes cursors easy.
What do you cache?
Timelines as lists of ids, and tweets by id. A timeline read is one list read plus a multi get of tweets.
How do you handle a viral tweet?
Like counts go to sharded counters and are summed in the background; the tweet itself is served from cache and CDN.
See the core idea run
Core logic (JavaScript, tested)
// Home timeline: merge the newest tweets from everyone you follow, without reading everything.
// Each author's list is already newest first, so take the top of each and repeat (a k way merge).
function timeline(byAuthor, following, limit) {
const heads = following.map(a => ({ list: byAuthor.get(a) ?? [], i: 0 }));
const out = [];
while (out.length < limit) {
let pick = null;
for (const h of heads) if (h.i < h.list.length && (!pick || h.list[h.i].at > pick.list[pick.i].at)) pick = h;
if (!pick) break;
out.push(pick.list[pick.i++].id);
}
return out;
}
// Snowflake style ids: time first, then machine, then a counter. Sorting by id sorts by time,
// and no two machines make the same id without talking to each other.
function snowflake(ms, machine, seq) {
return (BigInt(ms) << 22n) | (BigInt(machine) << 12n) | BigInt(seq);
}7. Failures
| Fan out backlog after a big event | Timelines may lag a few seconds. Readers still merge in very recent tweets on read. |
| Timeline cache lost | Rebuild from the follow list on the next read: pull, then cache. |
| Hot tweet hammering one cache key | Replicate hot keys across cache nodes, or cache in the app servers for a few seconds. |
8. Trade-offs
- Push makes reads fast but writes huge for big accounts; pull is the reverse; the mix gets both.
- Chronological is simple and predictable; ranked is more engaging but needs a scoring service.