Part 3 · System design · Design problems
News feed
Users follow other users and see a feed of recent posts from the people they follow, newest first.
Ask first
- Chronological or ranked?
- Are there accounts with millions of followers?
- How fresh must the feed be?
- How many reads per write?
Numbers
| feed reads | say 10M daily users x 10 reads, about 1,200 per second |
| posts | 1M a day, about 12 per second, 200 followers each on average |
| fan-out on write | 12 x 200 = 2,400 list pushes per second is fine; one post by an account with 5M followers is not |
API
POST /posts { text, media } { id }
GET /feed?cursor=... { items, next }Data
posts(id, author_id, created_at, body), sharded by author
follows(follower_id, followee_id), indexed both ways
feed:{user} Redis list of post ids, capped at about 800Deep dive: Push or pull?
Push (fan-out on write) copies the post id into each follower's list, so a read is one fast list read. Pull (fan-out on read) gathers posts from everyone you follow at read time: cheap writes, slow reads. Hybrid: push for normal authors, pull for huge accounts, merge at read time.
What goes in the cache?
Post ids only. Bodies come from a post cache by id, so edits and deletes show everywhere at once.
Inactive users?
Don't fan out to them. Rebuild their feed when they come back.
Paging?
A cursor on (created_at, id), so new posts don't shift pages.
See the core idea run
Core logic (JavaScript, tested)
// Hybrid fan-out. Normal authors: push the post into each follower's list on write (fast reads).
// Celebrities: skip the push (millions of writes); merge their posts in at read time.
function publish(feeds, followers, author, post, celebrities, cap = 800) {
if (celebrities.has(author)) return 0;
const fans = followers.get(author) ?? [];
for (const f of fans) {
const list = feeds.get(f) ?? [];
list.unshift(post); // Redis: LPUSH feed:{f} id, then LTRIM feed:{f} 0 cap-1
if (list.length > cap) list.length = cap;
feeds.set(f, list);
}
return fans.length; // writes this post cost
}
function readFeed(feeds, following, user, postsBy, celebrities, k = 10) {
const pulled = [...(following.get(user) ?? [])].filter(a => celebrities.has(a)).map(a => postsBy.get(a) ?? []);
return mergeNewest([feeds.get(user) ?? [], ...pulled], k);
}
// k-way merge of lists already sorted newest first (a heap if there are many lists)
function mergeNewest(lists, k) {
const idx = lists.map(() => 0), out = [];
while (out.length < k) {
let best = -1;
for (let i = 0; i < lists.length; i++)
if (idx[i] < lists[i].length && (best < 0 || lists[i][idx[i]].at > lists[best][idx[best]].at)) best = i;
if (best < 0) break;
out.push(lists[best][idx[best]++]);
}
return out;
}Failures
| Fan-out queue backs up | Feeds are late, not wrong. Posts still appear on the author's page. |
| Feed cache lost | Rebuild from posts and follows, lazily on the next read. |
| Huge account posts | Never fanned out; merged in at read time. |
Trade-offs
- Push: fast reads, many writes. Pull: few writes, slow reads.
- Ranking adds a scoring service. Leave it out unless asked.
Say out loud: Hybrid fan-out: normal authors push post ids into their followers' Redis lists, and huge accounts are merged in at read time. The cache holds ids only, bodies are loaded by id, and pages use a cursor.