News feed0/59 done
Part 3 · System design · Design problems

News feed

Users follow other users and see a feed of recent posts from the people they follow, newest first.

Ask first

  • Chronological or ranked?
  • Are there accounts with millions of followers?
  • How fresh must the feed be?
  • How many reads per write?
Numbers
feed readssay 10M daily users x 10 reads, about 1,200 per second
posts1M a day, about 12 per second, 200 followers each on average
fan-out on write12 x 200 = 2,400 list pushes per second is fine; one post by an account with 5M followers is not
API
POST /posts { text, media }   { id }
GET  /feed?cursor=...         { items, next }
Data
posts(id, author_id, created_at, body), sharded by author
follows(follower_id, followee_id), indexed both ways
feed:{user}  Redis list of post ids, capped at about 800
Deep dive: Push or pull?

Push (fan-out on write) copies the post id into each follower's list, so a read is one fast list read. Pull (fan-out on read) gathers posts from everyone you follow at read time: cheap writes, slow reads. Hybrid: push for normal authors, pull for huge accounts, merge at read time.

What goes in the cache?

Post ids only. Bodies come from a post cache by id, so edits and deletes show everywhere at once.

Inactive users?

Don't fan out to them. Rebuild their feed when they come back.

Paging?

A cursor on (created_at, id), so new posts don't shift pages.

See the core idea run

Core logic (JavaScript, tested)
// Hybrid fan-out. Normal authors: push the post into each follower's list on write (fast reads).
// Celebrities: skip the push (millions of writes); merge their posts in at read time.
function publish(feeds, followers, author, post, celebrities, cap = 800) {
  if (celebrities.has(author)) return 0;
  const fans = followers.get(author) ?? [];
  for (const f of fans) {
    const list = feeds.get(f) ?? [];
    list.unshift(post); // Redis: LPUSH feed:{f} id, then LTRIM feed:{f} 0 cap-1
    if (list.length > cap) list.length = cap;
    feeds.set(f, list);
  }
  return fans.length; // writes this post cost
}
function readFeed(feeds, following, user, postsBy, celebrities, k = 10) {
  const pulled = [...(following.get(user) ?? [])].filter(a => celebrities.has(a)).map(a => postsBy.get(a) ?? []);
  return mergeNewest([feeds.get(user) ?? [], ...pulled], k);
}
// k-way merge of lists already sorted newest first (a heap if there are many lists)
function mergeNewest(lists, k) {
  const idx = lists.map(() => 0), out = [];
  while (out.length < k) {
    let best = -1;
    for (let i = 0; i < lists.length; i++)
      if (idx[i] < lists[i].length && (best < 0 || lists[i][idx[i]].at > lists[best][idx[best]].at)) best = i;
    if (best < 0) break;
    out.push(lists[best][idx[best]++]);
  }
  return out;
}
Failures
Fan-out queue backs upFeeds are late, not wrong. Posts still appear on the author's page.
Feed cache lostRebuild from posts and follows, lazily on the next read.
Huge account postsNever fanned out; merged in at read time.
Trade-offs
  • Push: fast reads, many writes. Pull: few writes, slow reads.
  • Ranking adds a scoring service. Leave it out unless asked.
Say out loud: Hybrid fan-out: normal authors push post ids into their followers' Redis lists, and huge accounts are merged in at read time. The cache holds ids only, bodies are loaded by id, and pages use a cursor.