Study how a large-scale social media feed recommendation system works, from candidate retrieval to final ranking.
Use the architecture as a reference when designing a content ranking or personalization pipeline for your own platform.
Understand how engagement signals like likes, replies, and reposts are combined into a single relevance score.
This is a reference architecture, not a turnkey system. Running it requires large-scale ML infrastructure and proprietary training data.
This repository contains the core recommendation system that powers the For You feed on X (the social network formerly known as Twitter). When you open the app and see posts from accounts you do not follow alongside ones you do, that curation is produced by the pipeline documented and partially open-sourced here. The system gathers posts from two places. The first source, called Thunder, pulls content from accounts you actively follow. The second, called Phoenix Retrieval, searches a much larger global pool of posts using machine learning to find content you might find interesting even if you do not follow those accounts. Both pools are then combined and passed through several processing stages. Before ranking, the system enriches each candidate post with extra data: author details, media information, and your own engagement history, meaning what you have liked, replied to, reposted, or clicked on in the past. A filtering step then removes items that should never appear: duplicates, older posts already shown, content from accounts you have blocked or muted, and posts containing keywords you have silenced. The ranking itself is handled by a component called Phoenix, which runs a transformer model adapted from the Grok-1 architecture. This model predicts the probability that you will take each type of action on a given post, including liking, replying, reposting, or clicking. These predicted probabilities are multiplied by fixed weights and summed to produce a single score for each post. An additional step deliberately reduces the scores of posts from the same author when too many appear together, keeping the feed from being dominated by one person. The entire pipeline is coordinated by an orchestration layer called Home Mixer. The code is written in Rust and released publicly as an open-source reference showing how the feed actually works. The README also notes that the team has eliminated hand-crafted rules and heuristics, letting the Grok-based transformer handle relevance decisions entirely.
← xai-org on gitmyhub — every repo by this author, as a profile.
Verify against the repo before relying on details.