Deconstructing the Master Algorithm: How X Filters 500 Million Tweets in 1.5 Seconds

Every time you open X (formerly Twitter) and drag down to refresh your “For You” timeline, a massive distributed pipeline wakes up. In approximately 1,500 milliseconds, it evaluates an active catalog of roughly 500 million daily posts and delivers a customized batch of 50 to 100 items to your client.

For years, the inner workings of this engine were shrouded in mythology: Does posting links suppress you? Are verified badges truly algorithmic cheat codes? How does an obscure post from a creator you’ve never heard of suddenly hit your timeline with 30,000 likes?

To answer this conclusively, we spent the morning diving into the official open-source repository (twitter/the-algorithm). We traced the Finagle and Product Mixer request graphs, stepped through Scala pipeline configs, inspected Rust-based ML inference bindings in navi, and uncovered the raw mathematical formulas in Tweepcred and RerankerUtil.

Here is the architectural anatomy of how the world’s most influential public town square actually constructs your reality.

---

1. The 4-Stage Funnel: From 500M to 50

The core architecture is built around Twitter’s Product Mixer framework. Rather than running a monolithic machine learning model over all 500 million posts—which would be computationally impossible at sub-second latencies—the system processes candidates through a four-stage funnel:

[500,000,000 Global Posts]
           │
           ▼ Stage 1: Candidate Sourcing (~50% In-Network / ~50% Out-of-Network)
    [~1,500 Candidates]
           │
           ▼ Stage 2: Fast Pre-Filtering (Blocks, Mutes, Seen IDs, Drop Rules)
    [~1,000 Candidates]
           │
           ▼ Stage 3: Heavy Ranker (Navi / MaskNet Deep Neural Inference)
    [Scored Multi-Head Candidates]
           │
           ▼ Stage 4: Feed Mixing & Polish (Author Diversity, Fatigue, FOSNR, Ads)
   [Final 50–100 Tweets Rendered]

The orchestration entrypoint lives in ForYouMixerPipelineConfig.scala. Its job is not to score text directly, but to coordinate independent upstream candidate pipelines in parallel, merge the results using Stitch and Finagle Futures, apply filters, and marshal the data into URT (Unified Rich Timeline) responses.

---

2. The 50/50 Split: Sourcing Friends and Strangers

Your feed targets a strict equilibrium: approximately 50% In-Network (accounts you follow) and 50% Out-of-Network (accounts you don’t follow). These two halves are sourced through completely divergent architectures.

In-Network Sourcing: Earlybird & UTEG

When you follow someone, their posts bypass global candidate discovery. They are indexed in real-time by Earlybird, Twitter’s distributed Lucene-based search cluster. Simultaneously, an in-memory graph engine named UTEG (User Tweet Entity Graph), built on the C++/Java GraphJet framework, maintains bipartite interaction tables. If an account you follow tweets, Earlybird identifies it within milliseconds, scoring it using a light ranking model focused on recency and historical affinity.

Out-of-Network Sourcing: CR-Mixer

Finding relevant tweets from strangers across a catalog of hundreds of millions of posts is coordinated by cr-mixer (Content Recommendation Mixer). Instead of semantic keyword searches, CR-Mixer routes queries to three primary sub-engines:

1. Follow Recommendations Service (FRS): Identifies creators followed by users with similar graph signatures, extracting their top-performing recent posts.
2. RealGraph: An offline-computed logistic regression and GBDT model that scores the direct pairwise interaction likelihood between any two users on the platform.
3. SimClusters: The algorithmic backbone of content classification on X.

---

3. SimClusters: 145,000 Communities Without Manual Tagging

X does not rely on manual category tags, editorial curation, or simple hashtags to understand what content is about. It uses SimClusters (src/scala/com/twitter/simclusters_v2), a community-detection framework based on bipartite follow graphs.

1. The 20 Million Producer Matrix: The system takes the top 20 million most-followed accounts on X (the Producers) and builds a massive cosine similarity graph based on their overlapping followers (the Consumers).
2. Metropolis-Hastings Clustering: Running Markov Chain Monte Carlo community detection on this producer-producer graph discovers approximately 145,000 distinct latent communities (e.g., Rust systems programmers, Turkish political debate, Premier League tactics, generative AI researchers).
3. The User InterestedIn Vector: As you follow accounts, your personal profile is projected into a sparse vector across these 145,000 dimensions, representing your long-term topical affinities.
4. Real-Time Tweet Vector Accretion: When a tweet is published, its community vector is initialized to empty. Every time a user favorites or retweets that post, the user’s InterestedIn vector is added to the tweet’s vector via real-time Heron streaming jobs.

If 100 people whose profiles heavily overlap with the autonomous agent engineering cluster favorite a brand-new tweet, that tweet instantly crystallizes as an authoritative item for that community—allowing the candidate generator to push it to other interested users globally, even if the author has zero mutual followers with them.

---

4. Tweepcred: The Algorithmic Credit Score & The Follow-Spam Trap

Every user on X has an underlying reputation score between 0 and 100 calculated by Tweepcred (com.twitter.graph.batch.job.tweepcred). Tweepcred is an Apache Scalding batch job that runs an iterative, weighted PageRank over the interaction graph (mentions, retweets, replies, and follows).

If an account with a Tweepcred score of 85 likes your post, the weight transferred to your tweet in candidate ranking is orders of magnitude higher than if an account with a score of 20 likes it.

The Mathematical Penalty for Follow-Churn

Many growth hackers attempt to build audience size by aggressively following thousands of accounts hoping for follow-backs. In Reputation.scala, there is a hardcoded mathematical trap designed to crush this tactic:

def adjustReputationsPostCalculation(mass: Double, numFollowers: Int, numFollowings: Int) = {
  if (numFollowings > threshAbsNumFriendsReps) {
    val friendsToFollowersRatio = (1.0 + numFollowings) / (1.0 + numFollowers)
    val divFactor =
      scala.math.exp(
        constantDivisionFactorGt_threshFriendsToFollowersRatioReps *
          (friendsToFollowersRatio - threshFriendsToFollowersRatioUMass) *
          scala.math.log(scala.math.log(numFollowings))
      )
    mass / ((divFactor min maxDivFactorReps) max 1.0)
  } else {
    mass
  }
}

Examining the production constants:

• threshAbsNumFriendsReps = 2500
• constantDivisionFactorGt_threshFriendsToFollowersRatioReps = 3.0
• threshFriendsToFollowersRatioUMass = 0.6
• maxDivFactorReps = 50

If your account follows more than 2,500 people, the algorithm inspects your friendsToFollowersRatio. If you follow significantly more people than follow you back, your raw PageRank score is divided by an exponential factor that scales up to 50x. An account trapped in this division factor will have its reach throttled across all candidate generation pipelines.

---

5. The Heavy Ranker: Predicting 15 Discrete Human Reactions

Once candidate sourcing gathers ~1,500 candidates, they are dispatched to Navi—a high-performance machine learning server written in Rust that exposes gRPC endpoints compatible with TensorFlow and ONNX runtimes.

The Heavy Ranker is not predicting a vague “quality” number. As defined in PredictedScoreFeature.scala, it evaluates a multi-task neural network (MaskNet / DeepBird) that outputs discrete probabilities for specific human actions:

$$\hat{y} = \left[ P(\text{fav}), P(\text{retweet}), P(\text{reply}), P(\text{vqv}), P(\text{dwell}), P(\text{good_click}), P(\text{negative_feedback}) \dots \right]$$

In RerankerUtil.scala, these probabilities are aggregated into a unified scalar score:

$$\text{Score} = \sum_{i} w_i \cdot P(\text{action}_i)$$

The Engagement Weights Hierarchy

Looking at the feature weights configured in HomeGlobalParams.scala:

• Likes (fav) & Retweets (retweet): Provide the positive baseline momentum.
• Dwell Time (dwell) & Video Quality Views (vqv): Watching $\ge 10$ seconds of a video or pausing your scroll to read text adds significant positive bias.
• The Golden Multiplier: reply_engaged_by_author: A standard reply gives a modest boost, but if the author engages back with the commenter, the algorithm fires the reply_engaged_by_author head. This is one of the highest positive multipliers in the entire engine. It confirms that the post is facilitating an active, authentic conversation rather than broadcasting shouting into the void.
• Negative Feedback (negative_feedback_v2) & Reports (report): Carry massive negative weights that immediately zero out positive gains.

---

6. The Blue Checkmark Bypass: The Social Context Filter

One of the most consequential code paths uncovered in the open-source release is inside SocialContextFilter.scala:

val validTweetIds = candidates
  .filter { candidate =>
    candidate.features.getOrElse(InNetworkFeature, true) ||
    candidate.features.getOrElse(SuggestTypeFeature, None).exists(AllowedSources.contains) ||
    candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None).isDefined ||
    (enableIsVerifiedAuthorFilter && isVerifiedAuthor(candidate.features)) ||
    hasLikedBySocialContext(candidate.features) ||
    hasFollowedBySocialContext(candidate.features) ||
    (enableTopicSocialContextFilter && hasTopicSocialContext(candidate.features))
  }.map(_.candidate.id).toSet

Here is what this means in practice:

• If a post is Out-of-Network and the author is unverified, it cannot reach the user’s timeline unless it possesses explicit Social Context (e.g., someone you follow liked it, or it carries an explicit topic attachment).
• If enableIsVerifiedAuthorFilter is active and the author is verified (AuthorIsBlueVerifiedFeature, AuthorIsGoldVerifiedFeature, or AuthorIsGrayVerifiedFeature), all social context requirements are waived.

The verified badge is not just a cosmetic UI treatment or a flat score multiplier; it is an algorithmic border pass that allows out-of-network candidates to enter feeds without needing a mutual follow or like to bridge them into the recipient’s social graph.

---

7. Feed Diversity, Dampening, and the 140-Day Fatigue Memory

The raw Heavy Ranker score is never the final word. Before posts reach your screen, they must survive several rigorous dampening and fatigue stages.

A. Author Diversity Exponential Decay

If a creator posts ten tweets in thirty minutes, does the algorithm flood their followers’ timelines? No. In AuthorBasedListwiseRescoringProvider.scala, multiple tweets from the same author are grouped and subjected to an exponential position decay:

$$\text{Multiplier}(\text{index}) = (1 - \text{floor}) \cdot \text{decayFactor}^{\text{index}} + \text{floor}$$

The first tweet retains 100% of its scored weight. The second tweet from the same author within that ranking batch is penalized; the third and fourth are driven to the floor. Burst-tweeting cannibalizes your own reach.

B. Feedback Fatigue: The 140-Day Grudge

In FeedbackFatigueScorer.scala, we find the consequence of a user tapping “See fewer posts like this” (FeedbackType.SeeFewer):

val DurationForDiscounting = 140.days
private val ScoreMultiplierLowerBound = 0.2
private val ScoreMultiplierUpperBound = 1.0
private val ScoreMultiplierIncrementsCount = 4

When a user gives negative feedback to an author or topic:

1. The author’s score is immediately slashed to 0.2 (an 80% penalty).
2. The penalty has a half-life of 140 days (~4.5 months).
3. The multiplier only recovers gradually across four 35-day increment brackets.

One click of “Not interested” effectively silences an author on that viewer’s device for over a quarter of a year.

C. Out-of-Network Scaling

In OONTweetScalingScorer.scala, out-of-network candidates have a static scale factor applied:

private val ScaleFactor = 0.75

Tweets from people you don’t follow automatically suffer a 25% baseline handicap compared to in-network posts from friends you actually chose to follow.

---

8. Freedom of Speech Not Reach: The Visibility Engine

The final layer is visibilitylib, which implements the policy commonly summarized as “Freedom of Speech, Not Reach” (FreedomOfSpeechNotReach.scala).

Rather than performing binary takedowns or permanent suspensions, the visibility engine operates as a graduated continuum:

• Coarse-Grained Downranking: The tweet remains visible on the author’s profile and permalink, but its internal visibility score is crushed so it will never appear in any “For You” candidate retrieval query.
• Interstitial Wrappers: The tweet content is obfuscated behind a warning barrier (“This post may violate X’s rules”), requiring explicit click-through.
• Limited Actions: The platform programmatically strips interaction capabilities (retweet, quote, or like buttons are disabled via LimitedActionsPolicy).
• Brand Safety Ad Avoidance: Ads are prevented from rendering adjacent to toxic or sensitive conversational context, protecting monetization without deleting the speech.

---

9. Architectural Takeaways: How to Navigate the Algorithm

Tracing X’s source code separates empirical reality from social media superstition:

1. Conversations Over Broadcasting: Because reply_engaged_by_author is among the highest-weighted positive heads in PredictedScoreFeature, creators who genuinely reply to their audience experience compounding algorithmic velocity.
2. Respect the Follower Ratio: Once an account crosses 2,500 followings, allowing following-to-follower ratios to degrade triggers an exponential PageRank divisor up to 50x in Reputation.scala.
3. Pacing Trumps Frequency: Bursting multiple tweets in short succession triggers AuthorBasedListwiseRescoringProvider’s exponential decay, muting your own downstream posts.
4. Negative Feedback is Fatal: With an 80% discount spanning 140 days in FeedbackFatigueScorer, baiting cheap outrage that tempts users to click “See fewer” causes catastrophic long-term distribution damage.
5. Dwell Time and Native Video are Structural Arbitrages: vqv ($\ge 10$ seconds) and dwell provide steady additive probability mass that simple short text updates cannot generate on their own.

Underneath the chaos of the public timeline sits an astonishingly disciplined, highly mathematical distributed engine. It does not judge ideas; it calculates graph intersections, evaluates tensors in Rust, balances diversity decay, and projects human desire across 145,000 dimensions of community space.