1. This algo serves 150 billion Tweets to people’s devices every day. Now you can have all answers to designing a recommendation engine available for free! It’s a terrific time to learn the algorithm line by line which has evolved from ~2 decades of world-class engineering based on communication from politics to shitposts.
2. The algorithm uses a neural network with a whopping 48 million parameters that is continuously trained on tweet interactions to optimize engagement.
3. Twitter’s GraphJet analyzes hundreds of millions of tweets to extract the best 1500 tweets for each user request and runs approximately 5 billion times per day with an average completion time of under 1.5 seconds.
4. The “For You” timeline consists of 50% In-Network Tweets and 50% Out-of-Network Tweets on average, with around 15% of Home Timeline Tweets served through Twitter’s graph traversal heuristics.
5. Twitter’s SimClusters identifies over 145k communities, updated every three weeks, ranging in size from a few thousand users to hundreds of millions.
6. The entire recommendation pipeline consists of three main stages and uses various heuristics and filters to ensure a balanced and diverse feed, including visibility filtering, author diversity, and content balance.
7. Twitter fired most of its employees. It was easy for anyone to take advantage by revealing some small parts of the algorithm and become popular by attacking Elon Musk directly.
It could damage Elon’s brand if happens frequently.
The best option was to make it open source and write two lines about an era of transparency!
There could be many reasons, but having access to this code is a great resource to learn and build.
