Design Twitter
Variants:
Twitter System Design Requirements
Functional Requirements
Post Tweets
Users should be able to post tweets with a maximum of 280 characters.
Follow Users
Users should be able to follow other users to see their tweets.
View Timeline
Users should be able to view a timeline of tweets from the users they follow.
Non-Functional Requirements
High Availability
The system must be highly available, with minimal downtime.
Low Latency
The timeline should load in under 200ms.
Scalability
The system must be able to handle millions of users and tweets per day.
Durability
Once a tweet is posted, it should not be lost.
CAP Theorem Trade-offs
Trade-off Explanation:
For a social media platform like Twitter, we prioritize Availability and Partition Tolerance (AP). It's more important for users to be able to post and view tweets than to have strict consistency. Eventual consistency is acceptable, as a slight delay in a new tweet appearing on a follower's timeline is not a critical issue.
Scale Estimates
API Design
The API for our Twitter-like service will focus on the core functionalities: posting tweets, following users, and retrieving the user's timeline. Your interviewer will expect you to define clear and concise endpoints for these operations.
The POST /api/v1/tweets endpoint will allow users to create a new tweet. The POST /api/v1/follow endpoint will allow a user to follow another user. The GET /api/v1/timeline endpoint will be a read-heavy endpoint that retrieves the user's personalized timeline.
Post a new tweet
Follow another user
Get the user's timeline
Database Schema
The database schema for a Twitter-like service needs to efficiently store users, tweets, and the social graph (follows). Your interviewer will be interested in how you model these relationships to ensure fast reads and writes.
We'll use three main tables: users, tweets, and follows. The users table will store user profile information. The tweets table will store all the tweets, and the follows table will represent the social graph, with a composite primary key of follower_id and followee_id.
users
tweets
follows
High-Level Architecture
At a high level, our Twitter-like service will be composed of several microservices, each responsible for a specific part of the system. This approach allows for independent scaling and development of each component. Your interviewer will expect you to be able to break down the system into these logical components.
Core Services
-
Tweet Service: This service is responsible for handling all tweet-related operations. It will have endpoints for creating, retrieving, and deleting tweets. This service will be write-heavy, as users are constantly posting new tweets.
-
Timeline Service: This service is responsible for generating the user's timeline. This is a read-heavy service that needs to be highly optimized for performance. It will query the
followstable to get the list of users that the current user follows, and then retrieve the latest tweets from those users. -
User Service: This service is responsible for managing user accounts and the social graph. It will have endpoints for creating users, retrieving user profiles, and managing follows.
Deep Dive: Timeline Generation
Generating the user's timeline is one of the most critical and challenging parts of a Twitter-like system. Your interviewer will want to know how you would approach this problem, as it involves a trade-off between real-time updates and system performance.
Fan-out on Write (Push Model)
Why: In a fan-out on write approach, we pre-compute the user's timeline whenever a new tweet is posted. This makes timeline retrieval extremely fast, as we are simply reading from a pre-computed list.
How it works: When a user posts a tweet, we retrieve the list of their followers and insert the new tweet into the timeline of each follower. This is typically done using a message queue to handle the fan-out process asynchronously. The user's timeline can be stored in a key-value store like Redis, with the user_id as the key and a list of tweet_ids as the value.
Trade-offs:
- Pros: Fast timeline reads, which is the most common operation.
- Cons: Can be slow for users with a large number of followers (the "celebrity problem"). Wastes resources if many followers are inactive.
Fan-out on Read (Pull Model)
Why: In a fan-out on read approach, we generate the user's timeline on demand when they request it. This is simpler to implement than the fan-out on write approach and avoids the "celebrity problem."
How it works: When a user requests their timeline, we first retrieve the list of users they follow. Then, we query the tweets table to get the latest tweets from those users, sort them by created_at, and return the result.
Trade-offs:
- Pros: Simpler to implement, no "celebrity problem."
- Cons: Slower timeline reads, as we have to perform multiple queries and a sort on every request.
Hybrid Approach
A hybrid approach is often the best solution for a system like Twitter. We can use a fan-out on write approach for most users, and a fan-out on read approach for celebrities with a large number of followers. This gives us the best of both worlds: fast timeline reads for the majority of users, and a scalable solution for celebrities.
Deep Dive: Scaling the Database
Your interviewer will expect you to discuss how you would scale the database to handle the massive amount of data and traffic that a Twitter-like service would generate.
Database Sharding
Why: Sharding is a technique for horizontally partitioning a database. It allows us to distribute the data and query load across multiple servers, which is essential for a system at this scale.
How it works: We can shard our database by user_id. This means that all the data for a particular user (their profile, tweets, and follows) will be stored on the same shard. This makes it easy to retrieve all the data for a user in a single query.
Trade-offs:
- Pros: Highly scalable, allows for even distribution of data and load.
- Cons: Can be complex to implement and maintain. Requires a sharding key that will evenly distribute the data.
Complete Design
Now that we've covered all the major components individually, let's look at how everything fits together in our complete Twitter system design. This diagram shows the end-to-end flow from a user posting a tweet to it appearing in their followers' timelines.
The complete architecture demonstrates how the Tweet Service, Timeline Service, and User Service work together to provide a scalable and reliable Twitter-like experience. The use of a hybrid timeline generation model and database sharding ensures that the system can handle millions of users and tweets per day.