Image may be NSFW.
Clik here to view.
At the recent Silicon Valley Ruby Conference, Friends for Sale developer Alex Le outlined the approach they’ve taken to scaling up to deliver an impressive 300 million page views per month for their Rails-based Facebook application.
Friends for Sale is currently the #6 most popular application on Facebook. The application describes itself as follows:
“Buy and sell your friends as pets! You can make your pets poke, send gifts, or just show off for you. Make money as a shrewd pets investor or as a hot commodity!”
So I guess I there’s something here I don’t quite get — but in any case it has become perhaps the highest-traffic Rails app, delivering at least 50% more page views per month than yellowpages.com.
The app was created by the ironically named Serious Business Inc. The initial version was built in 10 days by Alex Le and Siqi Chen, using the RFacebook gem, and was launched on 11/1/07. Alex had previously worked on the rebuilding of yellowpages.com as a Rails application.
Referring to the oft-cited scaling aphorism, Alex asserted “Don’t scale until you need to… unless you’re on Facebook!” They had scalability problems on their first day because of the speed at which the audience ramped up. By the end of the first month they were delivering 1 million page views per day; by the end of the third month, they hit 10 million per day. They currently have 7 million users and 650,000 unique visitors per day.
Scaling to this traffic level required both adding lots of servers and tuning the database. They started out at Slicehost, and switched to SoftLayer when they needed more servers. Their decision to use SoftLayer was driven, in part, by the speed with which they could get them set up — 2 hours. Alex encouraged everyone to start with a hosting provider that can scale with you, to reduce the pain when traffic grows.
After two months, they hired a database administrator to handle the database tuning. Alex recommended using the Rails Query Analyzer plugin, and to index only the fields that you need to. He noted that while Active Record makes it easy to get the application going, it can get in the way when you need to optimize your database performance. They use Dr. Nic’s Magic Multi Connections plugin to handle the database connections, with some monkey-patching to fix issues they found; they plan to release their plugin code soon.
Their current server configuration includes:
- One front-end server running nginx with fair for load balancing.
- Eleven application servers, each with 8G RAM and four 2.4 GHz processors, running 16 Mongrel instances on each one (that’s 176 Mongrels!), with 4G on each server for memcached.
- One master database server and three slaves, each with eight 2.3 GHz processors and 32G RAM, using 10K RPMSCSI drives in a Raid 1+0 configuration, running MySQL 5.
Their next step is to seek more performance from their existing hardware. Toward that end, they’re planning to move from fair to qrp (queueing reverse proxy) for load balancing, and they’re exploring alternative implementations of Mongrel that have faster threading models.
They also cache extensively. Serialized models are stored in memcached, using write-through caching. They moved logic to view helpers to simplify the use of fragment caching.
Serious Business has been recruiting for Rails developers, and in these ads they claim they are building the world’s largest Rails cluster. Maybe selling friends isn’t such a bad idea.