Twitter flies off the rails
According to noted internet pundit Arrington twitter has made the call to rewrite and move off ruby on rails. Not really a shock, rails is fundamentally not going to scale up past a certain point.
The uncharitable part of me (and the part that is confident nobody reads this blog) thinks that the vast majority of rails developers have never been around the kind of scale we’re talking about when we say rails doesn’t scale. Especially when I see something like this:
I love that items 2, 4 and 5 are REALLY REALLY hard.
More likely though those developers haven’t faced an app that is particularly hard to scale. I think the issue is less “does my framework scale” and more “is my application going to be hard to scale”.
Twitter is a great example of the kind of app that is very hard to scale, regardless of the platform. Rails is probably not the culprit here, but I doubt it is helping at all. In fact, at this point I would suspect twitter will do what everyone ends up doing when this happens: php frontend talking to a web service driven by java (or c). Because it turns out when the problems get real hard you don’t want a scripting language to handle them.
Rails isn’t bad, but it is isn’t “easy” to scale, except in cases where the application is easy to scale (content). Rails is designed by small, custom app developers for small developers. It provides one major benefit: it makes it faster to develop a certain class of applications. Thats it. So if time is money (you are a small dev shop) rails will make you money. Otherwise rails give you very little. I don’t think it would be a bad choice for most apps, but I don’t think its the panacea many rails advocates would claim.
May 2nd, 2008 at 2:54 am
Actually, scaling an app the type of Twitter is really quite simple. Messaging is simple overall. I don’t even like Rails, but frankly if someone fail to scale an app like Twitter with Rails, the problem is in front of the keyboard…
This is a classic multi-tier type app, where scaling Rails should be as simple as adding more boxes - there’s simply no need for more than an absolutely minimal state on the frontends at all, certainly nothing you could trivially keep in memcached sessions. If they do something stupid, like trying to shovel lots of databases updates in via Rails, then they will of course run into problems (and one of my dislikes about Rails is that it makes shooting you in the foot far more easy than it need to be).
For the backend, messaging of the type Twitter does isn’t hard, it’s a simple matter of setting up a mesh of relatively simple servers. I commented on the Techcrunch thread that frankly there are off the shelf alternatives that will handle this nicely, notably a number of Jabber servers, which has the advantage that they’re designed from the ground to be distributed, so all that’s needed is to farm the userbase out over a large enough number of domains - the users never need to know. If they suffer from not-invented here syndrome, writing an efficient messaging server in Ruby isn’t hard - I built one in less than 700 lines that would be trivial to mesh and that I tested up to around 2-3 million messages a day.
Persistence adds complexity, but nothing that can’t quite simply be handed by a partitioned database, or even an e-mail like system - I’ve done messaging queues handling huge volumes layered over e-mail before, and it sounds ugly, but it’s trivial to scale and make reliable because the protocols are all well understood and you can cherrypick from so many components.
Back to the Ruby messaging server: At the 2-3 million messages/day level it took less than 10% of a single 2GHz Xeon core. Scaling it to at least 15 million on a single core on the conservative side brings us to between 120 and 240 million messages per server (assuming 8 or 16 core boxes, which seems to be most cost effective now). Assuming peaks etc. we could be generous and “only” try to handle 100million a day per 16 core box. That’s less than 25 cents/month per million or so delivered messages if you go for managed hosting of rented servers, including outbound bandwidth.
I’ve run into my share of nasty scaling problems with Ruby, but with something that’s actually tricky to handle. This isn’t it.
And yes, when the problems get real hard, I do want a “scripting laguage” like Ruby to handle them - it means the easy things get out of the way trivially easily, and I can focus on the few hard bits, even if that means dipping down to C to write a tiny extension to speed something up, or sot in a tiny server in whatever other language I decide to use.
Otherwise I agree with you that Rails isn’t by any means the panacea that Rails advocates tend to claim, but not because of Twitter.
May 2nd, 2008 at 9:11 am
I think you are right, I worked on an app that delivered emails that I think was pretty similar. We sent out “triggered” emails to about 8 million users every day. We had an event server written in C that watched the db and output a feed of change events. The notification machine ran a php script that built each email and called an email service to send the messages. We ran it on I think 4 machines total (mirrored for continuity in to colos). It took about five hours for us to send out the messages, but that was mostly because the database queries for each email were quite complex.