Question about Akiba-Online's server(s)

[shinta]

Hey, I was just wondering, does AO run off a single server or does it use load balancing across several servers? What OS, http server, hardware, special tricks, etc. do you use? I'm pretty amazed at the number of concurrent users this site is able to serve, and I'm curious how you do it, Chompy (or whoever runs the site now).

 

[Rollyco]

does AO run off a single server

Yes. Q8300 + 8GB RAM + Windows Server 2003 R2 32-bit + IIS 6.0.

What special tricks do you use?

Lots and lots and lots of monitoring. Thanks to perfmon.msc, mysqlsla, and mysqlreport, I've been able fix some major software problems. Now we're butting up against hardware limitations. When you see those FastCGI errors @ around 3,400 online users, you're seeing the symptoms of an overloaded CPU. Chompy will have to get a more powerful server in the future, to accomodate any further growth. And hopefully some variant of *nix, because Win2K3 is no fun.

 

[shinta]

Hmm... ya, I've noticed those FastCGI errors. Maybe you could consider load balancing? I've never worked with Windows Servers (other than playing around), but I've done load balancing on *nix, and there are several popular software options and lots of documentation online on how to do it. The major drawback is, of course, the cost of additional servers. And I guess there's also the setup and extra maintenance work, but that depends on which software you use.

 

[Sakunyuusha]

I get the FastCGI errors every 2 out of 3 times I click on any link now on the site. Just throwin' it out there for statistical referencing.

 

[guy]

I personally suspect the issue is IIS (admitted I'm an Apache fanboy when it comes to webserving; I've dealt with IIS before), which is probably fighting for CPU resources against MySQL.

Theoretically speaking, though the forum may see as many as 5,000 users online at any given time, only a small percent are executing any SQL INSERT/UPDATE commands (eg: posting replies); most people are probably just clicking through threads, checking unread threads, so those MySQL queries ought to be cached in memory.

The theoretical problem is if the server is executing the PHP script every time someone clicks on a thread, even though no posts have been made. Eg: the PHP ought to only be run when the MySQL query result changes; if the MySQL result is the same as the most recent query (and thus MySQL is serving cached data), the server ought to similarly serve a static HTML page of the most recent execution of the PHP script.

Whether IIS does this or not, I'm not sure (though I'm inclined to say no, because MS built IIS with ASP/X in mind, not PHP, so there may be a lot of missing optimizations). Something like lighttp to cache and serve static/unmodified content would be great (and would require minimal hardware upgrades), but how hard it would be to implement with this forum software and whether or not you can get IIS to work with it is beyond me.

 

[Rollyco]

That's right, IIS sucks.

Ideally we would go 100% lighttpd, but it's unstable under Windows. Same with nginx. We'll try Apache soon.