Hi all! I've been working on some performance-related stuff today and figured I should post an update.

Firstly, I have to say that the new servers have been (from a corruption standpoint), rock-solid. That means we haven't been hard-crashing every 30 hours as a result of slow erosion of the database, nor have we been forced to recover a database off the slave server (YAY!)

However, our new servers are not without their gremlins. You may have noticed the site has been somewhat slow off and on, and even unresponsive at times. In fact, I hear Titania and her team are putting together a new Subeta Weather Bureau to try and forecast some of these lag storms. In the meantime though, I thought I'd explain what's going on.

Normally more powerful servers are a good thing. After all, faster servers should mean a faster site, right? As it turns out, not always.

One of the main measures of how fast servers are is how many processors ("cores") it has. Your average desktop or laptop computer has 1 or 2 cores. Subeta's database servers have 8 cores. More cores allows the server to do more things at once. Great, right?

Unfortunately, there can also be downsides to having more cores. To understand why, imagine this: let's say you have 2 people, and they're both getting their picture taken. So they play rock paper scissors to decide who gets to go first, and then they take turns posing for the camera. Everything's smooth.

Now let's say there are 8 people. Now they all start fighting amongst each other, trying to decide who gets to go first. In the meantime, no pictures are being taken.

The same thing happens with complicated programs (like a database) running on really powerful computers. Lots of cores means the database creates lots of different threads to take advantage of those cores, and the threads compete for resources. This doesn't always work so well: sometimes the threads end up fighting over resources like disk access and access to protected memory, and spend so much time fighting that no queries are being answered.

It's this effect that's been affecting Subeta off and on since we moved to the new servers. For some reason, the combination of resources and server capacity is past some critical level where it's actually slowing things down rather than providing a speed-up.

Unfortunately, MySQL is such a complicated program, there is no easy way to determine what combination of settings is causing these concurrency issues. It's a matter of tuning the server through trial-and-error to see what mitigates (and hopefully eventually eliminates) this effect.

If this explanation confuses you, a less accurate analogy is that of a car. At slow speeds, the car wheels grip the road and move the car forward. But if the driver puts his foot on the gas, the car's wheels can spin in place, spinning very quickly but not going anywhere. That's kind of what Subeta does when it locks up... mysql CPU usage climbs to 800% (100% x 8 cores) but actual use of the CPU to do work drops to a mere 4%, down from 30-40% normally.

Anyway, I hope that was understandable enough. The long and short of it is, we're working hard to track down what's causing the lag and get rid of it. And apart from the occasional lag storms, the new servers have proved to be 10 times as stable as we were before.

-Alex