News Archive: Week 13, 2009

Having trouble seeing the dance floor or perhaps you just want to look cool? Whatever the reason, these new ocular implants from the <a href="http://www.subeta.net/shop.php?shopid=3">Holiday Shop</a> are perfect for you! ;)

Having trouble breathing from all the dancing? Perhaps you should pick up one of these Breather Implants from the <a href="http://www.subeta.net/shop.php?shopid=3">Holiday Shop</a>!

Who doesn't love a nice <a href="http://www.subeta.net/explore/gift.php">free</a> wrap? :o

The problem with constant dancing is one gets very very hot. The <a href="http://www.subeta.net/shop.php?shopid=3">Holiday Shop</a> has solved this problem with new fans! :D

Who knows how well they work, though. At least they look pretty. :D

Another bag seems to have found it's way to the <a href="http://www.subeta.net/explore/ticenter.php">Token Shop</a>! I wonder what is inside this one. ;)

Pretend to fly away with these new winged masks at the <a href="http://www.subeta.net/shop.php?shopid=3">Holiday Shop</a>! :o

Warning: These masks do not actually make you fly. :x

Another shipment of masks for the <a href="http://www.subeta.net/shop.php?shopid=3">Holiday Shop</a>!

It seems this Fashion Bag has been misplaced! Why would it be in the <a href="http://www.subeta.net/explore/ticenter.php">Token Shop</a>? :o

Are the drinks at the Libertine Lounge a bit too expensive for you? Perhaps these less fancy martinis from <a href="http://www.subeta.net/shop.php?shopid=22">SubetaBucks</a> would be a better choice for you. :o

Want to spice up your outfit with a new mask? Well, the <a href="http://www.subeta.net/shop.php?shopid=3">Holiday Shop</a> is the perfect place for all of you tired of using the same old mask. ;)

Thank god that's over!

So we're back, with no loss of time on the site (no r-word), as promised.

You may have already read the explanation of what happened on our downtime message, but if you missed it (or want to know more), I'll go ahead and explain it here.

For the past few months, one of our biggest threats to stability has been that our database file is gradually becoming corrupted (think of someone pulling files out of a filing cabinet and tossing them all over the floor). We're currently talking to MySQL, the people who make our database, and trying to determine if this is a problem in our usage, or (more likely) a bug in the database program itself, in which case we will help them track it down and fix it.

The temporary solution to this is a complex procedure where we dump the database completely and "re-import" it, creating a totally new database file free of any corruption. There is a way we can do this transparently, with only minimal downtime. The procedure takes about a day.

Unfortunately, we were too late - the site crashed while we were in the middle of dumping the database. Both copies of the database were so corrupt that they would crash immediately upon being loaded.

Our first choice, after realizing that we had this problem, was to restore from a backup made earlier that day, and then "replay" all of the events on Subeta that had happened that day (from about 10:30 am that morning until 7:45 pm when the site crashed). This is possible because the database logs all changes, and we can use those logs to replay events.

However, this method of recovery proved ineffectual - replaying yesterday's events against the backup merely brought the database to the same corrupt state that it was in before.

So we had no other choice but to go ahead and finish dumping and reimporting data (from the backup made yesterday morning). Once that completed (around 1:15 this afternoon), we began replaying yesterday's events, until the site caught up completely, and we had a copy of the database free of corruption.

There are upsides, however. Dumping and reimporting the database optimizes it (it shrank by about 66% as a result). An optimized database is a faster database.

So yeah, that's the whole story. We're working with MySQL to fix the corruption issue, and if that is taking too long or not turning out the way we want it to, we are going to look into alternative database systems. One way or another, we'll find a solution to the problem.

--Alex