Subeta -

December 19, 2008 8:32 pm

Hi everyone. Unfortunately, as many of you noticed we had 2 moderate (more than 15 minutes) periods of downtime today, one this morning and one this evening. I wanted to let you guys know what happened, and what needs to happen as a result. :)

This morning, our database server locked up (which happens to us occasionally, you could call it "growing pains.") This morning's lockup was particularly bad as all 3 web servers became overloaded with people trying to load the site, and so slow that it took forever for me to log into any of them (for example, to take the site down and stop people from hammering the web servers xD)

To further complicate matters, the database became corrupted as a result of the lockup, and refused to start. At that point we switched to our backup database, which is synchronized with the main one, and brought the site back up.

The downtime this evening was the result of db2 locking up, which was corrected with a restart of the database.

This left our primary database server offline though, so Subeta will be taken down over Monday/Tuesday (Tuesday morning) at 3 am in order to synchronize the two and return to operating on db1.

In case you're curious why the database locks up like this: Subeta uses caching to save the answers to frequently asked questions that would otherwise be asked of the database (for example: "give me a random avatar to show on the homepage?"). The cache expires after a certain time, however, and if there's lots of users on the site, hundreds of users suddenly ask the database the same question. You'd think this wouldn't cause a problem, but the database really doesn't know that lots of people are asking the same thing, so it dutifully tries and answer the same question hundreds of times. Meanwhile, all the normal Subeta queries get backed up while 200 database threads compete over access to the same information, and eventually it gets so bogged down that it only answers 100 or so questions/second, as opposed to the 5,000 we normally run.

There are ways around this, of course, and we've been implementing them whenever we come across this problem. For the moment, though, whenever we fix one of these problems (like the random HA on the home page), another one steps up to take its place. Rest assured that each time this happens, progress is being made!

Anyway, in summary, site will be down Tuesday at 3am - 4:30am to resync the database servers.

-Alex

December 31, 1969, 7:00 pm by

thank you for the explanation, you keep a lot of people from pulling out their hair ^^

December 31, 1969, 7:00 pm by

Thanks ;P

December 31, 1969, 7:00 pm by

and i deleted one blee, cause that was what i had twice oO but: man, it's getting full here - guess i rarely have seen about 1200 people online. i hope you guys can handle that =)

December 31, 1969, 7:00 pm by

I deleted all but 1 blee of course

December 31, 1969, 7:00 pm by

Oh, I also went to melody, it was just blank under her picture and I got a Blee.

December 31, 1969, 7:00 pm by

nothing is working, makes me wonder if the DB should be synced now? I went to melody and it gave me 3 blees, but I didn't know until I went to the inventory cuz melody was a blank page

December 31, 1969, 7:00 pm by

thanks for the info sounds like a big headache to me but i'm glad your having it and not me

December 31, 1969, 7:00 pm by

For once, I actually sat down and paid attention, and read this through.

Thanks for explaining. I have a huge lack of database knowledge, so this cleared things up. XD

December 31, 1969, 7:00 pm by

It's really awesome to play on a site where the staff goes the extra mile to explain any problems and let users know the issues are being addressed. Thanks for the update!

December 31, 1969, 7:00 pm by

Thanks for letting us know!

December 31, 1969, 7:00 pm by

As always, thank you for going to the trouble of explaining the reasons for the problems.

December 31, 1969, 7:00 pm by

All technology can have its quirks at awkward times. Not to worry, we all understand how hard you all work at the site and love all the new improvements. Hope the rest of your tech time goes easier!

December 31, 1969, 7:00 pm by

I love your update Alex, you put it in terms I can understand and I know that is not always easy. Thank you so very much for taking the time and effort to do this.

December 31, 1969, 7:00 pm by

Thanks for the update. That seems like so much work.

December 31, 1969, 7:00 pm by

Thanks for the update!

December 31, 1969, 7:00 pm by

Ack, I'm so sorry about the blank post. But I understand now, thanks for explaining! =)

December 31, 1969, 7:00 pm by

Thanks for posting this and keeping us up to date! :3 Also thank you for having it early in the morning..I'll be fast asleep when the downtime is happening

December 31, 1969, 7:00 pm by

Senynith: yeah, usually i try to put an explanation on the downtime page. yesterday morning though, the sheer number of users locked up the web servers (which i would need to access in order to post a downtime message), so i decided to just fix the problem asap instead.

dachshundsr4me/AuntRainey: unfortunately a new database server would not fix the problem as subeta's code currently does not allow using more than 1 db server at once (it is a design goal of the recode to fix this). while a more powerful server might technically help, it is a better use of our resources to continue improving the code so this doesn't happen at all. it's a slower, but far more rewarding process.

PsYcOtIc: actually the most recent lockup was over an old piece of code called configurator which seems to be loading something from the database that really doesn't need to be in the database at all. the new achievement system really didn't play a role in this.

December 31, 1969, 7:00 pm by

Very awesomely written !! I understand everything now

December 31, 1969, 7:00 pm by

yet again a problem caused by something new being introduced into the site and yet again it seems no foresight into what problems may result from this it is good to get a proper explanation though so thank you for that

December 31, 1969, 7:00 pm by

Thanks for letting us know. I'm so glad subeta doesn't suffer from DDoS attacks - that's usually what I'm dealing with + angry customers LOL

December 31, 1969, 7:00 pm by

Well glad know about it then thanks thats my main time for popping on here going to have to try get on before cause that timing is 8am till 9:30am for me.

December 31, 1969, 7:00 pm by

Thanks Alex you are awesome and I don't know about everyone else but i appreciate you going out your way to explain why the site was having problems. Thanks for all your hard work hun

December 31, 1969, 7:00 pm by

That almost went over my head. Almost.

But I got it.

It would be nice to know what's going on on the "site is down" page, but whatever.

THANKS FOR LETTING US KNOW ANYWAYS! ♥

(I really am grateful)

December 31, 1969, 7:00 pm by

have to also remember that alot of the overloading on the servers is 'cause alot of users on at once in one area of the site trying to get the achievements all at one time too...like when i was trying to get the dark matter game one the site all of a sudden when slowdown on me then said i claimed the points but i didn;'t and no achievement so had to wait 6 hrs...thankies for keeping us updated muchly...it is always appreciated...not many game sites do that

December 31, 1969, 7:00 pm by

Thank you for the update... it is like a foreign language to me but I totally understand the scheduled downtime part

December 31, 1969, 7:00 pm by

Big thanks to you for taking the time and having the patience to explain things to everyone in a way we can all understand.

December 31, 1969, 7:00 pm by

Posted by dachshundsr4me

Is there anyway with contributions that we can help possibly getting another server.

Great idea!

December 31, 1969, 7:00 pm by

Alex, you rock! I think this site is growing so fast, the servers can't keep up . Thank you for explaining this to us, and I hope you get the dbs fixed soon. maybe our withdrawals are not a good thing when subeta is down. hehee!

December 31, 1969, 7:00 pm by

Hugs Alex, some days I really think that the site would be eaten by a squeal if it wasn't for you and are other programeys.

December 31, 1969, 7:00 pm by

Thanks for keeping us updated, Alex! (I lol'ed at the downtime-squirrel. XD)

December 31, 1969, 7:00 pm by

You are amazing, Alex! You deal with all this stuff AND manage to keep us all informed, explaining everything so that even people like me can understand what's going on! Thanks for that.

December 31, 1969, 7:00 pm by

I thought the achievement system was causing the trouble XD Glad to know what's really up. Thanks for the update!

December 31, 1969, 7:00 pm by

even though downtinesucksi liked staring at the squirrel alx's name is alex

December 31, 1969, 7:00 pm by

As always thanks for keeping us posted .... and the new achievements are awesome .....

December 31, 1969, 7:00 pm by

Alex, you are wonderful! No other site keeps us as informed as Subeta. I even heard about IE problems here before it hit our news stations!

December 31, 1969, 7:00 pm by

Is there anyway with contributions that we can help possibly getting another server...wouldn't that help as well to help with the growing population..I know most of us here love this site and donate regularly if not to get cash shop items or just to have that super gold account.

December 31, 1969, 7:00 pm by

Thank you for explaining it so well! And for giving us a heads up about the time time. Keep up the great work.

December 31, 1969, 7:00 pm by

As long as it's being fixed.. thanks for the heads up!

December 31, 1969, 7:00 pm by

I don't understand much of what you just said, but I appreciate the effort, lol. And a 15 minute down time is a biggie? Not in my experience. Maybe those multiple day downtimes are a thing of the past! Alx, you're great.

December 31, 1969, 7:00 pm by

Thanks for the heads up! I'll be asleep then anyway, but it's nice to know we're being kept in the loop~

December 31, 1969, 7:00 pm by

I hope the Melody's Cottage items stack for those 2 days on the next day the site is up =3

December 31, 1969, 7:00 pm by

Better you than me- keep up the good work!

December 31, 1969, 7:00 pm by

Thanks for tellin' us. :3

December 31, 1969, 7:00 pm by

Oh, that's what that was. Alrighty, good thing it'll be resolved.

December 31, 1969, 7:00 pm by

Thanks for the warning ahead of time ^^ It's good to know when this site is going to go down so I can find ways to occupy myself :p

Is this going to affect the advent calender at all though? Or will the down time be short enough for us to still complete it?

December 31, 1969, 7:00 pm by

Thanks 4 telling me guys!

December 31, 1969, 7:00 pm by

Thanks for keeping all of us updated. I really appreciate the time the staff takes to do so, as well as keep the site in working order. It means a lot to me and the other users. &;

December 31, 1969, 7:00 pm by

thanks for the upate

Use @username to mention someone. Supports markdown formatting.

Logged Out

Leave a Comment