We are keeping an eye on things!

We did turn on DDoS protection just in case somebody was attacking the site. We don't know if that is in fact the case, or who it could be, but it looks like the site is loading better after we turned it on.

The avatar system is having problems loading wearable image data, which is why fully naked avatars still show up. I am still looking into why this error is suddenly occurring.

As far as we can tell, this has nothing to do with the holiday-specific activities or with recent site updates.

Finally, please remember that the other three have full-time jobs outside of Subeta (and live on the west coast) so they're not able to constantly monitor this situation. They do the best they can despite that, which is how DDoS protection and other measures were taken earlier today (probably sacrificed some lunch breaks). Right now I'm the only one here as far as I can tell, and as an assistant administrator I don't have full access to our production servers so it's limited what things I can do especially if we are actually under attack.

Since our SAPI avatar images and previews are broken right now, I wanted to let everyone know there is an option in your Preferences that you can turn on to show "simple" previews instead of the overlay previews generated by SAPI. This won't fix your avatars, but will at least let you see wearable previews in item hovers while we deal with the other problems.

You can toggle it on this page: https://subeta.net/preferences.php?act=prefs

Look for the option that says: "Force item preview hover to show hex. (Zombie skin, zapper skin, etc). Slower hover loading!"

And check disable to disable the special preview.

has (maybe? hopefully???) fixed it!! I can see all your beautiful HAs again :D

OUR SAVIOR.

Nope! Turns out, it was an issue with memcache (the thing we use to do all of the caching around the site) and you guys know how important caching is. When things stopped getting cached, it's more like we start ddos'ing ourselves with the amount of data we keep loading everywhere D:

It took us a lot of effort to pinpoint that the cache and not other stuff was the root cause of the issue. Once we figured that out, we just had to do a reset of the cache to resolve it.

Thank you for your patience with the issues today. We identified today's issue as having to do with old code on the Luminaire trees "Gifts Given" page, it was running a lot of slow queries that were tying up the database. I pushed an update to this code that should help a lot and resolve that issue. We will keep monitoring it to see if that's good or if anything else needs to be done.

There's just one last thing I wanted to say. I understand there's been issues in the past with these things. But right now, I promise you, we're doing our best to diagnose and address these issues when they come up, and we're making an effort to be better than we have in the past about dealing with these things promptly and providing communication when they happen.

We tried to keep the status page updated today, and will continue making an effort to keep that updated. However we might not be able to provide extensive or frequent updates while we are addressing these issues. That doesn't mean we're not working on it, it probably actually means that we are too busy working on it to look at the forums and stuff.

I understand the frustration and needing to vent. But please understand that we're also human, and it can actually make it harder for us to focus on work if you're directing anger at us. (I think many people can understand this, imagine doing your job while someone is yelling at you, it makes the job much harder.) On the other hand, small messages of encouragement, appreciation and support can go a long way, as when we read those it motivates us to work harder for you :)

We're not a big website like facebook or tumblr or, well, even neopets lol. We don't have a big company behind us, we don't have multiple dev teams, we don't have stacks of money - we mostly just have a small handful of people who are here only cuz we genuinely care.

Since this has been cleared up, I am going to lock this up guys! Glad the site is running smoothly :D