You may have noticed some issues with wardrobe, such as HAs not loading or wearable previews not showing up, along with forum issues.

Subeta has a lot of different features, but by far the most intensives are the wardrobe and the forums. Those in particular are grouped together with their own servers on Subeta V2 with more memory and ability to spin up for events and so forth. The issues you're seeing right now are related to the Subeta V2 servers running out of memory and not tripping the notifications when that happens, and thus not clearing the data automatically (this kind of situation is also the reason why sometimes pings don't go through, or shop events don't show up for people). Keith cycled things manually the other day and was trying to figure out why it's not handling running out of memory correctly, but Subeta's infrastructure is bespoke and it's going to take a bit more pinning down. So this is in progress! I'm going to be updating the status page as well. Thank you!

You don't need to do anything different, the matter is with the servers and that is what we need to work on to get things running right!

You are correct, it isn't the first time this has happened and it's part of why it's taking a bit at the moment.

Subeta's server and database needs are very particular, there aren't a lot of out-of-the-box solutions or even professional services that can deal with it all (Keith has tried). So he ends up doing a lot of code and having to troubleshoot it himself, and this is something where what looks like a good fix sprouts a leak later and there isn't another document or coder to help since, well, bespoke. Subeta is not a corporate product by any means, for better and for worse, but that makes it all the more meaningful when y'all stick with us through these sorts of things. Thank you so much. <3

Yes, apologies, it's my general practice to give people chances for forum tickets as well as being able to have updates show up in Staff Posts but it's not perfect when the forums are having issues. I wasn't quite tracking that part of it through this sinus infection when I posted it. Thank you for the reminder! The announcement should show up on the front page still and I will make sure to update that for visibility as we get more info.

Update: Keith is going to rebuild the API servers entirely from a backup, which will take about 15 minutes but should improve things rather than just patch the issue.

Try putting an outfit on again, please?

Ah, that's very useful! Thank you for the detail, I'll pass it along!

Have you tried clearing cache and resetting your avatar? keith restarted the servers again...fingers crossed

Well, boo, the restarts aren't working. Keith's going to roll his sleeves up and dig back in at around 6pm-ish Subeta time. Thank you, everyone, for testing and contributing data, we'll get this sorted!

In terms of what was going on, there were issues with the servers' local cache directories filling up, so we disabled that while investigating. Because of that, it was making a request for every image on the avatar (separate piece of clothing) and Amazon started rate limiting the requests thinking they were fraudulent. Restarting the servers wasn't helping because Keith needed to go into an account setting and confirm that Subeta hadn't been hacked by a botnet or anything. However, it should be fixed now!

We're going to keep looking at why those cache directories keep filling up, too, but for now things should be good with Morostide (my personal favorite season) just around the corner. :D