Specialty shops such as the Millionaire Center are being temporarily disabled as part of Keith looking into the lag, but they'll be back! Thanks for bearing with us!

EDIT: Specialty shops are re-enabled, and server and database load have been rebalanced as well! We'll continue to tweak as needed.

We'll be looking at the timing for the Hydragellos, but we do want to be careful about the Gelly War overlapping with Vesnali.

If you run into blank white pages or a page hangs, please reload! They should be less common, but site management can sometimes be an art rather than a science.

👋 Good morning everyone!

Just pushing out some infrastructure changes that should hopefully keep the site moving quickly, and coming here to give an update and a little bit more information.

The problems that face the site right now are cascading, which means that one single failure point can bring the rest of the site down. In this case, the boss battle added an extreme amount of stress on our caching layer (Redis), which everything else relies on now.

asked me the excellent question of "why now" in our internal chat, and the reason is honestly 🤷‍♂️. We have a lot more pages that rely on Redis and caching, we have a lot more information going through those places, and it's really hard to debug the battle code, most of which hasn't been touched in 10 years.

We have charts that show us the relative usage across the site, and roughly where load is coming from. In our newer areas of the site (wardrobe in particular) we can see down to the line of code that is causing problems, and unfortunately we can't see that level of feedback or detail in the old code. It would take weeks (at least) to rig up the old code to have that kind of visibility, and that is time and resources we just don't have.

So the solution for something like this is "quickly try to find the cause, and if not, throw more money at the problem". In this case, that meant spinning up additional "legacy" servers (what powers "old" subeta), and a new, more powerful Redis instance. These things take time to come up, and cost a significant amount of money on start up / shut down. For example, the next tier of Redis means an additional $500 (bi-weekly) and requires some hand holding to downgrade after an event. We do these things proactively (we were already running the largest Redis instance we've ever needed) and sometimes the duct tape still fails to hold things together.

So: We are not going to extend the boss fight. The nature of this is the case where if we extended, we'd just be doing the same thing (fire fighting in the background and hoping the site won't keep crashing) and promising to extend, and then again, and then again. I'm going to chat with galaxia, cranberry and jason about how we can do something that isn't extending this fight for everyone, or a way that we can at least give y'all SOMETHING for hitting the button over and over when the site is barely up.

Thank you for being patient with us, we want the site to work as well as you do (probably better, none of you get paged at 3am when the site goes down, a pager I've now carried for.... 15 years?) and are working on some exciting updates to get us in that direction, that we aren't quite ready to talk about publicly yet. I want to spend way less time fire fighting and way more time working on new things... ;)

And an image for illustration:

You can see a MASSIVE redis spike happen, which then throws the whole site into disarray, because everything is trying to grab from, and write to, the cache at once. And then people are refreshing to get the site to load, which is trying to grab more from the cache, and then the cron gets put into a queue, etc.

We're still working on the current war, so we're not doing specific planning for the future ones right now!