It seems to be better this morning after the initial shock of last night's news. Is it better for you?

I have reported the continued lag and errors. Please please please keep everything else to site feedback! Thank you ❤

We use something called composer to manage our third party scripts and this morning our installation across our web servers was acting up due to an old configuration setting we'd set :( We've repaired it (and changed our configuration to the most up to date) so that shouldn't happen again.

Otherwise we pushed some really big server changes the other day that should make the site move a little faster with hopefully less 5xx errors until we can get the new shops pushed out. This includes a new "proxy" between Subeta and our database that does some really clever connection stuff, but also lets us see all of the really bad queries on the site in a way we haven't been able to visualize in the past. Unsurprisingly all of the worst queries are connected to user shops, but there are some other ones that we didn't anticipate!

Just some older pieces of the site we don't think about very often. The donation center is an example of this haha

The old shops are still live. I got my first 502 in almost a week today, excited to see how we do without the old shops!

I want to address this idea that this shop change would completely eradicate the current errors. We said we hoped they'd help (and they have!) with the server errors because the legacy shops were (and still are, to an extent) a major bottleneck in our server-database connection. We're working on other things, too, but I'm afraid that we only believed the shops to be a MAJOR factor, not the entire factor in solving our problems. Removing this bottleneck, however, clears out a lot of our logs and lets us know what we should focus on going forward. The decision to keep legacy shops in the shop search has not helped that cause, but it was a necessary decision to keep those items available and in the economy.

I know it's frustrating to have to deal with this, and trust me when I say it's just as frustrating for us as it is for you. But just to demostrate that the new shops really have helped, I wanted to show you guys a graph of all of our 5xx errors. Something funky is definitely going on today, and I'm trying to figure out what, but our average 502/504 errors have dropped by more than 75% since we implemented the new shops.

I'm going to chime in to the great post that made and make a few points.

1. How we find these bottlenecks and how we deal with them. We have a few tools that include outputs from our database, and the queries that take the most time. For the past six to eight months the user shops tables have been a MASSIVE chunk of those logs, to the point where we have to do significant, digging to find other slower queries around that. NOW those queries are all gone, and we have the ability to really dig into the other queries that are causing problems. Fortunately, those queries are all smaller (taking a look right now - things like searching items by tag, the donation corner, etc) that don't require massive overhauls.
2. Vesnali is a great point. This gives us a great opportunity to understand why the site buckled under that pressure. We had the hypothesis that it was primarily items being moved to/from the user shops and that turned out to be mostly true, but now we can see what the smaller pressure points are.

Hopefully from all of these changes we can show you that chart again in a few days with it back on the downswing!

[-Guru] the thing is, most of the time when you're getting them, everyone is getting them. if a single page takes a while to load, you may see an isolated one, which was probably the case with you previously removing items from a large gallery. but if you're getting them in random places, that means the server-database connections froze up and it is a site-wide problem for everyone at that time.

They are causing them. You can read into my words if you'd like, but I didn't say anything about it being the only cause, or that the shops would be a 100% guaranteed fix. I'm really glad they've helped so much already--even with the old system still partially in use. It's pretty clear to us that it was a huge part of the problem.

went into an incredible amount of detail in the FAQ.

In her answer to the very first question:

I just pushed some server changes that should get new servers spun up before errors start happening. :)

Thanks for reporting these errors, , , and ! Will have a news post soon about them.

[edit]

As the news post said, we were looking into what exactly happened with our MongoDB provider. It was in fact a failure with our MongoDB database. We're going to work on getting the data hosted in multiple places so we don't have a single point of failure like yesterday. Once again, thank you everyone that let us know that you saw these errors popping up! :)

Looking into these new errors