If you check our status site (which gets linked when the site is down and has a forum in site updates), you'll see we've been investigating and monitoring a lot of the issues happening. It's better to update the status page than constantly posting on the forums because you can still access the Subeta Status page when the site is down!

Like Amber said, we're trying to consolidate all of this information to our Status Site. The reason is really simple: it's there if the site is down. If Subeta isn't running, you have to go to facebook/twitter/tumblr/myspace to get that information.

Here it's in one place that is persistent. We're going to get better at writing what are called postmortems where we write basically what went wrong, and how we're avoiding having that issue happen again in the future.

And for ongoing issues we continue to make tasks. You'll notice that it has pieces listed as partially degraded performance.

This has been posted around. It's now listed in the footer, available on every page.

The news is not the place for this kind of information specifically because it changes so quickly, and often.

So, if we are to go with what the status site says, no one from staff is doing anything to deal with the lag and errors that have been occurring since 12:11 EDT... is that right?

We're aware of the degraded performance :)

- in the past, we had many artists doing not that much art each. Now, we have fewer artists with more to do. Which is what we'd prefer, because it means we can really work on the art quality. We had one artist leave us in the last few years due to health reasons, and rather than replace this artist we increased the work load of another artist.

Also, I go on in that post to give further comments as to how we don't actually consider it necessary for that particular staff member to work any more quickly on the subetapedia articles as the rate at which they're being released works for the site. We're not able to hire extra staff members for things that don't require them as urgently. We have a really good team going!

The errors are hard and we're trying and very aware of how hard they are for everyone, including us, but the best thing people can do right now is to continue to support us if they can as we work to fix the problem. Scaremongering isn't going to help anybody. Support helps us. (I don't just mean spending in the cash shop)

- there's no need to be sorry. We'll be fighting for this site until the end, whenever that may be :) I'm not here to say TRUST US because, well, you can't really just tell someone to do that. But you can ask for support, and that is what we're asking for. You don't have to give it, that's really ok. Nobody should have to set themselves on fire to keep somebody else warm. But if you can, then that's nice! And we appreciate it a lot. Please don't be afraid!

(Oh btw, Scarsdale left us due to being pretty busy with school, he hadn't been able to do a lot of work for us for a while. Same as with the artists, another programmer took on his extra workload.)

Today's maintenance did fix something :)! Unfortunately not really what we wanted it to fix, but it gave us additional reporting tools to see what is going wrong.

I've got a call with the database people in a few hours to talk about why the in-memory stuff didn't work out and will report back later :)

Routine maintenance isn't the problem, or the solution. I've just finished with Clustrix support (this was a long process than anticipated to get the reporting up and running), and basically what this update provided was a significantly better reporting structure for the database system. Before we had "the database server load is x%" and now we have something more like this with detailed reporting on specific queries that are taking a lot of time.

We're holding off on any further posts about this because we're still going to have the 504/502 errors (those are essentially PHP freezing up) until these queries are shaped up. Beyond that, we need to remove MongoDB from the site entirely (the wardrobe is the last feature that currently uses it - and will be removed in a future update). Since the news is "we're still going to have errors" its not really worth posting - the update was completed (as shown on every page on the site until you removed the alert) and we're continuing to make tweaks. I'm working on a much longer post that gives an idea as to where we're heading (and API updates) but we need to really dig into this new information made available to us before we can start making any sort of comments on changes to come.

The post said that we'd post a reminder on Monday and we didn't, and that's absolutely my fault. I'm glad that the downtime was less than 20 minutes, otherwise I'm sure some people would have actually forgotten / missed out on something.

As far as communication goes, we can always be better, but on the other hand it's a fine line. Blizzard was used as an example, and you'll see through their 10 years of WoW they've gone back and forth and at some points removed ALL communication outside of official channels (posts about downtime, etc) and support channels. They've gone back and forth and right now they're pretty open and respond on twitter, but that hasn't always been the case.

I'll point out that 99% of the time in agile development feedback does not come from threads like this, but user interviews and research. Threads like this where it turns into a mob (either one way or the other) or users can influence each other != agile development.

I just want to remind everyone to stay on topic and remember the community guidelines.

Per the status page, we're working on it. We've got a couple of people on this morning monitoring issues and resetting the site as needed.

Hey everyone! and I just pushed a rewrite of a part of the class that handles a lot of our database queries around the site. Basically, anytime something requests something from the database, it goes through this file. We've made some optimizations based on current technology and spent the last day testing those changes.

It's mostly focused around the BIG queries on the site (so focused around places like the forums, user shops, battle, etc) so we're hoping to see some improvements there, which means less queries that lock up the database in general. This is a pretty big deal, and a pretty big shift, but we tested on the major parts of the site so it shouldn't make things worse (crossing my fingers)

I've got our charts open to the historical view (image here) watching to see the effect of this on the queries, and then we can tweak from there.

Autopricing was one of the fixes we pushed last night, and was tested, I just didn't want to say FIXED unless we're sure it's fixed (which we're watching now). Hopefully, it's working for you, and if it doesn't we get error reporting on our end (+ user error reports) that we're continuing to debug.

I didn't post to the site update group for the same reason - I'm watching to see how it works on the site. I'm posting here to say that we made a change and that we're seeing if it makes any difference.

Fixing ?! You'll get another notification once it's fixed ;)

Gold accounts are tied to your account, which is cached. Everyone should be seeing their GA benefits activate now or over the next few minutes as the cache expires :)

everyone was given GA time. Users with existing gold accounts got two weeks, everyone else got 7 days :)

Please remember to be civil. Community Guidelines are still in effect. Please do not attack other posters. I know that this is a hot subject and tempers are high, but be nice. Thanks. :)

- we're still very much trucking along in the Creative Department (Luminaire awaits!). I haven't personally had continuous errors for a while now, so I haven't been able to personally point out where and when they're happening anymore. It seems like different people are getting different errors (which makes things hard to pinpoint/fix, I assume) - but I'm afraid I don't have a lot of the answers you're looking for. I know that Keith is testing things a lot, I'm often asked to help if I'm around and not REALLY busy (rare).

Please don't take your frustrations out on staff members who are trying to be helpful. We may not have the specific information you're looking for at the time you're asking for it, but that doesn't mean that we're purposefully trying to be vague, and it certainly doesn't mean we're abandoning you. We are a staff of ~20 people, the vast majority of whom are part time, and have pretty specific duties. People who do artwork for us, or answer your tickets, or release site content just DON'T have the technical answers that you're looking for.

I will admit that I'm frustrated to see posts about the "staff at whole" going silent or "abandoning the site". I think that's an unfair generalization. We release new content just about every day. In order to do that, our artists are working consistently. We've fought through the frustrations of the technical difficulties of the site to continue to provide regular site events, special events, and to continue to try out new features where appropriate. Our customer service team claims and responds to tickets 7 days a week. We work hard to allocate the money you spend to pay our bills to keep the site running, to fund improvements where we can, and to pay our staff fairly for their work.

That being said - we all take days off. When issues pop up, sometimes the staffers that are on are not the staffers that can actually solve the problem. We have systems in place to get those people contacted, but on a Saturday morning of a holiday weekend...sometimes we drop the ball and can't make those systems work as quickly as we'd like them to. We apologize, we try to get better, and we try to keep going forward. All of the measures we have in place to give you guys information - the status site, social media, the news - all still have to be physically updated by one of us, and as much as we'd like to be perfect in doing all of that in sync, in a timely manner, and with as many specifics as we can, it's still a work in progress. We understand that's disappointing and frustrating for a lot of you who can visualize things working much better, but that's the reality. We're not perfect - we're not even great most of the time - but we're trying, we're listening, and each of us is doing what we can within the scope of our skills and our duties to the site (and to you).

We have made the decision to lock this thread. We are beyond the scope of "feedback" at this point, and some of the responses are bordering on being in violation of the community guidelines. As new information about the technical failures we have been experiencing are made available, they will be communicated to you.