Replies

Feb 2, 2018 8 years ago Official
Dill
is practically pickled
User Avatar
Caiman

Known issues that the staff is working to resolve. We appreciate your patience.

Avatar Version: Alaskan Winter

Feb 4, 2018 8 years ago Official
Bug
User Avatar
Segfault

Re: 503s/504s and "page not found on this server" problems.

We traced this back to an issue with our deployment process. Basically, sometimes it would actually take the whole site down for a bit when we deployed an update. In other words, whenever you ran into these issues, it was because one of us had just made a change, and the deployment made the site go down. This happened with about half of our deployments.

We're working on fixing that asap, we definitely don't want the site to keep going down every time we make a change!

more detailed explanation Since moving to PHP7, there are times when the load is so low that Subeta only needs one server. So when we do a deployment and that is the case, there will be a few minutes where the server goes down, and there is no other server for it to switch to.

When a deployment happens, the server has to shut down, get the new code, and then restart. If there are two or more servers, that's fine, because traffic can just be redirected to another server. (And they just do it one at a time.) But when we only have one server up, that server goes down and the whole site goes poof for a bit until it comes back up :c

We have not noticed extreme lag in general. However, we have noticed extreme lag being caused by the site coming back up after a deployment. Our tools show us a spike in activity right as the site comes back up and be related to human avatars (like rebuilding the avatar cache from scratch or something), so that's another thing we're looking into.

Thanks for your patience, I hope this sheds some light on these issues.

The Blue Building quests have their own topic here and is already being worked on: Blue Building Quest Return Please see my staff post on that topic :)

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Feb 11, 2018 8 years ago Official
Bug
User Avatar
Segfault

Just an update on the above: We changed our deployment process so you should no longer see the 404s and 503s! Hooray! (The downside is that deployment takes a bit longer now, but it should no longer take the entire site down.)

We're aware there are still random 504s, but these seem very intermittent and usually go away with a single page refresh. We're still investigating what could be causing these.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Feb 16, 2018 8 years ago Official
Bug
User Avatar
Segfault

We just increased the servers minimum in anticipation of the holiday, so that should help :)

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 8, 2018 8 years ago Official
Spotlight Champion
Destiny
User Avatar
Dexter

Guys, I know that this is frustrating, but lets try to keep the idle chatter to a minimum. If you have a new error to report or something to add, that is great! We are trying to figure out the issues currently and will try and let you know when we figure things out!

Dexter by

Mar 9, 2018 8 years ago Official
Bug
User Avatar
Segfault

We read every post here, which is why we asked to keep idle chatter to a minimum.

This topic isn't so much a bug report as it is just letting us know there's still 504s, 503s, and lag sometimes. We're aware of this, we've been aware of it, and as we've explained in the past, the efforts to address these issues are steady and ongoing.

In recent months, we've done a lot with the intention of addressing lag and errors, and we have been seeing a difference. Here are a few of the things we've done lately:

  • We upgraded our PHP to the latest major version.
  • We moved our servers to better infrastructure.
  • We improved caching in the item vault.
  • We installed new code monitoring tools to help us target inefficient parts of the code.
  • We re-indexed a large database table (battle records) to make it more efficient.
  • We've been rewriting and refactoring slow queries and refactoring old code.
  • We removed a layout dependency that was bloating every page load.
  • We also tried a bunch of things that we haven't kept, but were worth trying, like file caching with HAs - which we may bring back in the future. Not everything we try ends up working out, but it's a process.

You can rest assured that staff is not only aware, but constantly working on solutions. I understand it's frustrating when your personal experience doesn't match up, but we have been doing quite a lot to address this and we have seen a lot of improvement overall.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 25, 2018 8 years ago Official
Bug
User Avatar
Segfault

Thanks for asking! You guys are right, we should clarify what info is most useful to us, so I will do that now and ping everyone so they know. I apologize for not doing this sooner as I should've.

When there is a Cloudflare error, what helps us the most is knowing what specific page it occurred on. It is not necessary to copy-paste what the error says. Instead just include the following:

  1. the error code (504 or 502)
  2. what page you got it on
  3. approximate time you got the error
  4. how long the error lasted. (for instance, were you stuck on the 504 for the next half hour, or you wait 5 seconds, refresh, and it was back to normal?)

For example: "I got a 504 error on the page turning in the items for Pete's quests at 5:03 PM subeta time. The error went away after a single refresh."

This will help us identify if there are specific pages with inefficient/out-dated code we should take a look at. Generally if you only get the error once, and it goes away after a single refresh, it does not need to be reported. If you get the error on the same page more than once, please report it.

A few other things to note:

  • Unfortunately it seems that, while we've mostly fixed the issue, servers going in and out of rotation can still increase the chance of 504 errors. Errors caused by this usually go away if you refresh the page, and don't need to be reported. Servers will move in rotation when 1) we make a deployment (check admin posts as I usually post when I personally deploy anything) and 2) when the number of people online jumps up or down rapidly and extra servers spin up/power down, usually at the beginning of holidays and events.
  • This info applies to the Cloudflare error page, where the entire page turns into a big Cloudflare error. If you're just noticing a SQL error printed somewhere on a normal subeta page, check to see if it is the one explained in this topic and make a new topic if it's not.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 28, 2018 8 years ago Official
Bug
User Avatar
Segfault

Thank you! We are actually aware that the /friends.php page is notoriously laggy - the issue is worse the more friends you have. I will definitely look into updating that specific page.

That is unrelated to this topic. Please make a new topic or ticket about your problem.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 28, 2018 8 years ago Official
Bug
User Avatar
Segfault

I made a new topic for this so the information I wrote here (about what to include in the report, etc) doesn't get buried. Hopefully it will help you help us with your bug reports! :) I will lock this topic now.

Please use this topic from now on: Timeouts, Error Codes (5xx), Lag | FIRST POST UPDATED 7/22

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Please log in to reply to this topic.