Actively working to address this today!

I am going to be around for the remainder of the evening, to keep monitoring and I'm ready to react if it gets worse again. If you notice it becoming severe again please keep letting us know on this topic :)

(if you are getting multiple 504/502 errors, or if pages are taking longer than 10 seconds to load, etc)

--

You're welcome! I'm glad it is helping <3

Thanks for the ping! We've addressed the recent blip. and I will keep monitoring :)

Thank you! We think we've got it running smoothly again.

We are looking into the 404 errors (and related server issues) right now.

Hmm, I'm not experiencing any lag and all of our metrics are showing page loads under 300ms, are there particular pages that you're getting huge load on? Specific user shops?

We just pushed a caching fix that should hopefully resolve some of these problems! In pushing it we had to reset our cache, so that means for a few minutes the cache was "cold" and had to warm up with everything that should be in it.

Should be improving over the next few minutes.

Have you guys noted any recent improvement with the site speed and pages loading better?

If not, can you make a note with which areas you're having trouble with and on which browser?

Thanks! <3

Hey guys, quick update.

I relayed the information about midnight to , and he did some stuff to try and mitigate it. So we are glad to hear that midnight wasn't as bad tonight!

I did encounter a few server issues about 20-30 minutes ago, but I was manually working on those and hopefully they are resolved now.

I will continue monitoring for a little bit tonight before heading to bed. If it does become laggy again, please be patient and understand we'll get to it ASAP if not already on the case. <3

We're aware there was an incident that began around 2 AM and lasted throughout the morning. The cause was unrelated to the opening of the pawn shop. I'll ask if he can post a little more here about what happened (but please understand if he is not able to disclose more information)

Things seem to be loading decently fast right now though - at least on my end? Is that the case for you guys too?

I took the time to list how we've been communicating over this, so we can see what we did right, and where we dropped the ball:

12/2 - Brief period of 404 errors, addressed here.

12/5 - Keith posts a response with some questions about the lag to help him debug it

12/6 - Keith explains another fix he tried and people reply that they notice it helps a lot.

• Please note 12/8 and 12/9 were the weekend so we weren't online much -

12/14 - Reya (one of our lovely minimods!) made a staff post as Keith and I were too busy coding during the week to post.

12/16 - That brings us to my update this morning and today.

Firstly... we want to thank you for sticking by us for so long.

You've seen us through 10 years, and that's awesome. It's been frustrating on both sides; sometimes we feel like we're in some kind of internet race where technology is moving so fast and we'll never quite catch up! But we're not giving up on the race. We'll be taking that participation medal, and we appreciate everyone that's helped us not give up - by being here with us.

What we'd like to say is... please don't give up on us. We're so close. You can see how close we are just on this topic - we've defeated the lag more than a few times, we just have to make it stick!

Please don't feel bad for using your voice. We care to see that YOU care, and you (all of you!) are the reason we keep trying.

It looks like 2 AM is when the site gets unusably laggy, and stays that way through the morning. We've tweaked some server settings that affect that block of time, so hopefully tonight it will not be as bad. Fingers crossed!

[edit] If you are wondering about the lag in the past hour - That was actually due to a deployment that randomly caused some issues on just ONE of the servers we had up. I was deploying some code to tweak the new quest's settings. I had to manually murder the rogue server, which may have caused lag and errors while that was going on. It seems to have cleared up by now?

Hi everyone - I just logged on.

Sounds like our changes yesterday helped a LOT with the 2 AM and overnight lag. Hooray!

It looks like today, there was an incident around 4 PM (Subeta time) affecting one of our servers. It looks like it lasted about an hour? We are investigating what happened and how to prevent it.

Right now, all servers are showing Healthy again - Let's hope it stays that way :)

[edit] Hm, I do notice a couple forum loading issues are ongoing. Looking into it.

We have identified and resolved the 404 errors. Thank you for the report.

The forum issue seems to be ongoing unfortunately, and we are still actively looking into this. What I notice is that they frequently don't load (that is, seem to load forever) but then they'll load if you refresh the page. So if you run into this, as a temporary workaround, try refreshing the page if the forum is taking longer than a few seconds to load.

We are aware of problems with the forums today. We are looking into it and will try to keep you updated! (You can check Admin Posts for updates if you're unable to load the forums.) The rest of the site seems to be okay :)

Thanks for the reports! We are still watching this constantly even if we don't reply here. Recently, our issue is that sometimes a single server instance will go down (which explains why it'll be laggy for some users but perfectly fine for others at the same time) - we're working on a more proper fix from what we have now.

• Are you experiencing this right now?
• How long has this been ongoing?
• Does it ever seem to get better (if only for a few minutes)?

Ok, thank you for the info :)