Followup
Filter
Show Official Posts OnlyReplies
Hello everyone! I'm sure you've noticed that things didn't go exactly as planned during our site maintenance.
Goals Our goal for the site maintenance was to:
- Move our site application code to more robust servers, with better managing of auto scale. There were a few reasons for this, but one is cost reduction. If we can run fewer servers in the nighttime, and more high power servers during the day, overall we lower costs. We've had auto scaling for awhile, but it's never worked exactly well.
- Move our database to digital ocean, where we have way greater control over backups, adding read replicas, etc.
Didn't go well Our application code move :(
You'll notice that the site is 502'ing, and having other issues. We worked with the provider before hand to verify that Subeta was a good fit, and after a day on the service, it's clear that it isn't the case. We're working with the provider to come up with a solution (we'll be moving).
The good news is: This can happen without downtime. I'm investigating other options, and also considering spinning up servers on other services (aws, azure, google cloud) to give us some support.
Our code can run anywhere as long as it can connect to our various other services (database, caching layer, etc) so I'm hoping to get this starting to get leveled out throughout today and tonight.
Went well The database move! It's humming along happily, having four months of security and performance related patches applied :heart-eyes: I can see the backups from the last two nights, and can create additional ones with no worries.
Another benefit is we have a failover node for our database, if things start to get too out of control or the database locks up, it can route it's traffic to it's failover node. This should protect us from a lot of the database related pain we've had in the past.
We're aware of
- Bathhouse issue - we're going to reset those counts for everyone, don't worry!
- Users online
- Events going out
- General lag / pages breaking :(
Next Steps
Continue to try and find a good home for our application code. The API code (sapi.subeta.net) can live anywhere relatively easily, the aging codebase is another story. I'll have more updates on that hopefully in the upcoming weeks.
We're going to extend Masq as well!
💖 ✨ 🤗
tbh I'd be fine with downtime if it meant the site running better!
That said, at least there's work being done. Thank you guys for not giving up on the site and continuing your best to improve it! Especially since a lot of the userbase (me included) can get cranky. Maybe we need that downtime so we can take naps. LOL
[edit] Not sure if related whatsoever but it seems that quoting a person specifically causes the error at the top of my post!
To any Staff:
DO we go to bathhouse and "take a dip", even though it says "This is your first day visiting the bathhouse! - Which is very wrong { should be something like 70 days ] OR Just leave it alone, til count is corrected???
thanks for the update! cheering you all on in your continued work ❤ i hope it goes smoothly~
lovely art by
"grey would be the colour if i had a heart" ~nin ❤️
Would it be possible to get an extra day of weekend quest once things are settled to make up for yesterday?
A new error I havent seen before. I got it like 15-20min ago
[flower=CastlesInTheSky] [tot=CastlesInTheSky] Dear users who keep blocking me....Remind me again, who TF are yous? ¯(ツ)/¯
So yeah, can't read this...
Would it be possible to make a news post, please? I haven't been able to have a single post load since the downtime.
Would love to read the followup but NOTHING LOADS.
infinite parallel universes containing every possible situation.
It makes me happy, because I know, somewhere, you love me back."
While I appreciate the update, maybe this should be in a news post given that the forums are barely loading?
Couldn't read your update info, because the forums won't load for me! Using Google chrome. Cleared my cache, and restarted but still can't read anything. If anyone can read this.., please make a news post. That, at least, is loading.
Provides link to forum post. FORUMS WON'T LOAD.
I won't say that was a bad idea... but... that was a really bad idea.
Not sure if this will even post - but the only reason I have any idea what this thread says is because of source code that a friend was able to copy and paste. Forums are completely broken on the user end.
I cannot load a single forum anywhere. I cannot access my masq dancing threads that I've been working on to slot on items, I cannot access any chats any slotting threads, nothing.
On top of that, it appears any of the notification timers that usually let you know that you can train/play/etc again are completely broken as well. I hope y'all can see this on your staff side because I have no clue if it's going to go through. The entire thread is just that loading symbol for me.
Oh, it'd be nice if this was posted in a regular page since... er... forums aren't loading??
I sure wish I could read this, instead I have the black loading bar loading forever.
I will now try to guess what it says: Something something something screwed up, we're working on it?
Why update us here? We can’t see it, forums won’t load.
I wish I could actually read this announcement because I'm sure it's important, but I haven't been able to load any threads since the downtime. :(
he/him / 31 / EST |  | My best friend is |
I clicked 5 times, 4/5 I got an error message. I hope we don't lose things we've worked 12 years to get. :/
Cannot read thread. Forum doesn't load. No idea if this post will go through.
Only way I could read the OP was through the Admin Posts link. Hope there will be a working way to read future updates soon! :)
“Justice will not be served until those who are unaffected are as outraged as those who are.” - Benjamin Franklin
"Woe to those who make unjust laws, to those who issue oppressive decrees, to deprive the poor of their rights and withhold justice from the oppressed of my people, making widows their prey and robbing the orphan." - Isaiah 10:1-2
