Hello everyone! I'm sure you've noticed that things didn't go exactly as planned during our site maintenance.

Goals Our goal for the site maintenance was to:

• Move our site application code to more robust servers, with better managing of auto scale. There were a few reasons for this, but one is cost reduction. If we can run fewer servers in the nighttime, and more high power servers during the day, overall we lower costs. We've had auto scaling for awhile, but it's never worked exactly well.
• Move our database to digital ocean, where we have way greater control over backups, adding read replicas, etc.

Didn't go well Our application code move :(

You'll notice that the site is 502'ing, and having other issues. We worked with the provider before hand to verify that Subeta was a good fit, and after a day on the service, it's clear that it isn't the case. We're working with the provider to come up with a solution (we'll be moving).

The good news is: This can happen without downtime. I'm investigating other options, and also considering spinning up servers on other services (aws, azure, google cloud) to give us some support.

Our code can run anywhere as long as it can connect to our various other services (database, caching layer, etc) so I'm hoping to get this starting to get leveled out throughout today and tonight.

Went well The database move! It's humming along happily, having four months of security and performance related patches applied :heart-eyes: I can see the backups from the last two nights, and can create additional ones with no worries.

Another benefit is we have a failover node for our database, if things start to get too out of control or the database locks up, it can route it's traffic to it's failover node. This should protect us from a lot of the database related pain we've had in the past.

We're aware of

• Bathhouse issue - we're going to reset those counts for everyone, don't worry!
• Users online
• Events going out
• General lag / pages breaking :(

Next Steps

Continue to try and find a good home for our application code. The API code (sapi.subeta.net) can live anywhere relatively easily, the aging codebase is another story. I'll have more updates on that hopefully in the upcoming weeks.

We're going to extend Masq as well!

Alright, after a day of working on the servers, I've made some progress. There is still a couple of things needed to get the site running 100%, but they should happen throughout tomorrow.

Subeta.net Subeta.net is now running on an entirely new configuration, with failover. The logs are all being centralized and actually scanned for important details (like a service failing)

sapi.subeta.net This powers the wardrobe and the forums, and is currently not moved. That means it is likely to have speed and degradation issues.

The code has been moved and set up, but there is a final piece of work needed from the service to get it running.

More Updates!

• The site (subeta.net) should be more or less stable now in terms of going up & down.
• The "path" URLs (/explore/, /games/, etc) were redirecting to the home page, this has been fixed
• Home page being blank has been fixed

Still problems

• Some events are not going out
• Autopricer is broken (we will be giving more gold account time once things are stable enough for it to be enjoyed!)
• API (forums, wardrobe) are on a different caching system, and will posts may show up late

Next Steps Getting the API onto the same new system as the site. The code is there, the services are set up, we just need support to answer and resolve one issue with the domain (pointing to sapi.subeta.net). Once that's ready, we'll move it over and the cache issues between the two should resolve.

💤 Final update for the night!

• API server moved over, woo! The way I had to do it wasn't the way I'd have liked to, so I'm going to revisit this once the rest of things have flattened out
• Events should be going out!
• Autopricing should work now
• Cache on the forums & elsewhere should be resolved

💤 Final update for the night!

• API server moved over, woo! The way I had to do it wasn't the way I'd have liked to, so I'm going to revisit this once the rest of things have flattened out
• Events should be going out!
• Autopricing should work now
• Cache on the forums & elsewhere should be resolved

💤 Final update for the night!

• API server moved over, woo! The way I had to do it wasn't the way I'd have liked to, so I'm going to revisit this once the rest of things have flattened out
• Events should be going out!
• Autopricing should work now
• Cache on the forums & elsewhere should be resolved

Hey guys! This is a message from the programmers and staff!

We are continuing to work on the issue, and should have a time for scheduled maintenance posted in the next day or two.

We’ve successfully migrated the code (both the site, and the api) but that has resulted in the DB being in a different location, which is what is causing the lag / downtime, is the round trips. We make a lot of queries (way more than we should) and so that adds up to basically php just queueing up after an hour or so.

We’re figuring out what the next phase is and firefighting while we’re doing that. We do have our own slack channel that we’re all in communicating in hourly, with updates on user issues and server issues and next steps on solving the user issues as they come up. So please don't think we are ignoring this! We are just really trying to work behind the scenes.

We apologize for all this downtime and really hope to have the site back to working soon. Thank you for your patience! We know this is really frustrating especially with the additional real life stuff going on. We hope you are staying safe in the meantime.

[edit] I also want to remind that staff posts do show up almost automatically in the admin posts since the cache is lagging on posts right now! I know that is frustrating. https://subeta.net/forums.php/admin_posts/

I apologize for not responding sooner to the posts about me posting to the news. I personally do not have access to post to the news.

There is a separate board for the job agency which is listed below. Programmers are aware!

Job Agency page not showing my pets

Hey everyone, still continuing to make improvements. I'm keeping a public changelog here, it's a little more technical in nature but if you're interested in what I'm doing hour by hour this is a pretty good log.

I'm not going to backfill the last few days, but hopefully seeing something like this helps you understand that a lot of work is going on behind the scenes daily.

Just pushed out some major changes to our infrastructure, which, should make the site usable again.

More technical updates here with a backlog of the changes as they happened as well

And I know this isn't really a consolation for anyone, BUT, you can use any of your saved outfits here now! Was a bug with the new limb system.