today's update: kumos downtime and fixes

whew, today was... a LOT. apologies to everyone trying to make kumos.subeta.net your new subeta home. here's a breakdown of what happened, what caused the issues, and what’s been resolved so far. 🛠️

"we've updated the site" errors

• this error has been popping up for weeks due to two reasons:

  1. anytime we push new code.
  2. when your session moves to a different kumos server (sessions typically stick to one server for ~15 minutes, but sometimes you're moved to balance traffic).

• when moved, the system couldn’t properly transfer your session, which caused you to either:

  • get signed out (if "keep me logged in" wasn’t on).
  • see the error and have to refresh a few times.

✅ fix: sessions now transfer properly between servers. this issue should be gone!

scaling issues

• kumos scaling has been more complex than legacy due to its architecture. for legacy, adding more "web" servers was straightforward, but kumos (built on laravel) requires more coordination. laravel’s complexity allows for cool features (e.g., fast quests) but makes scaling harder.

• today, after pushing quest-related code, the kumos servers crashed entirely. i had to switch kumos into "single server" mode to investigate the problem.

✅ fix: kumos can now deploy code and handle traffic across multiple servers again.

livewire issues ⚡

• livewire (https://livewire.laravel.com/) powers instant interactions like the shoutbox and new smail pages. the sidebar and its manager also rely on it.

• recently, i’ve been using a new method called livewire volt, which fits my usual coding style but isn’t mature enough for a project as big as subeta.

• over the past few months, i’ve realized volt isn’t ready for prime time. i’m in the process of rewriting all volt-based code back to standard livewire.

  • today, the shoutbox and live topics (which relied on volt) caused major performance issues. i rewrote those components quickly to fix the problem.

✅ fix: livewire volt code causing issues has been rewritten. more rewrites are ongoing.

bots 🤖

• bots are a growing issue. in the past, we’d see a few crawlers (google, amazon, etc.), but now there are thousands of "ai agents" scraping data daily.
• today, 500+ bots were crawling subeta simultaneously, loading every post, news update, etc. this caused major strain on the servers.

✅ fix: i’ve updated our configurations to block the most aggressive bots. unfortunately, this is an ongoing battle, and there's always a risk of accidentally blocking real users. 😞

time ⏳

all of these issues happened today. here’s what i worked on:

• rewrote parts of the infrastructure and deployment process.
• tracked down bot farms and adjusted server configurations to block them.
• debugged and rewrote volt-based livewire components.
• ensured kumos could deploy properly across servers again.

just a reminder: i’m the only programmer on subeta. while i’d love to focus on fun updates like avatar contests and quests, a lot of my time gets eaten up by invisible (and sometimes very visible) issues that keep the site running. i’m working hard to balance everything and make progress on the exciting stuff too. 💖

kumos vs legacy

• today’s issues were kumos-specific, but don’t let that fool you—it’s not the "problem child." legacy is actually the bigger resource hog, running on servers 8x the size of kumos because of massive memory leaks.
• the reason today’s issues were resolved so quickly is because of kumos’ updated infrastructure. back in the day, similar problems could’ve taken subeta.net down for days. kumos is a big improvement in many ways.

thanks for your patience as i work through these challenges. 🙏 i promise, the fun stuff is coming! 💖

very nervous a ping will break it all lol 😭

I have noticed kumos has been kinda slow lately, so hopefully this helps! thanks for the update

Thanks for the update, Keith!

Thank you for the update!

Did not break it, at least, so far. sorry you're having such a time

thanks for keeping us in the loop!

i was just reading about how claudebot (and others, but i've heard claude is particularly aggressive as of late and ignores all instructions & requests to stop scraping) are causing problems across the web like this 💀 sorry to hear it's hitting subeta, i know it's tough to manage. thank you for keeping us updated!

After ruthlessly pruning my "public" outfits to under 50, I could finally enter the contests I'd previously been unable to. So, I have determined that it does only 50 of the outfits and then stops - it says it's going on to do more, but it doesn't - and it keeps thinking there's more outfits to do, so it never shows you the ones it already has analyzed so you can pick one. Also it seems to go oldest-first which is the reverse of what it should be doing.

I made most of my previous contest entries "unlisted" because that allows them to still be in contests by linking...

It's incredibly frustrating, and these tools have grown in power over the last few months. Now they'll "browse the internet" for you, which means you can build apps that do things like "Log in to Subeta and navigate the forums" if you're trying to build a bot that can scrape 20 years worth of forum posts for your AI slop machine 😡

I'll just pretend I understood all that technical talk and will smile and nod. Thanks for the update

Thanks for the update!

thanks for the update, and of course, for all of your hard work!!

Thanks for the update

Thank you for all your hard work, Keith! The bots especially are so tiresome.

Thank you for the info. :)

Thanks for the update! All your hard work is GREATLY appreciated!

first time im actually able to read a site update on here 😐

Amazing work, I hope the bots stay gone!