Urgent Update: Disruption of Images

April 7, 2024 6:33 pm

Hey everyone, I’m sure you’ve noticed images around the site starting to flicker out of existence, again. I was hoping to not need to write a post the last few times it’s been resolved, but here we are again and images aren’t loading.

As many of you know, all our images are hosted on the domain https://img.subeta.net, directed to a secure storage "bucket" on AWS. For over a decade, this setup has served us well, housing our item images, avatars, and more.

A few years back, we migrated our servers away from AWS, keeping only a few essential components there, including the S3 bucket. These components are linked to older parts of the site that require a significant overhaul to transfer elsewhere.

Regrettably, for the past few months, our AWS account has been facing frequent suspensions due to "suspicious activity" flagged by external sources, likely trolls. I've had to jump through hoops multiple times, resetting passwords, and rotating tokens just to regain access and restore our images and queues. Our minimal usage means our bill hovers under $100, disqualifying us from upgraded support.

After pleading with AWS support during the last suspension, I learned that false reports of illicit content in our buckets trigger these suspensions. Due to our modest bill, Amazon suspends first and investigates later, leaving us stuck in this frustrating loop. We're constantly at risk of suspension due to these malicious reports.

Currently, I'm awaiting AWS support to reinstate our account. Once resolved, I'll strategize moving our remaining AWS infrastructure elsewhere to prevent future disruptions. This situation is exasperating, and is mostly out of my control. Your patience and understanding are sincerely appreciated, and images should be loading again very soon.

April 9, 2024, 1:43 am by

The transparency is really appreciated - so many sites go into radio silence as if it would be better to stay quiet until they fix the problem, and end up causing user frustration. It’s always better to say something even if that something is just ‘we know there’s a problem and we’re working on it’ - that small gesture goes a long way! <3

April 8, 2024, 10:43 pm by

As always the transparency is appreciated. If I knew something about programming I'd try to be helpful. But you'll always have my support. This is such a great space. You're always trying to better it..

April 8, 2024, 10:30 pm by

Thanks for the explanation Keith. I had noticed 90% of my item images breaking a few times and wondered what was up, lol. It didn't really effect my use of the site so I always just navigated based on text and ignored it until it was fixed. It sounds incredibly frustrating on your side to repeatedly deal with this nonsense from Amazon. You have my sympathies!

April 8, 2024, 3:07 pm by

"I was hoping to not need to write a post the last few times it’s been resolved"

I just wanna let you know, in my personal opinion of someone who has enjoyed this site for a good number of years, I think it's always okay to make some kind of communication and talk about when things are on the fritz.

April 8, 2024, 2:34 pm by

Thanks for letting us know. Is there anything that we, as users, can do to help? :D

April 8, 2024, 2:25 pm by

I haven't noticed that many images missing & all the dice are there when I play bluff. Hope everything gets worked out OK.; Thanks.

April 8, 2024, 12:33 pm by

typical bezos L, you hate to see it

April 8, 2024, 12:14 pm by

Thank you for telling us what's been going on. Images being broken can be incredibly annoying and I think it helps us remember that it's even more annoying being on the backend and trying to deal with mega-corporations who don't give a shit about their smaller customers because they have more money than any company should ever have.

April 8, 2024, 9:30 am by

Thank you for the transparency! Seeing the images go down so often was troubling, but having a reason behind it and knowing that not only are you doing all you can, but that you can’t do anything to prevent it, makes it easier to understand. It honestly makes me feel much better about supporting the site when we’re actually being given the reason and it’s not just being ignored like some things were in the past.

April 8, 2024, 8:38 am by

I'm so sorry for this. I am a content creator on a different platform and I swear people just poke around and report things as explicit or inappropriate when they aren't. I've had my account suspended for "substance abuse", and other creators have been flagged for random things and it's very old and frustrating. I can't imagine how much more so in this situation!

I hope that you'll be able to find a good solution that won't give you too many headaches!

April 8, 2024, 7:52 am by

Thank you for letting us know. Sorry you have to deal with this hassle.

April 8, 2024, 7:37 am by

That sounds so frustrating, I'm so sorry you guys have to deal with this! 😩😫

April 8, 2024, 5:27 am by

Thanks very much for the update and the work you are doing to keep things going.Cheers!! :)

April 8, 2024, 5:00 am by

that sounds so frustrating :( we really appreciate you keeping us informed about things like this! <3

April 8, 2024, 4:31 am by

That sounds very frustrating - for you! Me, I can just sit here and wait for it to work again sometime.

April 8, 2024, 3:11 am by

Thank you for the update. It's really sad that some people are always trying to hurt others...

April 8, 2024, 2:53 am by

Thank you very much to Keith and everyone else left on that skeleton crew of staff who's trying to resolve the issues. I'm a huge fan of Subeta, am here every night for several hours, and this site keeps me motivated to stay in the world and have a little bit of a social life. Thanks again everyone who works so hard to keep us going here.

April 8, 2024, 1:35 am by

People are such assholes! How did things get so bad in the world that people have started to amuse themselves by hurting others? We are so doomed as a species...luckily there's an asteroid on the way in 2036 that may or may not hit us, so fingers crossed we find a way off the planet for the good people by then

April 7, 2024, 11:25 pm by

Thanks for the update and your continued work on the issue! <3

April 7, 2024, 11:01 pm by

Keith, your efforts and the efforts of your staff are beyond appreciated. Thank you so much for keeping Subeta around, it really makes my days so much better to be a part of this community regardless of whether the images are loading or not! <3

April 7, 2024, 10:53 pm by

Oof! What a headache. Some people got nothing better to do than be the absolute worst -.- hope y’all can get to a solution quickly.

April 7, 2024, 10:41 pm by

i can't believe it's someone actively making an effort to report the site...like that's really something else & that's coming from me, the prince of tedium sorry it's been so crap to deal with, that really sucks

April 7, 2024, 10:10 pm by

You can report any public AWS asset (publicly accessible EC2 instances, S3 buckets, etc) as holding child pornography or illegal / stolen / copyrighted material, and they'll typically (Unless you're on a support plan) suspend as specific as they can get. If it's the third or fourth time it's happened (and this is the... 20th?) they'll ban the account until confirmation that it's not doing any of the above.

We'll just move off S3 to something s3-compatible (which there are a ton of now!) and be done with it soon, hopefully!

April 7, 2024, 10:02 pm by

as someone who works with AWS (using their system, yes including buckets, ec2s, migrations, all that), I'm so deeply curious as to how someone would report an s3 bucket for false info and why AWS wouldn't flag that 5 reports are probably not in good faith (but they have cut a great deal of staff, sooooOOooooOo)

If you want a temporary contract for someone to look and help, I'm here.

April 7, 2024, 9:05 pm by

Thank you for the update, Keith. The transparency is greatly appreciated. I hope that a more stable solution can be found that leads to less frustration all around.

April 7, 2024, 9:05 pm by

Thanks for the update. I was wondering why it was happening again.

April 7, 2024, 8:57 pm by

Thank you for the update! I'm sorry you have to deal with jerks :(

April 7, 2024, 8:53 pm by

I hate Amazon. Good luck!

April 7, 2024, 8:39 pm by

Thanks, Keith and I'm really sorry you're having to deal with all of that.

April 7, 2024, 8:32 pm by

Thanks for the update and explanation. Hope everything can get worked out.

April 7, 2024, 8:23 pm by

I promise we've had the crowd funding idea kicking around for awhile, I'm coming back to Subeta full time in a few days (at least for a few months 🎉) and can think about it more seriously. We've got a skeleton crew of us, so even the idea of putting together something like that would require a level of thinking about the future we haven't had space to do in quite some time~!

April 7, 2024, 8:23 pm by

I'm a total ignorant on this matters but I do appreciate on giving us an update. Thank you so much for the effort <3

April 7, 2024, 8:21 pm by

+1 to the fundraiser idea here too! I would especially love donating specifically to smaller goals like $xxx until the next plot. $xxx to fix this feature. Then people can donate to something they're really passionate about.

April 7, 2024, 8:15 pm by

Thanks for the update~

April 7, 2024, 8:04 pm by

Thank you for the transparency of what is going and and why . It is very much apricated in understanding the workings of Subeta behind the scenes. It is frustrating to hear that paying lower brings less respect by the company. I hope you are able to get Subeta off that server sooner than later and stop paying them for their lackluster service.

April 7, 2024, 8:02 pm by

Thanks for the update, and oh I feel for you. Within the last year I have had a terrible time with Amazon due to their security system being decidedly too sensitive. It is as though it unlearned everything that taught it how to recognize a human from a computer, and to tell normal human migration from foreign fraudulent interference. The result is this mess. At least in my case I am not under attack directly. I think honestly the attack is being directed at Amazon, and we are the ones who suffer because its our files and accounts that get flagged. May your efforts to get away from them prove to be a good thing.

April 7, 2024, 7:56 pm by

huh, i was wondering what was going on. hope it gets resolved smoothly!

April 7, 2024, 7:54 pm by

Oh I would also like to +1 the fundraiser idea. LOTS of players have wanted to do this and help Subeta out. Like sure, we can buy CSC, but there's a fairly noticeable desire in the playerbase to boost Subeta without relying on purchasing credits. You guys could do a Kickstarter-esque deal where there's prize tiers — perhaps CW and pet overlay vouchers, super rare but still available-to-all items, maybe even really high goals where backers can decide the next species/color to release or be revamped (if a revamp is deemed necessary), or a new set of wearables (one item, multiple palettes) for site-wide use, etc. Actually,separate and combo packages for HA people and pet people ain't a bad idea. Maybe a battler package too since I know that's some players' vibe! And then ofc the tracking of donation goals, like (total amount) means you guys can do (needed thing). Hiring more artists, more coders, finishing the new site, etc.

Just an idea!

April 7, 2024, 7:50 pm by

Really appreciate the update, even though you didn't want to share all that. For me, it's more reason to keep contributing to the site monetarily, to see that a better service can be acquired and paid for without taking money from the very few staff that remains.

End users need to stop focusing on the "whodunit" and focus that energy into things that would support the site and it's new transition of services.

April 7, 2024, 7:49 pm by

I have noticed that this tends to happen on a monthly basis now- around the same time. It's tragic that somebody or some people have that much time on their hands a month to do such things. I'm not even mad necessarily, more pity them for wasting their breath and precious time on earth like that when they could be doing something far more productive and life changing. ಠ⁠︵⁠ಠ Thanks for the update, Keith.

April 7, 2024, 7:45 pm by

Thank you for the update! Take the time you need, we aren't going anywhere!

April 7, 2024, 7:42 pm by

How frustrating! Thanks for the communication. Hopefully this resolves with minimal stress for you.

April 7, 2024, 7:41 pm by

who even has the time and energy to false report a small pet site in 2024 for shits and giggles like, problems for us aside, what a needless waste for them? it's exasperating but it's not like... y'know... actually clever and worthwhile trolling. someone is being a total clown for free instead of getting a job as a professional one (professional clowns get paid pretty good, actually).

April 7, 2024, 7:37 pm by

Thank you for the update Keith, and also many thanks to you all for working so hard to try to resolve these issues!!

April 7, 2024, 7:33 pm by

What a bunch of assholes. Do people really have nothing better to do? Apparently no, they do not. I like the idea of a fundraiser, if that looks like it would help us get away from AWS sooner/easier.

And I like the idea had. Only make it a Lego a dog’s chewed on. Nice extra sharp bits.

April 7, 2024, 7:33 pm by

It's about time Amazon and others remember that we are customers. They are in business because of us

April 7, 2024, 7:31 pm by

That is so nice! I've got a lot of experience from my other work in AWS, so I'm fairly sure I've followed best practices (IAM users with rotating and expiring tokens, root account with no API access and 2fa enabled) and it's just going to be a case of getting suspended every time before I can send in a ticket :( :(

April 7, 2024, 7:23 pm by