Replies

Mar 27, 2020 5 years ago Official
SubetaTeam
User Avatar

From March 23rd 7:15 AM to March 24th 10:50 PM Subeta time, there was an error where every time an item was removed through the new wardrobe, it would be deleted from other users' wardrobes as well. We fixed this error as soon as we could, but many items have been lost. Our daily backup service had also stopped working, so we were unable to roll back the database by a day to restore the items.

We own up to this; this was an egregious, unacceptable error, and there is no excuse for not better monitoring our backups.

This error has now been fixed, and we are taking steps both to resolve this in the immediate future and to prevent this from ever happening again. Here's what you need to know.

Immediate Actions

We understand many items are irreplaceable, and we're going to do everything we can to restore all individual items. While our daily backup service had failed, we've booted up an older backup from about 4 months ago which will help a lot. We will be using this backup, various logs, and existing avatar data to further determine which items need to be restored.

We've set up a report form here that should streamline the process of reporting items you know you're missing. From now on, please go ahead and use that report form instead of the ticket system to make your reports.

You can now check to see what has been re-added to your wardrobe here.

We've also anticipated some questions which we've put in the FAQ spoiler below. If you have further questions, please leave them in this thread; we want to make sure your concerns are heard and taken care of.

Reports FAQ Do I need to use the report form in order to get my items back? No; our goal is to restore every item possible, regardless of if the user has reported it or not. We understand that most users don't remember every item in their wardrobes, and we don't want to punish anyone for that. We do ask you use report what you can, however; this information will help us immensely in restoring the items both for you and anyone else who was affected.

How long will I have to report things? The Report Form will be open indefinitely, so if you realize an item is still missing, say, a month from now, you can report it then.

I have already made a ticket about my items. Should I still use the report form? Yes, please do. This will help us consolidate the information.

I am missing a lot of items and it will take forever to enter them into the form one at a time. The Report Form will be open indefinitely, so you can take your time reporting what you've lost. If it is too much to list at once, you can start with the items you're missing most. We do encourage people to start by listing more expensive items and CWs so we can restore the most valuable items first.

There's an item that I'm only 90% sure if I had it. Should I still report it? Yes; if you're mostly certain, go ahead and report.

Finally, we are giving 1000 CSC to every user as additional compensation. Please click here to receive your CSC. This link will also be available indefinitely for all accounts potentially affected by the glitch.

Future Preventative Measures

While coding errors will sometimes slip through, the full scale and impact of this error could have been prevented. And while it is too late for this situation, we are taking actions to ensure this never happens again in the future.

  1. More reliable backups - We've improved our backup service and have added monitoring so that we will be alerted immediately if a problem with it occurs.

  2. Changing the way wardrobe data is deleted - We've changed the wardrobe to no longer delete item data immediately when you remove items; we'll now have recent delete data in our database itself as an extra layer of backup. This gives us more ability to respond to all wardrobe-related incidents moving forward, not just incidents like this one, and makes our system more robust in general.

  3. Further database reinforcements - Finally, we're investigating other changes we can make to our database infrastructure to further ensure resilient data and backups.

We want to make it clear that we understand how truly upsetting this is for everyone. We are so, so sorry. We are working hard to make this right, and are putting every effort in to restore your wardrobes and your faith in us.

April 24 Update:

We promised that we would do our best to figure out what items were missing and restore them using all the data/resources available to us. We've now wrapped up this process. In other words, we've now restored everything that was possible to automatically restore from available data.

It was never possible to restore items 100% accurately through this process, since we didn't have a working database backup - an unacceptable mistake that we've been open about since the start, and are still doing our utmost to own and correct.

In light of this, the missing items report form will remain open indefinitely so that, if you notice a missing item that we overlooked - even months after the fact - you'll still have a way to bring it to our attention. We deeply regret that any onus has to be put on you as part of this process, and we appreciate each report as giving us an opportunity to set things right.

We've released another report tool with a complete list of items we have confirmed were deleted from ALL wardrobes due to this glitch. This list automatically checks if items are in your wardrobe and has easy, one-click report buttons next to each item. Click here to check it out.

To end on a somewhat more positive note: Following this incident, we did an extensive post-mortem as to why this occurred, why our backups weren't reliable and how we can make sure this never slips through the cracks again - since, we agree, it was unacceptable that it did this time. In the last month, we've put a TON of work into creating more robust site infrastructure, which includes making sure we're on a working backup service with daily backups and constant monitoring. will post a site update with more details about this work in the near future.

Mar 27, 2020 5 years ago
Lantern
is actually pumpkin spice
User Avatar
Ito

Thank you for making a formal announcement about it. I hope you guys are able to sort out this mess.

Mar 27, 2020 5 years ago
Flowey
is the pumpkin king!
User Avatar

is there currently a way to find out which items were affected? i'm unsure if i was affected but terrified that i missed something.

Mar 27, 2020 5 years ago
Slaeden
is a fossil
User Avatar
Zaer

Thank you for letting us know so quickly & thoroughly and also for just, like, in general running one of my favourite small gamesites!! Subeta has been extremely comforting for me in the last month, and even a glitch like this doesn't take away from how awesome y'all are!!!! <3 <3 <3

| - Joined April 8th | [egg=slaeden] | [tp=slaeden] | | | |

Mar 27, 2020 5 years ago
Damon
is a demon
User Avatar
Evee

Thank you. But I'm still a little iffy on the wardrobe right now. I know you said you corrected the error but I still feel unsure. :(

Mar 27, 2020 5 years ago
Laurey
is a lovebug
User Avatar

Still pretty unbelievable that the backup system was down for apparently 4 months and nobody noticed.

I've figured out most of the items I'm missing, so I guess I'll go ahead and report those at least.

Mar 27, 2020 5 years ago
dragonrider88
has a dragon
User Avatar
Kwixis

I've been compiling a list of missing items. It's a slow process, but are you guys aware that there are Lumi tree items in the wardrobe too?? I've started a list of those too.

Do not meddle in the affairs of dragons for you are crunchy and taste good with ketchup!

Mar 27, 2020 5 years ago
Ferinsy
will put a spell on you
User Avatar
Nicki Minaj

Oh my CWs... I now regret I didn't do those wishlists "CWs owned" people used to do, I thought it was kinda pointless cries 4 months is a lot to remember

Mar 27, 2020 5 years ago
Searchname
is INCONCEIVABLE
User Avatar
Quixote

I know what CWs I am missing because I WL those (so I do not buy them again- have done that) but I have no idea what SP items I am missing. There's so many! Hopefully anything added in the past 4 months is not among the unaccounted for I guess

[font=garamond]I will absolutely take life seriously, the moment it stops being so absurd.[/font]

Mar 27, 2020 5 years ago
Delphi
is a sun worshipper
User Avatar
illusionary

I know some of the items I'm missing but I would like a list of known items that are missing on staff's part to check my wardrobe against.

♥️

Mar 27, 2020 5 years ago
One, Two
Victoria
is coming for you...
User Avatar
Colson

Ouch! Sorry yall are having to deal with this. I would not even know where to begin to try looking at my items and what may be missing.

Mar 27, 2020 5 years ago
Angelic
is a SUPER USER!!!
User Avatar

Thank you for your quick response in taking care of this situation and in letting us know about it.

Mar 27, 2020 5 years ago
Marianne
parties with the undead
User Avatar
Salima

I have no idea of my items - there are so many, including ones I paid for. I guess I have to trust I'll get them back? Not too comforting...



Flowers for Marianne

[TP=Marianne]
[egg=Marianne]

Mar 27, 2020 5 years ago
Straywillow
will always find their way
User Avatar
Delbchaem

Thank you SubetaTeam.. your thoughtfulness is one reason I have stayed all these years and this bug just brings it out again.

Mar 27, 2020 5 years ago
Opossun
is booty-ful
User Avatar
Yorick

Thank you so much for the quick response, but is there a way for me to know what item i'm missing? I don't keep track of their names and I usually change a lot of items...


Hey, I have an 【 art shop 】

Mar 27, 2020 5 years ago
Reno
is a tomb raider
User Avatar

Ooh, omg. I had thought I was losing my mind because I'd have things marked as had on subetalodge that I don't here. Well! Thanks for the headsup, and for fixing it. ;w; You guys work hard, things happen!

Mar 27, 2020 5 years ago
gitto
is a billionaire
User Avatar

Thanks for the update and the CSC, but I’m really not sure how we are supposed to know what items we’ve put into our wardrobes in the past 4 months for sure? We might have added items that we will never get back because we don’t know we added them!

Mar 27, 2020 5 years ago
Merlin
parties with the undead
User Avatar
Mullberry

oh my god, this is horrible. i don't keep track in any way of what i have. why the f should i? trying not to freak out but jfc. like many i've spent a LOT on cws. I don't feel like this should be on the users to figure out. this should be on you guys. you need to figure out what's missing exactly and replace everything for everyone. if this doesn't happen. giving ppl 1000 csc is gd LAUGHABLE. i can only imagine what people are going to do about getting back the hundreds of dollars you owe them. i'm just floored rn. i can't even rn i'm legit so angry. Adding that i don't have TIME to go searching around for what you guys deleted. you all messed it up, you all need to fix it.

Mar 27, 2020 5 years ago
Last call for
TOKYO
User Avatar

Maybe there should be a list of what people remember removing from their wardrobes too. I'm sure didn't remove anything, but have no idea if I lost something

Mar 27, 2020 5 years ago
nicolas
is a survivor
User Avatar
Wei Wuxian

Quote by Victoria
Ouch! Sorry yall are having to deal with this. I would not even know where to begin to try looking at my items and what may be missing.
Aye, I'm in the same boat. I've been taking mini-hiatuses from subeta in between holidays so I don't really have much of a memory of what was in my wardrobe beforehand, if any of my stuff got affected or not...

"Do not go gentle into that good night. Rage, rage against the dying of the light." { Tumblr || Flight Rising } [tot=nicolas]

Please log in to reply to this topic.