From March 23rd 7:15 AM to March 24th 10:50 PM Subeta time, there was an error where every time an item was removed through the new wardrobe, it would be deleted from other users' wardrobes as well. We fixed this error as soon as we could, but many items have been lost. Our daily backup service had also stopped working, so we were unable to roll back the database by a day to restore the items.
We own up to this; this was an egregious, unacceptable error, and there is no excuse for not better monitoring our backups.
This error has now been fixed, and we are taking steps both to resolve this in the immediate future and to prevent this from ever happening again. Here's what you need to know.
We understand many items are irreplaceable, and we're going to do everything we can to restore all individual items. While our daily backup service had failed, we've booted up an older backup from about 4 months ago which will help a lot. We will be using this backup, various logs, and existing avatar data to further determine which items need to be restored.
We've set up a report form here that should streamline the process of reporting items you know you're missing. From now on, please go ahead and use that report form instead of the ticket system to make your reports.
You can now check to see what has been re-added to your wardrobe here.
We've also anticipated some questions which we've put in the FAQ spoiler below. If you have further questions, please leave them in this thread; we want to make sure your concerns are heard and taken care of.
Reports FAQ
Do I need to use the report form in order to get my items back?
No; our goal is to restore every item possible, regardless of if the user has reported it or not. We understand that most users don't remember every item in their wardrobes, and we don't want to punish anyone for that. We do ask you use report what you can, however; this information will help us immensely in restoring the items both for you and anyone else who was affected.
How long will I have to report things? The Report Form will be open indefinitely, so if you realize an item is still missing, say, a month from now, you can report it then.
I have already made a ticket about my items. Should I still use the report form? Yes, please do. This will help us consolidate the information.
I am missing a lot of items and it will take forever to enter them into the form one at a time. The Report Form will be open indefinitely, so you can take your time reporting what you've lost. If it is too much to list at once, you can start with the items you're missing most. We do encourage people to start by listing more expensive items and CWs so we can restore the most valuable items first.
There's an item that I'm only 90% sure if I had it. Should I still report it? Yes; if you're mostly certain, go ahead and report.
Finally, we are giving 1000 CSC to every user as additional compensation. Please click here to receive your CSC. This link will also be available indefinitely for all accounts potentially affected by the glitch.
While coding errors will sometimes slip through, the full scale and impact of this error could have been prevented. And while it is too late for this situation, we are taking actions to ensure this never happens again in the future.
More reliable backups - We've improved our backup service and have added monitoring so that we will be alerted immediately if a problem with it occurs.
Changing the way wardrobe data is deleted - We've changed the wardrobe to no longer delete item data immediately when you remove items; we'll now have recent delete data in our database itself as an extra layer of backup. This gives us more ability to respond to all wardrobe-related incidents moving forward, not just incidents like this one, and makes our system more robust in general.
Further database reinforcements - Finally, we're investigating other changes we can make to our database infrastructure to further ensure resilient data and backups.
We want to make it clear that we understand how truly upsetting this is for everyone. We are so, so sorry. We are working hard to make this right, and are putting every effort in to restore your wardrobes and your faith in us.
We promised that we would do our best to figure out what items were missing and restore them using all the data/resources available to us. We've now wrapped up this process. In other words, we've now restored everything that was possible to automatically restore from available data.
It was never possible to restore items 100% accurately through this process, since we didn't have a working database backup - an unacceptable mistake that we've been open about since the start, and are still doing our utmost to own and correct.
In light of this, the missing items report form will remain open indefinitely so that, if you notice a missing item that we overlooked - even months after the fact - you'll still have a way to bring it to our attention. We deeply regret that any onus has to be put on you as part of this process, and we appreciate each report as giving us an opportunity to set things right.
We've released another report tool with a complete list of items we have confirmed were deleted from ALL wardrobes due to this glitch. This list automatically checks if items are in your wardrobe and has easy, one-click report buttons next to each item. Click here to check it out.
To end on a somewhat more positive note: Following this incident, we did an extensive post-mortem as to why this occurred, why our backups weren't reliable and how we can make sure this never slips through the cracks again - since, we agree, it was unacceptable that it did this time. In the last month, we've put a TON of work into creating more robust site infrastructure, which includes making sure we're on a working backup service with daily backups and constant monitoring. will post a site update with more details about this work in the near future.
Thank you for making a formal announcement about it. I hope you guys are able to sort out this mess.
is there currently a way to find out which items were affected? i'm unsure if i was affected but terrified that i missed something.
Thank you for letting us know so quickly & thoroughly and also for just, like, in general running one of my favourite small gamesites!! Subeta has been extremely comforting for me in the last month, and even a glitch like this doesn't take away from how awesome y'all are!!!! <3 <3 <3
| - Joined April 8th | [egg=slaeden] | [tp=slaeden] | | | |
Thank you. But I'm still a little iffy on the wardrobe right now. I know you said you corrected the error but I still feel unsure. :(
Still pretty unbelievable that the backup system was down for apparently 4 months and nobody noticed.
I've figured out most of the items I'm missing, so I guess I'll go ahead and report those at least.
I've been compiling a list of missing items. It's a slow process, but are you guys aware that there are Lumi tree items in the wardrobe too?? I've started a list of those too.
Oh my CWs... I now regret I didn't do those wishlists "CWs owned" people used to do, I thought it was kinda pointless cries 4 months is a lot to remember
I know what CWs I am missing because I WL those (so I do not buy them again- have done that) but I have no idea what SP items I am missing. There's so many! Hopefully anything added in the past 4 months is not among the unaccounted for I guess
I know some of the items I'm missing but I would like a list of known items that are missing on staff's part to check my wardrobe against.
Ouch! Sorry yall are having to deal with this. I would not even know where to begin to try looking at my items and what may be missing.
Thank you for your quick response in taking care of this situation and in letting us know about it.
I have no idea of my items - there are so many, including ones I paid for. I guess I have to trust I'll get them back? Not too comforting...
Flowers for Marianne
[TP=Marianne]
[egg=Marianne]
Thank you SubetaTeam.. your thoughtfulness is one reason I have stayed all these years and this bug just brings it out again.
Thank you so much for the quick response, but is there a way for me to know what item i'm missing? I don't keep track of their names and I usually change a lot of items...
Ooh, omg. I had thought I was losing my mind because I'd have things marked as had on subetalodge that I don't here. Well! Thanks for the headsup, and for fixing it. ;w; You guys work hard, things happen!
Thanks for the update and the CSC, but I’m really not sure how we are supposed to know what items we’ve put into our wardrobes in the past 4 months for sure? We might have added items that we will never get back because we don’t know we added them!
oh my god, this is horrible. i don't keep track in any way of what i have. why the f should i? trying not to freak out but jfc. like many i've spent a LOT on cws. I don't feel like this should be on the users to figure out. this should be on you guys. you need to figure out what's missing exactly and replace everything for everyone. if this doesn't happen. giving ppl 1000 csc is gd LAUGHABLE. i can only imagine what people are going to do about getting back the hundreds of dollars you owe them. i'm just floored rn. i can't even rn i'm legit so angry. Adding that i don't have TIME to go searching around for what you guys deleted. you all messed it up, you all need to fix it.
Maybe there should be a list of what people remember removing from their wardrobes too. I'm sure didn't remove anything, but have no idea if I lost something
{ Tumblr || Flight Rising }
[tot=nicolas]