Replies

Mar 27, 2020 5 years ago Official
SubetaTeam
User Avatar

From March 23rd 7:15 AM to March 24th 10:50 PM Subeta time, there was an error where every time an item was removed through the new wardrobe, it would be deleted from other users' wardrobes as well. We fixed this error as soon as we could, but many items have been lost. Our daily backup service had also stopped working, so we were unable to roll back the database by a day to restore the items.

We own up to this; this was an egregious, unacceptable error, and there is no excuse for not better monitoring our backups.

This error has now been fixed, and we are taking steps both to resolve this in the immediate future and to prevent this from ever happening again. Here's what you need to know.

Immediate Actions

We understand many items are irreplaceable, and we're going to do everything we can to restore all individual items. While our daily backup service had failed, we've booted up an older backup from about 4 months ago which will help a lot. We will be using this backup, various logs, and existing avatar data to further determine which items need to be restored.

We've set up a report form here that should streamline the process of reporting items you know you're missing. From now on, please go ahead and use that report form instead of the ticket system to make your reports.

You can now check to see what has been re-added to your wardrobe here.

We've also anticipated some questions which we've put in the FAQ spoiler below. If you have further questions, please leave them in this thread; we want to make sure your concerns are heard and taken care of.

Reports FAQ Do I need to use the report form in order to get my items back? No; our goal is to restore every item possible, regardless of if the user has reported it or not. We understand that most users don't remember every item in their wardrobes, and we don't want to punish anyone for that. We do ask you use report what you can, however; this information will help us immensely in restoring the items both for you and anyone else who was affected.

How long will I have to report things? The Report Form will be open indefinitely, so if you realize an item is still missing, say, a month from now, you can report it then.

I have already made a ticket about my items. Should I still use the report form? Yes, please do. This will help us consolidate the information.

I am missing a lot of items and it will take forever to enter them into the form one at a time. The Report Form will be open indefinitely, so you can take your time reporting what you've lost. If it is too much to list at once, you can start with the items you're missing most. We do encourage people to start by listing more expensive items and CWs so we can restore the most valuable items first.

There's an item that I'm only 90% sure if I had it. Should I still report it? Yes; if you're mostly certain, go ahead and report.

Finally, we are giving 1000 CSC to every user as additional compensation. Please click here to receive your CSC. This link will also be available indefinitely for all accounts potentially affected by the glitch.

Future Preventative Measures

While coding errors will sometimes slip through, the full scale and impact of this error could have been prevented. And while it is too late for this situation, we are taking actions to ensure this never happens again in the future.

  1. More reliable backups - We've improved our backup service and have added monitoring so that we will be alerted immediately if a problem with it occurs.

  2. Changing the way wardrobe data is deleted - We've changed the wardrobe to no longer delete item data immediately when you remove items; we'll now have recent delete data in our database itself as an extra layer of backup. This gives us more ability to respond to all wardrobe-related incidents moving forward, not just incidents like this one, and makes our system more robust in general.

  3. Further database reinforcements - Finally, we're investigating other changes we can make to our database infrastructure to further ensure resilient data and backups.

We want to make it clear that we understand how truly upsetting this is for everyone. We are so, so sorry. We are working hard to make this right, and are putting every effort in to restore your wardrobes and your faith in us.

April 24 Update:

We promised that we would do our best to figure out what items were missing and restore them using all the data/resources available to us. We've now wrapped up this process. In other words, we've now restored everything that was possible to automatically restore from available data.

It was never possible to restore items 100% accurately through this process, since we didn't have a working database backup - an unacceptable mistake that we've been open about since the start, and are still doing our utmost to own and correct.

In light of this, the missing items report form will remain open indefinitely so that, if you notice a missing item that we overlooked - even months after the fact - you'll still have a way to bring it to our attention. We deeply regret that any onus has to be put on you as part of this process, and we appreciate each report as giving us an opportunity to set things right.

We've released another report tool with a complete list of items we have confirmed were deleted from ALL wardrobes due to this glitch. This list automatically checks if items are in your wardrobe and has easy, one-click report buttons next to each item. Click here to check it out.

To end on a somewhat more positive note: Following this incident, we did an extensive post-mortem as to why this occurred, why our backups weren't reliable and how we can make sure this never slips through the cracks again - since, we agree, it was unacceptable that it did this time. In the last month, we've put a TON of work into creating more robust site infrastructure, which includes making sure we're on a working backup service with daily backups and constant monitoring. will post a site update with more details about this work in the near future.

Mar 27, 2020 5 years ago Official
Bug
User Avatar
Segfault

For those who are worried they don't know which items they're missing to report, this is addressed in the FAQ:

Quote
Our goal is to restore every item possible, regardless of if the user has reported it or not. We understand that most users don&;t remember every item in their wardrobes, and we don&;t want to punish anyone for that. We do ask you use report what you can, however; this information will help us immensely in restoring the items both for you and anyone else who was affected.

In other words, we don't expect anyone to be reporting a complete list of their missing items. We intend to use the reports to help us restore items for everyone. They will help us understand which items were affected and to what scope.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 27, 2020 5 years ago Official
Bug
User Avatar
Segfault

Thank you for the concerns brought up thus far.

(Also: my posts like this will also be visible from Admin Post Highlights.)

Is there a list of items we know to have been affected, for users to check against? We agree that a list like this would be helpful for everyone. We can't provide one yet because we're still working on compiling it. Your reports will be a helpful piece of this puzzle.

I've checked with my friends, and it seems like some items were not deleted from everyone's wardrobes, only some people's. Thank you to those who have brought this up. This has been noted on our end, and we've edited the announcement to reflect this. We will restore lost items whether they were lost for everyone or only some people.

Are we going to lose all items from the last 4 months? No. As stated, our intention is to restore ALL items possible. We agree that it would be unacceptable to ask users to eat this kind of loss.

It should be staff's responsibility to clean up this mess. Don't put responsibility on the users to remember what items they lost. We hear you 100%. We created the user reports form to streamline the reporting process for those who want to make reports, and help us consolidate information from user reports. We don't intend to rely on user reports to restore lost items. We're also using the backup we have, various logs and other existing data to piece this together. You don't have to make any reports if you don't want to, and restoring your items will NOT depend on you making a report.

1000 CSC is not enough compensation for the items we lost. The CSC is not intended to replace lost items. We will restore everything we possibly can, starting with high-value items including CWs. The 1000 CSC gift is on top of that as an apology.

How long will it take to get our items back? We're looking to start restoring items as soon as possible. All of our other work is on hold until we sort this out. We know it would be super frustrating to have to wait weeks to get your items back, and we also want to avoid that.

A lot of mistakes were made by staff in handling this. Yesterday's post was too hidden, and the delay between that post and today's announcement only incited further confusion and anxiety. We own up to this. Yesterday's post was a mistake; It was hastily written, vague and not informative, and hard to find. Today's post is the proper announcement we should have given you yesterday. We understand it doesn't undo the mistake, and we apologize.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 27, 2020 5 years ago Official
Bug
User Avatar
Segfault

Quote by Foghawk
Any chance that users who kept offsite records of their wardrobes can send them in?</p>
<p>I&;ve written a userscript to export the list of items from the SubetaLodge wardrobe checklist, so I can provide that in whatever format (and it should be pretty plug-and-play, so other people could too).

Thank you for this excellent idea. We're discussing this behind-the-scenes and seeing what we could get set up. Sit tight for now. <3

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 27, 2020 5 years ago Official
Bug
User Avatar
Segfault

Are only wardrobe items affected? Yes, this glitch only affected the wardrobe. No items should be missing from inventories, treasure chests, armories, shops, vaults, etc.

People are seeing Luminaire ornaments in the new wardrobe. Is that related? This is an unrelated glitch, where the wardrobe is also showing ornament inventory. We intend to fix this when we're able, as well as other bug reports within the new wardrobe, but we're currently prioritizing the missing items issue at hand.

If we have compiled spreadsheets or lists elsewhere such as SubetaLodge, should we send them in? We thank those who have offered this information, and are discussing it behind-the-scenes. Please hold off on sending them in for now until we have a decision on this.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 28, 2020 5 years ago Official
Bug
User Avatar
Segfault

A quick clarification: Some of you have noticed that the new wardrobe is showing a higher item count than the old one. This is not related to the item deletion glitch; this is an unrelated glitch where the new wardrobe is erroneously including your Ornament Inventory in the count.

And an update: We've been working nonstop all day to boot up the backups, investigate the information we have, and figure out a plan for restoring items and notifying users that items have been restored. I will continue to give updates as things happen.

Finally, I'd like to clarify our stance on a prominent concern we're seeing expressed:

You do not need to use the Report Form to get your items back. Restoring the items correctly is our responsibility to worry about, not yours. We provided the Report Form to streamline the report process for those who wish to make reports, and to consolidate the information we gain from these reports to better help us utilize this information. While we appreciate any information users choose to volunteer, we aren't relying on it; it's just one piece of the puzzle. We're working very hard to figure out what items were affected and restore them to accounts using ALL the tools, logs, backups and data we have at our disposal - not just user reports.

If you know you are missing a few items and would like to report it, we sincerely thank you; we know we're not entitled to your time and effort, especially to fix a mistake that was our fault. If you don't know what you are missing, or don't feel like making a report, please don't feel pressured to figure it out for us or force yourself to make the reports. You are allowed to take a step back from the website and wait for site admins to sort this out. We are working hard to begin restoring items as soon as possible, and there is nothing you guys need to do while you wait. And again, we apologize for the wait and for the understandable stress, anxiety and frustration this issue has caused.

☕ If you haven't, please check out the Admin Posts Highlights page as well as the FAQ in the first post to see if your remaining concerns have already been answered by staff. Thank you so much.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 28, 2020 5 years ago Official
Bug
User Avatar
Segfault

Is there an easier way to view all wearables on site than viewing them within the wardobe? Yes; You can use the Item Directory to see all wearables. Here is a quick link to the wearables listing. There are multiple options for sorting and filtering these items from within that page.

Can I tell what items I'm missing by comparing the new and old wardrobe? No; both old and new wardrobe share the same items. If you don't remember what items you are missing, it's okay; you can take a step back and wait for staff to sort it out. We are doing our utmost to restore your items regardless of if you can remember them or report them.

Are only CWs affected? No; both CW items and regular site items have been affected by this glitch. However, CWs are being prioritized as we restore items to accounts, because these are often the most valuable, rare, and often sentimental items for many users.

For some of the items I lost to this glitch, I already went and bought new copies. What happens when staff tries to restore these items? Our scripts will simply skip over any items that are already in your wardrobe. However, we understand that some users may have already spent a lot of currency to replace more valuable items. If this is the case, we ask that you make a ticket explaining the situation and we will make sure you are reimbursed so that you're not punished for replacing it yourself.

Can you make a tl;dr of these updates? There are a lot of admin posts and it's too much to read. I understand where this is coming from; it is a lot of text, and walls of texts can be frustrating in themselves - especially when you're looking for just one specific thing. However, I'm afraid we don't have the capacity to re-answer questions or compose summaries of our responses thus far. While we want to make sure all concerns are promptly addressed, we're putting most of our time and effort into following up with actions, not just words. (And we will have something to show for this effort very soon!)

That said, I have noticed a lot of users are answering other users' questions in this thread and I want to say a HUGE thank-you to those users for stepping up and helping out. You guys didn't have to help and yet you did, and it's noticed and appreciated. <3

Why is the only one responding to this topic? Where is everyone else? To prevent duplicate responses and make sure everyone's time is used as efficiently as possible, it's a good idea to have only one staff member handle user communications for this incident. I happen to be in the best position to handle this right now.

Just because other staff aren't posting here doesn't mean they aren't working just as hard as I am! Our UAs are working hard to respond to tickets on this incident (and tickets in general). Our artists, content and CW staff are working hard to make sure other site operations continue smoothly in the meantime. has not been posting here, but he and I have been working jointly behind-the-scenes to analyze our data, formulate plans, construct and test necessary scripts and solutions, and get everything set up to begin restoring items to accounts ASAP. (This is an equal effort; everything we're doing is going through both of us to be extra safe.) And finally, all staff have been involved in discussions about staff decisions, including multiple facets of our reaction to this incident.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 30, 2020 5 years ago Official
Bug
User Avatar
Segfault

Are you seriously asking users to prove how they got an item? And refusing to give items back without proof? No!! This was a huge misunderstanding.

Before we realized the extent of the glitch, initial tickets were asked for more information such as how the user obtained the item. This is a standard question we ask to get a clearer picture of what happened. It was NOT meant to imply that the item wouldn't be reimbursed if the user didn't have proof.

No one should've been told that this proof was required, and no one should've been told that their proof was insufficient. (We actually aren't seeing evidence that this was explicitly said to anyone, but we're still looking through tickets and we'll make sure it's addressed if so.)

Since the creation of this announcement, UAs are not handling this through tickets. They have been instructed to redirect tickets to our official report form and forum topic for more information.

We are NOT asking for or requiring proof for anything related to this glitch. Again, you don't need to do anything right now but sit and wait for staff to continue sorting this out.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 31, 2020 5 years ago Official
Bug
User Avatar
Segfault

Hey everyone; I'm seeing some misinformation I'd like to clear up real quick.

Understandably, people are looking for ways to tell which items they've lost, or at least how many items they lost.

However, all of the methods we've seen come up so far (comparing old/new wardrobe counts, checking CS logs for blank images) are entirely unrelated to this glitch despite sounding very plausible. None of these methods will tell you anything about what you may have lost to this glitch.

It really sucks, but there is truly no way for users to check how many items or what items were lost.

You are also not required to know to get your items back. As staff, we have various logs, data, analytic tools available, and we are using everything at our disposal to sort this out.

We understand that this is still not a happy thing to hear, and that not having a way to check is pretty anxiety-inducing in itself. Nothing we say here invalidates your feelings on the reality of the situation. We own up to that, and are doing everything we can to make it right.

I just want to clear this up because it's misled some people to panic, thinking they've lost hundreds of items or dozens of CS purchases based on faulty information. I hope my post here can spare you some unnecessary panic; we know the reality of the situation is stressful enough as-is.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 31, 2020 5 years ago Official
Rah
is magical
User Avatar
Rah

To clear some things up:

We absolutely believe you guys when you're telling us that it's been deleted for you, even if it hasn't been for other people.

There WAS a miscommunication here amongst staff regarding this. Our UA thought that the item had to be deleted from everyone's wardrobe in order to be this glitch, which we have explained now isn't the case. This miscommunication has been now cleared up within staff.

That said, when we ask for information, it's to get a clearer picture of what happened - not because we think you're trying to scam us! But we need to figure out what happened in case there are other glitches we have to worry about, and if so we want to find out who else might have been affected.

That's what the UA here was trying to do - because she mistakenly thought it was an unrelated glitch, again due to a miscommunication that was not her fault. Regarding the CW bought from 's shop - our UA bought that to try and test if it would disappear for her. This item is currently still in the UAs wardrobe, and this test was only done in order to best figure out what happened so the UA could help other users. The UA now understands the mistake, but she was genuinely trying to act in the best interest of the users given what she believed at the time.

We sincerely apologize for the miscommunication behind this. We believe that we've cleared up the miscommunications within staff at this point and hopefully there will be less confusion from now on.


Rah image drawn by the dear !

Mar 31, 2020 5 years ago Official
Bug
User Avatar
Segfault

We've started restoring items to accounts. Some of you may have received sidebar alerts; however, we've been trying to alert in batches instead of alerting for every single item restored, so that we don't spam you. You can check this page to see if anything has been re-added to your account so far.

This restoration process has only just begun, so you may see more items show up over time. Please don't be alarmed if there aren't many items on that page for you yet.

We are basing this on cross-referencing several sets of data: our November backup, various logs, and user-volunteered information. That said, for some items we aren't able to verify with 100% certainty if it was lost. In these cases, we chose to err on the side of getting you your items back; that is, we are choosing to re-add the item into your wardrobe even if we were only 70% or 80% certain about if you had it.

You will NOT be punished for any false positives.

We understand there are certain items where a false positive could cause issues: for example, custom clothing items intentionally limited by the releaser. If we become aware of these cases, we may "undo" it by removing the erroneously restored item, but again no one will be punished for what they do with a false positive.

Since Subeta does not officially track CW "status" (private, public, etc), we unfortunately must rely on users to tell us if these false positives occur. Again, we understand you shouldn't have to deal with this because of OUR mistake. We really messed up here, and the onus shouldn't be on you to fix it, in any way. We're currently discussing how we'd best take these kinds of reports should it come up, and are open to hearing what the community would want us to do as well.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Mar 31, 2020 5 years ago Official
Bug
User Avatar
Segfault

Re: The issue of removing private/limited Custom Wearables. Thank you for your feedback. You've brought up some very valid points and concerns. Right now, we are also leaning towards not deleting false positives, even if we have clear evidence that it was a false positive. We are definitely not going to delete based on a CW's "private" status alone.

Our main concern is with certain personal or extremely sentimental CWs. We don't know of any of these that have been affected, but reports are still coming in, so we want to be prepared if this happens.

We do typically stay out of CW "status", but in this case we are open to hearing from the community what you'd like our involvement to look like, since this particular issue is a direct result of our error.

You should give users back the option to download backups of our own account data. That would've helped at a moment like this. We appreciate your suggestion, but the solution is not to make users responsible for their own backups. This is not the users' burden to bear; this is our responsibility and we were the ones who messed up.

We aren't excusing what happened, because, again: there was no excuse for being unaware that our backups had stopped working. This was unacceptable, and we are doing everything we can to fix the damage and make sure users don't have to pay the price for our mistake.

We have now made sure our backup service is working as intended AND being regularly monitored, and make sure staff get an immediate notification if it stops working. We put this into action on the 26th. So right now, our backups are back up and being monitored round the clock so we will know immediately if they fail again.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Apr 1, 2020 5 years ago Official
Spotlight Champion
Destiny
User Avatar
Dexter

I went ahead and added the link to the OP :)

Dexter by

Apr 2, 2020 5 years ago Official
Spotlight Champion
Destiny
User Avatar
Dexter

I was one of the staff members helping with item returns and I am pretty sure they were starting with the cheap stuff and they are not finished. It is a lot of boring stuff but it does take time. The CWs are different so I cannot say anything about that since I was not involved. Hold tight guys!

Dexter by

Apr 2, 2020 5 years ago Official
Bug
User Avatar
Segfault

If you want to tell us about specific items you're missing, please use the report form here instead of posting on this forum topic.

Otherwise, just sit tight <3 We have only just begun to restore items. We are doing our best to make sure it doesn't take longer than it needs to, but there's parts of the process that can't be safely rushed.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Apr 2, 2020 5 years ago Official
Bug
User Avatar
Segfault

Sorry for not being more clear! No, all it means is that we're not done restoring items to everyone. We just started and unfortunately it's going to take some time.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Apr 20, 2020 5 years ago Official
Spotlight Champion
Destiny
User Avatar
Dexter

Hey there! I just wanted to post and let you guys that this indeed is still being resolved. The site issues did halt the restoration but please be assured that the programmers are still working on this!

Dexter by

Apr 20, 2020 5 years ago Official
Rah
is magical
User Avatar
Rah

It is part of 's job to respond on the forums with messages and information that the admins/programmers and other staff want to communicate. She also communicates back to us when particular problems or issues are cropping up more frequently.

She is not being thrown to the wolves, she is providing a useful service and making sure that a lot of staff can get on with our jobs of getting the site running smoothly without having to keep stopping and reading pages of user feedback every day. When our programmers have a big/good update to give, they will give it.

As Destiny has said, items have not finished being returned. It's a drawn-out process, and we're trying to do it correctly and amongst everything else we have to do to keep our site functioning and flourishing.


Rah image drawn by the dear !

Apr 24, 2020 5 years ago Official
Bug
User Avatar
Segfault

Hello <3 I'm back with an update on this issue.

We promised that we would do our best to figure out what items were missing and restore them using all the data/resources available to us. We've now wrapped up this process. In other words, we've now restored everything that was possible to automatically restore from available data.

It was never possible to restore items 100% accurately through this process, since we didn't have a working database backup - an unacceptable mistake that we've been open about since the start, and are still doing our utmost to own and correct.

In light of this, the missing items report form will remain open indefinitely so that, if you notice a missing item that we overlooked - even months after the fact - you'll still have a way to bring it to our attention. We deeply regret that any onus has to be put on you as part of this process, and we appreciate each report as giving us an opportunity to set things right.

We've released another report tool with a complete list of items we have confirmed were deleted from ALL wardrobes due to this glitch. This list automatically checks if items are in your wardrobe and has easy, one-click report buttons next to each item. Click here to check it out.

To end on a somewhat more positive note: Following this incident, we did an extensive post-mortem as to why this occurred, why our backups weren't reliable and how we can make sure this never slips through the cracks again - since, we agree, it was unacceptable that it did this time. In the last month, we've put a TON of work into creating more robust site infrastructure, which includes making sure we're on a working backup service with daily backups and constant monitoring. will post a site update with more details about this work in the near future.

🐝 ☕ bug (he/him) | your friendly neighborhood code wrangler. stay in the loop! join and check out the latest admin post highlights

Apr 24, 2020 5 years ago Official
Spotlight Champion
Destiny
User Avatar
Dexter

I see a lot of people talking about items being replaced when they don't think they owned them. This is going to happen! This is posted on the confirmed items list:

"Go ahead and make reports even if you're only mostly sure, not 100% sure, if you had an item. We would like to reduce the burden on you, and we are choosing to err on the side of getting everyone their items back, and will NOT be punishing or correcting false positives."

Dexter by

Please log in to reply to this topic.