Replies

Oct 27, 2024 1 year ago Official
Keith
is sweet
User Avatar
Eradication

Noticing this as well, taking a look! Seems like our images got out of sync somehow.

💖 ✨ 🤗

Oct 27, 2024 1 year ago Official
Keith
is sweet
User Avatar
Eradication

This should be resolved, one of the servers got out of sync with the images recently.

💖 ✨ 🤗

Dec 9, 2024 1 year ago Official
Keith
is sweet
User Avatar
Eradication

hi everyone – i’m actively working on a fix right now and hope to have things resolved within the next hour or so! before that, i wanted to share a few thoughts for clarity:

  1. this issue started over the weekend. while we aim to prioritize critical, game-breaking bugs during that time, we try to keep our work largely within normal working hours. this balance helps us stay effective and avoid burnout, so it’s not likely we’ll start operating a 24/7 on-call system for issues like this.
  2. this specific issue couldn’t have been solved by more people stepping in, even volunteers. i appreciate the sentiment every time it comes up, but if we had a volunteer team they'd have been powerless to do anything, but what Destiny did. it required access to critical engineering functionality that we wouldn’t provide to non-staff members for security reasons.

I'm genuinely sorry for the inconvenience this caused, especially over the weekend. I know how frustrating it is when a feature you rely on isn’t working.

thank you for understanding, and i’ll keep you posted with updates soon!

💖 ✨ 🤗

Dec 9, 2024 1 year ago Official
Spotlight Champion
Destiny
User Avatar
Dexter

This should be resolved now. If you are still experiencing this, please clear your cache and try again. That worked for me! And I am sorry if there was any miscommunication on my end. I tend to post on that other thread because that is where I get pings. I did not see this board before.

Dexter by

Dec 9, 2024 1 year ago Official
Keith
is sweet
User Avatar
Eradication

here’s a quick technical explanation of what happened and what’s been done to prevent it in the future:

we have dedicated servers for each major part of the site (api, wardrobe/cw shops, kumos, and legacy subeta). these servers typically don’t restart unless we specifically tell them to. we’ve made great strides in stability over the past few years, and our servers usually have uptimes of a month or more, with planned restarts during major security updates.

however, since these servers are “in the cloud” (hosted by digital ocean), the provider can restart them at any time, often without notice. on saturday morning, our SAPI server—which handles building and storing avatar images—was reset by the provider. when it came back up, a crucial “storage link” didn’t reconnect automatically. that link is what gathers all the avatar image assets (base images, wearables, etc.) and creates a cached version on the server to speed up requests. normally, we’d catch this on a planned rotation, but this unexpected restart caused the issue to slip by.

to make matters worse, our caching service compounded the issue. it stored the failed images temporarily, so even after we applied a fix, some users still saw broken or missing images while the international cache refreshed.

we’ve now adjusted our monitoring to catch this kind of issue earlier and updated our processes to ensure the storage link reconnects automatically in the future. thank you for bearing with us while we worked through it!

💖 ✨ 🤗

Dec 12, 2024 1 year ago Official
Keith
is sweet
User Avatar
Eradication

😡 taking a look at this again today, sorry everyone!

I think this is a problem between the images (which exist!) and our cache provider. I've also started the work of moving these images to somewhere more stable that doesn't rely on the storage linking method we're using now.

💖 ✨ 🤗

Dec 12, 2024 1 year ago Official
Keith
is sweet
User Avatar
Eradication

just pushed out a fix that should get it working in the short term until that work to move the images.

pls ping me if it starts to break again as soon as you soon it ❤

💖 ✨ 🤗

Jan 16, 2025 1 year ago Official
Keith
is sweet
User Avatar
Eradication

I think this shoud be fixed. Very annoying that it keeps happening 😡

💖 ✨ 🤗

Jan 18, 2025 1 year ago Official
Keith
is sweet
User Avatar
Eradication

If you give the page a hard refresh, do they show up? I feel like you might have cached the 'bad' results in your browser :(

💖 ✨ 🤗

Please log in to reply to this topic.