Noticing this as well, taking a look! Seems like our images got out of sync somehow.
💖 ✨ 🤗
This should be resolved, one of the servers got out of sync with the images recently.
💖 ✨ 🤗
hi everyone – i’m actively working on a fix right now and hope to have things resolved within the next hour or so! before that, i wanted to share a few thoughts for clarity:
I'm genuinely sorry for the inconvenience this caused, especially over the weekend. I know how frustrating it is when a feature you rely on isn’t working.
thank you for understanding, and i’ll keep you posted with updates soon!
💖 ✨ 🤗
This should be resolved now. If you are still experiencing this, please clear your cache and try again. That worked for me! And I am sorry if there was any miscommunication on my end. I tend to post on that other thread because that is where I get pings. I did not see this board before.
Dexter by ❤
here’s a quick technical explanation of what happened and what’s been done to prevent it in the future:
we have dedicated servers for each major part of the site (api, wardrobe/cw shops, kumos, and legacy subeta). these servers typically don’t restart unless we specifically tell them to. we’ve made great strides in stability over the past few years, and our servers usually have uptimes of a month or more, with planned restarts during major security updates.
however, since these servers are “in the cloud” (hosted by digital ocean), the provider can restart them at any time, often without notice. on saturday morning, our SAPI server—which handles building and storing avatar images—was reset by the provider. when it came back up, a crucial “storage link” didn’t reconnect automatically. that link is what gathers all the avatar image assets (base images, wearables, etc.) and creates a cached version on the server to speed up requests. normally, we’d catch this on a planned rotation, but this unexpected restart caused the issue to slip by.
to make matters worse, our caching service compounded the issue. it stored the failed images temporarily, so even after we applied a fix, some users still saw broken or missing images while the international cache refreshed.
we’ve now adjusted our monitoring to catch this kind of issue earlier and updated our processes to ensure the storage link reconnects automatically in the future. thank you for bearing with us while we worked through it!
💖 ✨ 🤗
😡 taking a look at this again today, sorry everyone!
I think this is a problem between the images (which exist!) and our cache provider. I've also started the work of moving these images to somewhere more stable that doesn't rely on the storage linking method we're using now.
💖 ✨ 🤗
just pushed out a fix that should get it working in the short term until that work to move the images.
pls ping me if it starts to break again as soon as you soon it ❤
💖 ✨ 🤗