HAPPY NEW YEA...
Filter
Show Official Posts OnlyReplies
Thanks for all the work and the updates.
RIP Mom, I love & miss you more than you know. Tell Dede hi.
Ocean Conservation Namibia on YouTube... they do good work! https://www.ocnamibia.org
Things seem to be working much more smoothly!
Thanks for all of your hard work getting Subeta back up and running :)
<p>So, what happened? A few things went wrong here, some of them that can be prevented, that will hopefully make DDoS attacks like this easier to handle in the future.</p>
<ol>
<li>Our memcache server was hit on it&;s own by millions of requests. When we query the database, we store results in memcache so that we don&;t have to keep using expensive hits to the database. It makes the site really fast. Unfortunately the security preferences allowed it to be hit on one specific (non-invasive) port which resulted in someone being able to bring it down without us really noticing.</li>
<li>When the site can&;t hit the memcache, it hits the database. When ALL of the requests go to the database, it gets overwhelmed and things have to wait. A database works on a locking/searching method, where if I make a change (say, I get 100sP) it locks the table until that process is updated. If I try to see how much sP I have during that update, it makes it wait until it&;s been updated. That is great and how it should work, but it means that if the database gets really slammed it really slows everything down.</li>
<li>When the database gets slowed down, connections hang and timeout, making the webservers sit waiting to have finished results from the database but never getting them, so the connections never close.</li>
</ol>
<p>Basically, it comes from a DDoS attack on the memcache server. When I resolved the permissions on it, the attackers moved to DDoSing the site itself, where cloudflare came in handy.</p>
<p>What can we do going forward? Better tools to notice when something like this is happening. Also, on specificially my end, I was in Israel on my birthright trip and had almost no ability to reach the computer to help - this was a situation that required a lot of debugging and didn&;t have all the tools she needed from me, whch should be resolved now :/</p>
<p>
Sorry this happened . But your reply is a little techy.... Were we actually attacked by someone or not? 😕 ❓
[tot=PiplupMagby34]
:) Thank you for the updates and explaining it so clearly. And thanks to you, and all those working hard to get everything fixed. Happy New Year!
Thanks for the updates!
Hmm... what idiot(s) really would want to attack a pet site? They should have better things to do or learn. >_<
Thanks for the time and dedication to fixing it. :D
I can't access the site on my phone since the DDoS check was put in place.
I realize why the site needs to check that I'm human and not a program attacking. Don't get me wrong, access only on my laptop is much better than no access to the site at all. But I really like to be able to get on here from my phone again.
Paramnesia
The unmatched power of the distortion of memory, arcane magic, and artisanal cheeses!
Are we still supposed to be getting the Cloudflare DDoS check interruption? I am getting it pretty frequently.
Are you mobile or on like a public network? I get it again on my phone I think when my ip changes.