I've just turned off the additional cloudflare protection, and spun up an additional 4 servers (way more than we need) to hopefully mitigate the problem if it continues.

So, what happened? A few things went wrong here, some of them that can be prevented, that will hopefully make DDoS attacks like this easier to handle in the future.

1. Our memcache server was hit on it's own by millions of requests. When we query the database, we store results in memcache so that we don't have to keep using expensive hits to the database. It makes the site really fast. Unfortunately the security preferences allowed it to be hit on one specific (non-invasive) port which resulted in someone being able to bring it down without us really noticing.
2. When the site can't hit the memcache, it hits the database. When ALL of the requests go to the database, it gets overwhelmed and things have to wait. A database works on a locking/searching method, where if I make a change (say, I get 100sP) it locks the table until that process is updated. If I try to see how much sP I have during that update, it makes it wait until it's been updated. That is great and how it should work, but it means that if the database gets really slammed it really slows everything down.
3. When the database gets slowed down, connections hang and timeout, making the webservers sit waiting to have finished results from the database but never getting them, so the connections never close.

Basically, it comes from a DDoS attack on the memcache server. When I resolved the permissions on it, the attackers moved to DDoSing the site itself, where cloudflare came in handy.

What can we do going forward? Better tools to notice when something like this is happening. Also, on specificially my end, I was in Israel on my birthright trip and had almost no ability to reach the computer to help - this was a situation that required a lot of debugging and didn't have all the tools she needed from me, whch should be resolved now :/

Thanks for the updates ! :)

Happy new year Keith, thanks for the explanation! I'm beyond amazed anyone would attack a site like Subeta o__o

can we get DDoS as a weapon for the coliseum

Some people need way better hobbies I guess.

Thx for the update.

wow ..crazy man

thank you for the update keith and all off the staff for doing the best they could with every thing.

Thank you for the explanation! I was starting to wonder if it was an attack because of how suddenly it happened.

Thanks for the updates ...Happy New Year

Thank gawd you're back ! I hope you had a great time!!!! :-)

Thanks for the update! :)

What an inconvenient time for all this to be happening. Thank you so much for the update! I really appreciate knowing what's going on. Oh, and happy new year!!

happy new year! thanks for the update :)

Thank you for the update and all your hard work during this period!

Here's to a problem-free, happy 2016 ahead.

Thanks for the updates! It sounds like a nightmare, but I'm glad you've got it all figured out now!

Thanks for the update. I hope you were able to enjoy at least a part of your Birthright trip. Happy New Year to you and the staff. :)

Happy New year to you, Keith :)

Ugh, who makes it their mission to take down a purely fun site like Subeta? Don't they have better things to do? They need to go hug a puppy, for the love of all that is holy!

Thanks for the update. What a horrible occurrence! :o

Why would anyone attack a pet site? Seriously, robbing a bank and then getting arrested (because let's face it, it's gonna happen!) would have been more productive on their end, in my opinion. 0_o

Thank you so much for the update and congrats on going to Israel for you birthright trip, hope it was a wonderful experience for you.

Thank you for fixing it all! I hope you had a wonderful trip...I saw some camel-riding somewhere, looked amazing! ;D Happy New Year! ^-^ ❤️

~ Was this attack causing us to receive this message: