As most of you have seen, the web server has been down for 2 days, mainly because of DDOS.
I'll try to explain you what happened :
For months now, we are every day DDOSed. As we rent our dedicated servers at OVH (Souyoustart to be precise), they offer us an automatic DDOS detection, and a DDOS protection after that (i'll won't explain more, if you want more informations, just check their website).
The problem is the DDOS detection timing : it can takes a few minutes up to a few hours to detect it, depending on the DDOS attack size.
Just to be clear : a DDOS usually involve several thousands of zombie PC opening several hundred to thousands connections per second on a specified IP/port at the same time. So, it's not possible to detect who's doing that, and even not possible to block all those computers, as they are too numerous.
Usually, DDOS are limited during the day, from several hours, to nearly 20 hours. Most of the time, OVH does a great work, and limits those attacks. Website is a little bit slower, some of legitimate requests doesn't reach the server btw, but most of them does.
As said before, the problem is the timing : before detecting the DDOS, our dedicated server takes everything in its head. Most of the time, website is not accessible for a few minutes to a few hours (depending the DDOS detection timing), but everything stays up and running.
This thursday, our dedicated crashed completely before OVH's DDOS detection. Some of our VMs (won't explain what it is, just check a search engine
) died. As i'm working everyday (yes yes, i've a job !), i didn't have much time to look at this until last night.I won't explain everything, but i had to made a huge update of some stuff, and it breaked a little more the dedicated VMs, so that i couldn't even access them. After some work, i finally managed to copy them (about 300GB !) to another dedicated server, and reinstall completely the first dedicated server (web part). After that, i had to transfer VMs back on the fresh install (300GB again...), and up everything.
This could have been solve right now... but no...
Firewall VM was completely dead, so i had to configure one from scratch.
After that, our web server for return had DDOS attack again, and wasn't accessible. After a while searching what is the problem, i figured it out (yes yes !!!!) : infected Wordpress websites makes thousands of requests each seconds, and makes apache go crazy. I tried to put some security things to block access to those websites.
For the moment, it seems to be OK. If not, i've some other tools not too far.
Sorry to have written such a long post, but i think it's important for all the return family to know what happened.
To finish, i can confirm you that lags on app server is nothing to do with those DDOS. Our app dev are working hard on that (i think even harder than me against DDOS and web stuff), and i'm sure they'll find a solution soon.
