Tech:Incidents/2015-05-ddos

Orain suffered 9 days of consistent issues and downtime for most users due to a UDP DDoS attack which caused DigitalOcean to null route our hosts.

(Times are UTC+0)

Timeline

Note: small(er) issues have been reported before.

Note: DigitalOcean's automated emails for some reason did not get sent during the first stages of the attack.

20 May
06:15 Nagios: nagios sends last alerts out
06:56 Southparkfan: discovers Orain is down and sends mail to stafforain.org notifying of downtime
07:06 Southparkfan: realizes that prod6 seems down, sends mail to Dusti's and addshore's personal email addresses
10:15 Addshore: mails back and says Orain is up. It is unknown whether addshore accessed Orain with or without IPv6.
10:25 Addshore: mails a picture of prod10 graphs, which includes a graph of the inbound and outbound private and public traffic. There is a public inbound traffic spike, with one hitting 800mb/s inbound traffic
17:54 DigitalOcean Incoming DDoS Detected -- prod5.orain.org
17:54 DigitalOcean Incoming DDoS Detected -- prod6.orain.org
17:54 DigitalOcean Incoming DDoS Detected -- prod8.orain.org
19:57 Southparkfan: FastLizard4 tells me Orain is accessible when using IPv6, but not when using IPv4.
19:58 Southparkfan: tries to SSH into prod10 by using either prod10.orain.org or its public IP, but none of them work. SSH'ing into prod10 by using prod8 as a proxy works though.
22:33 Southparkfan: proposes to FastLizard4 (since he was able to do things, and Southparkfan wasn't) to redirect *.orain.org to prod13-temp.orain.org
22:40 Southparkfan: above setup will break IPv6 support, Southparkfan proposes to revert the whole DNS repo to b83d1ca08fe6bc728427d061b040d0245078e031
22:45 FastLizard4: tries to push commits to the DNS repo, but gets stuck with permission errors.

21 May
06:46 Southparkfan: revert /config to b83d1ca08fe6bc728427d061b040d0245078e031
06:46 Southparkfan: grant operations full access to DNS repo (it already should have, but just add the group in 'Colloborators' too
07:01 Southparkfan: All The Tropes and TestWiki are both confirmed back online, Meta is still down
10:44 FastLizard4: confirms Orain is up
14:36 Southparkfan: confirms Orain is up
16:37 DigitalOcean Incoming DDoS Detected -- prod8.orain.org
16:37 DigitalOcean Incoming DDoS Detected -- prod9.orain.org
17:24 DigitalOcean Incoming DDoS Detected -- prod7.orain.org
17:24 DigitalOcean Incoming DDoS Detected -- prod11.orain.org
17:24 DigitalOcean Incoming DDoS Detected -- prod10.orain.org
17:25 DigitalOcean Incoming DDoS Detected -- prod12.orain.org