From Orain Meta
Jump to navigation Jump to search

At Friday January 23th, 2015, Orain experienced four hours of downtime because users couldn't resolve DNS records.


Around 18:55 (UTC +0) I (Southparkfan) experienced issues with accessing Orain, and I asked some other sysadmins and stewards on IRC if they had problems too, but they didn't. Almost an hour later some of them experienced issues too now, and so far I can be sure it wasn't just me. Around 21:50, after a lot of investigation which didn't had much sense, in cooperation with John I've discovered that there were DNS loops on Orain. While the IPv4 IP of prod6 ( nameserver) is, requests were redirected to, which is an IP not assigned to any DNS or Orain server. John stated that the nameservers in Namecheap (the registrar of might been wrong. I've asked Dusti to check the Orain nameservers, and at my request he set them to and back (again). Around 10 minutes after Dusti changed the nameservers, the farm became accessible again.


  • 18:35 Pingdom: Orain is reported down
  • 18:55 Southparkfan: I discovered that Orain seems down to me. Started to investigate why
  • 19:35 Southparkfan: other users are reporting issues too now
  • 21:53 Southparkfan: ask Dusti to set Orain's nameservers to and
  • 21:59 Dusti: nameservers are set to and
  • ~22:10 Southparkfan: Orain works again


  • A plan should be made so we are prepared for these issues. Southparkfan takes partial responsibility for looking at unreasonable causes.
  • While John left the team three days ago, that does not mean he can't help Orain out anymore. Use other people experienced with our infrastructure when possible.


  • Staff on hand in downtime: Dusti, GethN7, Southparkfan, Tanner
  • Report published by: Southparkfan
  • Timestamp: 22:45, 23 January 2015 (GMT)