Tech:Incidents/2014-06-12

From Orain Meta
Jump to navigation Jump to search

A missed payment resulted in Prod4 being temporarily suspended.

Timeline

  • 05:26 BST - Pingdom reports orain down
  • 06:06 BST - RamNode has suspended prod4 again reports JL via email
  • 07:46 BST - Addshore facepalms via email
  • 20:15 BST - Paid and all back up.

In Hindsight

  • "As Labster's pointed out several times, we need redundancy not Single Point of Failures though I'll leave that for you to deal with :)"

Action taken

  • John has been granted access to the main RamNode account and methods to gain access to the PayPal account when necessary.