Tech:Incidents/2015-01-27

From Orain Meta
Jump to navigation Jump to search

Tuesday, January 27th, 2015, Orain was down for about 15 to 20 minutes (varies slightly due to the fact it did not affect the farm globally at the same pace)

Description

Around 14:51 I (Arcane) received permission from Dusti in the orain-staff channel to attempt installing the Validator extension manually as a global extension to resolve issues with installing it manually, as it normally requires Composer (this was an attempt to see if we could sidestep the hassle of needing to integrate Composer with ansible). I then added it to the global extensions configuration file and commented out changes to the local extensions file, where I experienced some limited success on the extload wiki. Unfortunatrly, this result in downtime globally, as reported by Anthonyk747 at 15:09. I immediately reverted the changes to fix matters, as was promised to Dusti and added to the commits, and the downtime was cleared up around 15:29.

Timeline

  • 14:51 - Arcane adds Validator to GlobalExtensions.php
  • 15:09 - Anthonyk747 reports downtime on his wikis, Arcane confirms this is global and reverts the changes, doing an ansible update to resolve the issue.
  • 15:29 - All Orain wikis confirmed up again.

Lessons

  • Composer integration is required to install extensions requiring it, and needs to be smoothly integrated with our current ansible configuration, this cannot be sidestepped despite the fact they appear to work initially when installed manually.

Meta

  • Staff on hand in downtime: Dusti, GethN7, Southparkfan, Tanner, FastLizard
  • Report published by: Arcane (alias GethN7)
  • Timestamp: 15:49, 27 January 2015 (GMT)
Retrieved from "https://meta.orain.org/wiki/Tech:Incidents/2015-01-27?oldid=13747"