Tech:Server admin log: Difference between revisions
+addshore fixed problems yesterday with disk space
Southparkfan (talk | contribs) No edit summary |
Reception123 (talk | contribs) (+addshore fixed problems yesterday with disk space) |
||
| (39 intermediate revisions by 3 users not shown) | |||
Line 1:
== July 19 ==
* '''Addshore''' Fixed problems with Disk space --[[User:Reception123|Reception123]] ([[User talk:Reception123|talk]]) 06:36, 20 July 2015 (BST)
== July 4 ==
* I fixed everything (see git history) .... '''[[User:Addshore|<span style="color:black">·addshore·</span>]]''' <sup>[[User_talk:Addshore|<span style="color:black;">talk to me!</span>]]</sup> 09:28, 4 July 2015 (BST)
== June 30 ==
* ~09:10 Southparkfan: pooled prod9 back in prod with below changes applied. Ansible on both servers disabled. DO NOT run ansible on those servers unless you are 100% sure it won't cause issues.
* 08:54 Southparkfan: disable CSS, OnlineStatus and EmbedVideo on All The Tropes Wiki. Meta (why Meta too?) and All The Tropes are now back online and running without throwing MWExceptions.
== June 29 ==
* 14:48 Southparkfan: shutdown & destroy prod11
* 12:21 NDKilla: Not experiencing issues on any wiki's that reported issues. extloadtest still shows frequent errors
* 11:20 NDKilla: Rebuild LC on extloadwiki per SPF
* 10:48 NDKilla: Ran all jobs on metawiki and allthetropeswiki
* 10:38 NDKilla: Investigating DB (and hoping I didn't cause them)
* 02:02 GethN7 notifies #orain of a lot of DB issues on allthetropeswiki
== June 28 ==
* Late afternoon: Manually ran "sudo /root/ans-all --skip-tags=slow" on prod 8.9, and 11
== June 16 ==
* 17:16 Southparkfan: "sudo usermod -u 2020 www-scripts" on prod9 and prod11
== June 13 ==
* 11:46 Southparkfan: destroyed prod8 for testing
== May 14 ==
* 20:49 Southparkfan: DROP DATABASE spamwiki; on prod12 - massive disk space free up :D
* 20:49 Southparkfan: ran php5 /srv/mediawiki/w/maintenance/Orain/removeDeletedWikis.php --wiki loginwiki on prod9
== April 28 ==
* 13:09 Southparkfan: pooled prod11 back
* 13:02 Southparkfan: reboot prod11
* 12:51 Southparkfan: depooled prod11 from haproxy
== April 4 ==
* ..... Stuff happened, [[Tech:Incidents/2015-04-04-prod7-resize]]
* 20:10 Addshore: Restart prod7
* 19:43 Addshore: prod9 back up and resized
* 19:41 Addshore: resize prod9 to 512mb instance and restart
* 19:36 Southparkfan: removed prod9 from haproxy config (planned for downgrade/re-install as needed)
* 14:51 Addshore: Login issues, Redis down, Restarted (We should really have a watchdog or something check and restart this)
== April 3 ==
* 21:24 Addshore: "pear install net_smtp" on prod9
* 20:35 Addshore: added prod9 back to LB, Cheers SPF!
* 20:24 Addshore: added prod9 back to LB -> it broke stuff -> promptly removed
* 20:20 Addshore: restarted redis-server on prod7 (yes everyone got logged out...)
* 20:16 Addshore: removed prod8 from LB for reboot then added back
* 20:10 Addshore: removed prod11 from LB for reboot then added back
* 19:00 Addshore: removed prod9 from LB and rebuilt (SPFCloud to add everything to prod9 and add back to LB)
== April 1 ==
* 13:15 Addshore: got reports users were unable to login. Redis was no longer running on prod7, restarted.
== March 26 ==
* 19:53 Southparkfan: noticed great things on prod9 :D
* 19:34 addshore: resize complete, powering back prod9
* 19:27 addshore: shutdown prod9 for resize
== March 17 ==
* 15:00 Southparkfan: ran update.php on memewiki
== March 16 ==
* 13:09 Southparkfan: HHVM died on prod8 for an unknown reason, causing downtime on the farm - restarted it
== March 14 ==
* 15:42 Southparkfan: ran update.php on lovelifesiftwwiki
== March 10 ==
* 17:23 Southparkfan: restart HHVM on all servers for HHVM admin password reset
== March 6 ==
* 23:47 Southparkfan: enable ansible on prod7
* 20:59 Southparkfan: disable ansible on prod7
* 20:48 Southparkfan: restart ssh on prod7
== March 5 ==
* 16:24 Southparkfan: kill'd & restarted HHVM on prod9 and prod11 too. Let's see if performance is improved now.
* 16:21 Southparkfan: disable ansible cron on prod9 and prod11
* 16:06 Southparkfan: start HHVM on prod8
* 16:06 Southparkfan: kill HHVM on prod8
* 15:48 Southparkfan: disable ansible cron on prod8 for HHVM testing
| |||