[US Server] Phoenix Datacenter - Server Offline - 22 Nov 2013

  • November 22, 2013 by Tech #1
‚ÄčDear Phoenix/US server Customers,

We've received a report from the datacenter on this issue now. Several months ago, there was a power outage on one of our power circuits, where PhoenixNap staff had to move our power circuit to a different transformer as emergency maintenance to get power back online for that power circuit. we have now learned that the outage today was because PhoenixNap felt the need to move us off of this new circuit and onto a different transformer, even though the circuit we are on has been working perfectly well while we have been using it. We are subscribed to receive maintenance notifications from PhoenixNap, of which we have many notifications about trivial issues such as testing their fire alarms and other things we don't care about, but we never received any notification that they intended to intentionally cause us a power loss. We also did not receive any notification that they intended to shut down the affected servers prior to this maintenance. In addition to these issues, they f  ailed to properly shut down some affected servers, and they also shut down some servers that were not going to be affected by the power outage, which in turn then caused some servers to not come back up properly without manual intervention.

Naturally we are very disappointed that they would first of all, but we are truly sorry for what has happened to you.

It was rather unfortunate series of events actually. It turns out that the preparation that we did for the processor migration has also caused the NIC card to failed to load the correct one. The supermicro that we used, has 2 nic chips, and linux get confused on which mac ordering.
This is because we commented out the one particular network config in preparation for the processor migration in the new server chassis.
On the normal occasion, this wont happen as we have set the network config properly even in the case of server reboot.

So, it's from the power loss that caused the server to shutdown, then server reboot into the wrong nic configuration on which we had to intervened to manually set to the correct NIC.

Initially we thought of the RAM issue, but through checking and the RFI from datacenter, we have comed into this conclusion.

On behalf, im truly sorry for the inconvenience caused.

Thank you

Kind Regards,

BK Web Solutions