Problems - Virginia Data Center

[martin]

Hello,

Due to "act of god" weather conditions our Virginia data center is experiencing problems.

One of our U.S servers is currently still having issues. We are currently working with tech support to resolve this issue ASAP.

Thanks.
.

[martin]

Update from the data center:

Currently, there are electrical engineers in our data center now investigating why we lost power in certain areas of our data center.

Servers impacted:

Server #2 us.voxalot.com (server #1 in same data center has backup power and is not impacted)
Server sipbroker.com

We will keep you informed.
.

[martin]

For those people having problems connecting, please temporarily use any of the other servers:

eu.voxalot.com
au.voxalot.com
.

[martin]

From what I understand, it seems that segments of a UPS have failed which is causing continued outages in the PDU's containing 2 of our 3 U.S. servers.

We will keep you updated.
.

[martin]

All voxalot servers are now fully operational. We are however still waiting for sipbroker.com to get power.
.

[martin]

All servers including sipbroker.com are back online. Thanks for you patience.

Official response from data center.

We would like to address the power loss which occurred in our Virginia
Datacenter on Wednesday, June 13th. We are still investigating the root
cause, but in the interest of full disclosure, here are the facts as we
know them today. A more complete post-mortem will be sent to you as soon
as possible.

- A massive wind and hail storm struck the mid-Atlantic region of the
United States including Northern Virginia yesterday afternoon.

- Our internal monitoring system alerted us that the local power grid
dropped at approximately 4:01 pm EDT.

- We have three generators on site:

- - The first generator failed to start up correctly. We are not sure why
and our engineers are investigating further.

- - The second generator started as expected.

- - The third generator started initially but soon failed for an
unknown reason. We are investigating root cause here as well.

- Because we had only partial generator power, not all the Power
Distribution Units (PDUs) were receiving power.

- Power was restored to approximately 50% of our servers in a matter of
minutes. The remaining ~50% required further attention from our
engineers.

- All power was restored at approximately 6:15pm EDT although there were
isolated power fluctuations over the next two hours.

All available technicians were on-site throughout this incident and have
remained in the datacenter over the last 24 hours to ensure all servers
came back online.

.