Ozzu Downtime

On Saturday, May 31st at 2:55pm PDT in the H1 data center where ozzu is located, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding the electrical equipment room. Thankfully, no one was injured.

This caused a loss of power to all of the networking gear and servers. The fire marshall wouldn't allow the backup generators to come online and much of the wiring was destroyed anyhow. I am very pleased that the ozzu server was not destroyed in the explosion and now everything is back up and online as of about 1 PM PDT today.

All I can say is it has been a very frustrating weekend and I am glad everything is back online. I think I might be taking a few days off soon.

I wouldn't blame you. Was great to see you on top of everything for the duration. You had OZZU back as timely as possible and worked a way to have updates where few others could have. My hat's off to you again chief!

Made a backup yet lol

This morning at approximately 12:45 a.m. PDT, the temporary generator supplying power to the servers and environmental control systems located in Phase 1 of our H1 facility shut down. This was caused by some faulty current sensors in the output breaker. The sensors detected an out of balance current condition that did not exist.

Technicians from the generator company were onsite within 15 minutes. After working on the breaker for an hour, they believed the issue was remedied, and the generator was restarted. As the servers and environmental control systems were brought back online, the breaker again caused the generator to trip offline.

Fixing the faulty breaker on the generator powering H1 Phase 1 was not successful.

The new H1 Generator has arrived on site and after load testing they finally brought floor 1 back online and that is why ozzu is now fixed.

For half of the day today ozzu was being loaded from a backup server. I may completely move ozzu out of the datacenter that is having problems to avoid any future downtime. I will keep everyone posted.

Did you move to the Dallas center?

Ditto. Great work boss!