Ozzu Downtime

  • Bigwebmaster
  • Site Admin
  • Site Admin
  • User avatar
  • Posts: 9092
  • Loc: Seattle, WA & Phoenix, AZ

Post 3+ Months Ago

On Saturday, May 31st at 2:55pm PDT in the H1 data center where ozzu is located, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding the electrical equipment room. Thankfully, no one was injured.

This caused a loss of power to all of the networking gear and servers. The fire marshall wouldn't allow the backup generators to come online and much of the wiring was destroyed anyhow. I am very pleased that the ozzu server was not destroyed in the explosion and now everything is back up and online as of about 1 PM PDT today.

All I can say is it has been a very frustrating weekend and I am glad everything is back online. I think I might be taking a few days off soon.
  • ATNO/TW
  • Super Moderator
  • Super Moderator
  • User avatar
  • Posts: 23455
  • Loc: Woodbridge VA

Post 3+ Months Ago

I wouldn't blame you. Was great to see you on top of everything for the duration. You had OZZU back as timely as possible and worked a way to have updates where few others could have. My hat's off to you again chief!
  • SpooF
  • ٩๏̯͡๏۶
  • Bronze Member
  • User avatar
  • Posts: 3422
  • Loc: Richland, WA

Post 3+ Months Ago

Made a backup yet ;) lol
  • Bigwebmaster
  • Site Admin
  • Site Admin
  • User avatar
  • Posts: 9092
  • Loc: Seattle, WA & Phoenix, AZ

Post 3+ Months Ago

This morning at approximately 12:45 a.m. PDT, the temporary generator supplying power to the servers and environmental control systems located in Phase 1 of our H1 facility shut down. This was caused by some faulty current sensors in the output breaker. The sensors detected an out of balance current condition that did not exist.

Technicians from the generator company were onsite within 15 minutes. After working on the breaker for an hour, they believed the issue was remedied, and the generator was restarted. As the servers and environmental control systems were brought back online, the breaker again caused the generator to trip offline.

Fixing the faulty breaker on the generator powering H1 Phase 1 was not successful.

The new H1 Generator has arrived on site and after load testing they finally brought floor 1 back online and that is why ozzu is now fixed.

For half of the day today ozzu was being loaded from a backup server. I may completely move ozzu out of the datacenter that is having problems to avoid any future downtime. I will keep everyone posted.
  • SpooF
  • ٩๏̯͡๏۶
  • Bronze Member
  • User avatar
  • Posts: 3422
  • Loc: Richland, WA

Post 3+ Months Ago

Did you move to the Dallas center?
  • LAbrego
  • brego from LA
  • Web Master
  • User avatar
  • Posts: 2855

Post 3+ Months Ago

ATNO/TW wrote:
I wouldn't blame you. Was great to see you on top of everything for the duration. You had OZZU back as timely as possible and worked a way to have updates where few others could have. My hat's off to you again chief!


Ditto. Great work boss!

Post Information

  • Total Posts in this topic: 6 posts
  • Users browsing this forum: No registered users and 2 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum
 
 

© 1998-2014. Ozzu® is a registered trademark of Unmelted, LLC.