Thread: Major Outage at The Planet

"From Doug Erwin:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and I apologize for the impact."


There is a lot of The Planet Customers discussing this now. It seems they are quite troubled by the lack of communication. I hope there aren't many if any WTF members affected. At least, it seems that the hostgator sites are not on this datacenter.

Although I am glad that I am not with them, there is nothing to gloat about this. All my hosting is with two dedicated servers with GNAX and I am not sure how I will cope if a similar thing happens over there. I wonder whether people with multiple servers spread their risk by using different datacenters?

That problem sounds massive, I am glad I am not with them.

Dedicated users are having a rough time this weekend, with outages, and huge price rises mg:

grrrr!!!!

It's da old, original ev1servers datacenter that's dead right now.

My server has been in this datacenter since 2001 and this is da first time I've ever seen something like this happen. I was one of da first users there, way back in the day where some one had to drive to the datacenter just to do requests to re-boot servers, so servers were down for hours!!

This is a hard core example of Murphy's law. I finished making an online RPG game, started a summers contest to open it up, then today I E-mailed thousands of my forum members about da contest...then a few hours later this happens!!!!!! gah!!!!!!

er, so no, you can't access Wacko University right now to torment AnthonyCea!!! :twisted:

Originally Posted by Nintendo

er, so no, you can't access Wacko University right now to torment AnthonyCea!!! :twisted:

Dayam!

That's terrible news for those customers at that datacenter. Glad we're using the LiquidWeb datacenter right now, we used to be with RackShack back in 2002 who changed names to Ev1Servers which is now ThePlanet.

Autumn: Care to elaborate about the huge price rises? Who/What/Where? :2072:

Keep in mind, no matter what datacenter you use, the same thing can happen to them.

Originally Posted by iowadawg

Keep in mind, no matter what datacenter you use, the same thing can happen to them.

I fully understand that, and we've had it happen to us before. I'm just glad it didn't happen in our datacenter this time around Like I stated before it sucks for those who are there, I truly feel their pain as a web host who has been through it myself.

grrrr!!!! Still down!!!!!

Originally Posted by felit0

Autumn: Care to elaborate about the huge price rises? Who/What/Where? :2072:

Layeredtech upped their prices by up to 30%.

See this huuuuuuuuuuge thread.

Price increased @ LT [ Merged ] - Web Hosting Talk - The largest, most influential web hosting community on the Internet

TechBlog: Explosion, fire takes Houston data center offline

Makes for interesting reading, and it shows the planet is doing all it can to get their servers in Houston back on line.

UGH! I feel so bad for you Nino and others that are effected by this. I freak at the smallest downage, I can't imagine this!

Son's company servers down because of the problems there.
But they have backup servers elsewhere, so not really affecting their business.

He told me that once they have everything in place at their own datacenter, they will drop theplanet.
They are going for a level 4 datacenter, bit costly, but damn that company makes money.

(trying to get FREE servers, etc from him.....)

One of my hosts had severs with The Planet, all my sites are down but the one's not using databases can still be accessed, cpanel can be accessed so why is it just my mysql down? I am glad my other two hosts aren't with The Planet.

Interesting.
I always thought mysql databases were on same server at site.
Maybe not?

Not usually in my experience, databases seem to always be separate

Houston Data Center fire / explosion takes PTO off line.. June 1, 2008 - Prison Talk

has pictures of the datacenter. er, I see no sign of an explosion!!!!!!!

Disclaimer: I am not typing this from a prison cell!!!! :twisted::twisted::twisted:

Originally Posted by Nintendo

Disclaimer: I am not typing this from a prison cell!!!! :twisted::twisted::twisted:

Now why would you need to add that? :evil:

I'm back up!!!! There's rumors that we might get two months credit or refund....

Upon the passage of five (5) continuous minutes of a Qualified Network Downtime Event, the Service Credit shall equal five percent (5%) of the monthly fees payable by Customer in respect of such Products and Services for the month in which such Qualified Network Downtime Event occurred and thereafter during the pendency of such unavailability, the Service Credit shall increase by an additional five percent (5%) for each continuous thirty (30) minutes of a Qualified Network Downtime Event up to a maximum of one hundred percent (100%) of monthly fees payable by Customer in respect of such Products and Services for the month in which such Qualified Network Downtime Event occurred.

http://www.theplanet.com/about_us/le...07%20FINAL.pdf

I pay $167.00 a month, so if I save $334, I'll start wishing this occured every two months on the last day of the month and came back on a day later!!!! :twisted::twisted::twisted:

Originally Posted by Autumn

Now why would you need to add that? :evil:

One look at da link and you'll quickly know!!! :twisted::twisted::twisted:

I have noticed that lots of website are down due to ThePlanet outage.

I didn't even know that ThePlanet has that many customers.

They are the largest public datacenter, with IBM being the largest private datacenter.

iowadawg added 355 Minutes and 40 Seconds later...

Keep up with theplanet here:
Maintenance at The Planet

http://service-update.theplanet.com/Erwin-msg6-2-08.wav

Audio from the CEO. Just about every server should be back up. I know of atleast one site I go to that's not!

Great job on their part getting everyone back in as short as possible time as possible under the circumstances

Originally Posted by Autumn

Great job on their part getting everyone back in as short as possible time as possible under the circumstances

They havn't done that yet!! About seven hours ago, all of us on floor one went off again, and we are still off. They FINALLY got the brains to offer to move us to another datacenter, which they should of done Saturday instead of saying they couldn't do that any more!!!!

http://service-update.theplanet.com/Erwin-msg6-4-08.wav

Two months credit!!! The second one comes in December.

Two months is pretty good

Its better than a slap in the face!

We had 10 servers affected by the outage, and issued all customers with a 2 month refund also. Its all you can really do in that situation.

We did also offer to move them to any other of our locations across the US at no charge.

TP had to go through a datacenters worst nightmare, and under pressure they did do a great job...hats off to their staff.

did it was an terrerist attack?

Originally Posted by Cocoras

did it was an terrerist attack?

Sheesh, more conspiracy theories? :eek:

I heard about this awhile ago and it was about a week after I moved my servers out of that very data center that had the minor explosion. Man I got good timing!