Is there a network/server problem?

[alexv]

Anybody else expirience problems with their VPS?

Thank you,
Alex.

[attilas]

I experienced same problem, now vps is back.

[rytavi]

I saw mine come back for exactly 91 seconds, then disappear again.

[ndndixie]

I'm back up ........lol fielding e-mails from customers. Anyone got a bottle of 20/20? :D

[Jon]

Hey Guys-

If you didnt already get this response from sending in a ticket, here it is:

We are working an issue in our Ashburn Equinix location that is causing an outage/high packet loss across the network, we will update this thread when we find out more.

We are sorry about this issue, and we have everyone on this urgent matter at this time.

This ONLY is causing issues for customers on the BGP mix, the single transit customers alone are fine, but this email blast is going to all customers.

We will update you shortly when we figure this issue out.

Regards,
Defender/PowerVPS Support

[ndndixie]

Thanks Jon, just got it.you guys are great as always!

[Jon]

I'll update you again when all BGP sessions are back up and everything is 100%.

[mpattman]

Up and now back down again, traceroutes are all over the place.

[mpattman]

Thanks for the update Jon.

[ndndixie]

Yeah, I'm gone again too mpattman. Tylenol anyone?

[PvUtrix]

Ouch!
I wonder how long it will take to have this fixed...
Come on guys, you can do it!

[ndndixie]

Lol, I just got a call from a customer and after explaining to her what was going on, that it was an "across the board" problem and was being taken care of...she says "well do you think you could ask them to get just my site up first?".........it was real hard not to laugh and say "yeah sure, we'll hold the presses and make sure one ip gets priority status." She runs a homemade crafts site....what a horror that one less ugly toilet paper cover won't be sold today.:rolleyes:

[PvUtrix]

Well, I have an international union of combat karate down and they will kick my a$$, if their site is not up soon :) and they are already smsing me... :)

[ndndixie]

Lol OOPS! You could tell them that evil forces took it down .
My biggest concern is the air races site......it's getting mega hits per day and with the races in a month.........things are really gearing up. I haven't heard from them yet but I'm sure I will.

[fourbit]

Hi folks.

I am new to this VPS stuff. And this company. So far it's been great! The support is a lot of help. But, then today I can't access my system. At first I thought it was something I had done.

Then I saw their email. I like being informed. Good job guys.

Paul

[dean_dodd]

Whats the difference between the BGP mix and single transit customers?

[PvUtrix]

I already did :)

I'm glad this had to happen on a Sunday and at this time... not а very busy time, for me... I hope the VPS is back up in a few hours, cause I am watching the uptime percentage going down, down, down and I don't think it will be able to stay above the uptime guarantee for too long...

I'm gonna go meditate and send positive energy across the astral pathways to the engeneers fixing this... hope it helps :)

[Jon]

Most customers should already be back up, we are working to restore the rest.

[ndndixie]

Lol, mine is back up right now. Sunday is slow for me too so it's not as bad as a Monday would be! I think I'd disconnect my phone lines if that happened!

Hi Fourbit....this isn't a normal occurance at all. I think that's why we're all in shock. We've been spoiled by great service around here.

It WAS back up.lol down again. I think I'm going for some of that meditation as well.

[PvUtrix]

Mine is back...

[PvUtrix]

Damn... gone again...

[ndndixie]

Mine is coming and going. I guess I can go argue about the draft lol.......that always eases my tensions.....:P

[fourbit]

Mine's not yet. But, I am sure they are working on it.

I didn't think that this was a common occurance. I did a lot of research before I bought. Great co. with great references.

I just figure everyone is out to get me. LOL . (not really) I just take it in stride. Fortunately, I only have one customer on the server so far. But, I hope to migrate a few more this week.

Paul

[dean_dodd]

Mine is up and down to :(

Hopefully wont be to long now before the top support bods have it sorted ;)

[ndndixie]

Lol, I KNOW they are out to get me!
There are some NFL fans that are really wishing I would go away and go back to work. After telling them that armchair general managers didn't get paid to draft, I'm not really popular at the moment.:D

[Jon]

Should be stable now guys.

[ndndixie]

Should be but isn't :D .......still down here Jon.

[cprompt]

No it's not. Mine is still coming and going.

[Jon]

Yea it might be a little bit unstable until I can get all the BGP sessions back up.

[PvUtrix]

Yep, still unstable... down at the moment...

[ndndixie]

Up at the moment

[Jon]

Seems to be a Juniper bug. We are routing around the affected core router until we can fix it.

[Jon]

Okay everything should now be up and 100% stable.

If you are still down, please reply to your opened ticket or send a new one into support@powervps.com or support@defenderhosting.com

[nobody]

Can somebody actually update us on what caused the problem with the network?

[fourbit]

YEA!!! Good job Jon.

Can somebody actually update us on what caused the problem with the network?
Yea. I need to tell my customer what happened. ;)

Or would you prefer to do this through the Ticket system.

Paul

[Jon]

This is preliminary, we are still doing a postmortem to find out if this was indeed the root cause.

We had an issue with one of the trunks off the Juniper to the core switches. We are not sure if it is/was a JUNOS issue or something on the core switches, all subnets are now back up and operational.

I want to thank Richard from nLayer who came on-site with us and helped troubleshoot these issues.

I'll post later once we have more information, if you are still down and having problems, please submit an email to support@powervps.com or support@defenderhosting.com if the helpdesk isn't working for you.

We are truly sorry for this major outage.

[Jon]

Please see the above post

[fourbit]

Thanks Jon,

I hate those bugs that get into the Junipers. But, here in Montana we call them "Pine Beetles". :D Man, they can devistate a whole forest in no time.

Sorry, just had to be silly. I used to be a logger and now I'm a "webmaster". HA!!! Yea, right.

Anyhow, thanks for the job well done. And the information. Hopefully, when the autopsy is done, we'll get some information that is in a language that I can understand. ;)

Paul

[dean_dodd]

Mine seems pretty stable now - thanks for the good work Jon.

[TomK]

Guys, see here, I'll update this thread as needed, but we are up now for the last couple hours 100% stable network-wise.

http://forums.deftechgroup.com/showthread.php?t=173

[ndndixie]

Hey, stuff happens. Just glad we're with someone who didn't say "we'll fix it Monday." You guys rock, thanks for the fast response! I've been up and stable since my last post up there so all is good on this end.

[Robert]

Whew! It feels good to have that behind us now. As you can imagine, most of us were running around like chickens with our heads cut off. LOL

While it may have seemed like forever to most of ya seeing your site down, it was 50 times worse for all of us who were working at the time (including those called in as a result). Hopefully everyone felt that we handled the situation appropriately. We did our best trying to keep people updated as to what happened with sending a blast e-mail to all customers, posting on WHT, and responding back to trouble tickets ASAP with a note as to what was going on.

No one likes to see downtime, but when the worst happens we try to do our best to make sure you are not kept in the dark.

[cprompt]

One of the problems this incident highlighted is having your contact email for PowerVPS hosted on your VPS like I did! Dooah!... no email explaining what was going on 'cos my emails down! :D
If I change my contact email to an off-site address, is it enough to change it in the PowerVPS Member Center or do I need to notify support as well to get my mailing list entry changed?

[charles]

One of the problems this incident highlighted is having your contact email for PowerVPS hosted on your VPS like I did! Dooah!... no email explaining what was going on 'cos my emails down! :D
If I change my contact email to an off-site address, is it enough to change it in the PowerVPS Member Center or do I need to notify support as well to get my mailing list entry changed?

No, that just changes it in the members center which we don't use for announcements. Please email support as well so we can change in our other systems.

thanks
charles

[charles]

Whats the difference between the BGP mix and single transit customers?

Dean, single transit means the bandwidth comes from only one provider. If they have a problem, you have a problem

The BGP mix has multiple providers, and if one gives trouble, your packets get routed through another until the problem clears up.

This **shouldn't** have happened, and we are still looking into why, but have some plans in the works already to ensure it doesn't happen again.

charles

[Chris]

Well now that the dust has cleared . . . I'll put my 2 cents in . . . whether you want it or not.

Not sure what happened, but "stuff" happens. Always has . . . always will . . . they key is how you deal with it.

To me the blanket email was a great thing. I got two of them and truly appreciate it. It allowed me to go to my customers with knowledge instead of that . . . uh, let me look at it response. If I could only get my other providers to do that with such outages. This again sets you apart and above. Thanks

[MDurai]

My backend is running terribly slow... I've had friends try it and they too see it. I'm not sure what the problem is...

[TomK]

What does this have to do with the outage from a week ago?