Have we gone down again ...

[danfulton]

Is it just me ?

[Robert]

If your VPS is located on VZ25, then your VPS is offline right now. :(

Please see the message I posted in the outtage and network announcements forum.

http://forums.powervps.com/showthread.php?t=493

We're working on this as quickly as possible. It may take an hour or so to bring the raid array online if it tries to do a file system check (fsck).

[danfulton]

Thanks for the info, nice to be kept informed.

Cheers

[shadowcat]

These guys are superb I would not worry too much. Usually support replies come with exclamation marks but these people are the real deal.

Polite and to the point.
Chipfryer.

[netrider]

If your VPS is located on VZ25, then your VPS is offline right now. :(

Please see the message I posted in the outtage and network announcements forum.

http://forums.powervps.com/showthread.php?t=493

We're working on this as quickly as possible. It may take an hour or so to bring the raid array online if it tries to do a file system check (fsck).

As a customer on this node, I would like to give a big THANKS and APPLAUD Rob Yates and his PowerVPS support team http://www.netrider.net.au/forums/images/smiles/eusa_clap.gif http://www.netrider.net.au/forums/images/smiles/eusa_clap.gif

My website community are quick and notified me of the problem almost immediately. By the time I had logged a support ticket, PowerVPS were already aware of the problem, and were working on it. My support docket response was within 10mins (the norm for PowerVPS support) and arrived even before my "mass mail-out to all customers on this node" email.

We (customers on this node) received regular email updates as PowerVPS fully realised what the problem was, put in to place their fix (once they had determined the best/quickest path), and then when it was underway/complete. As the PowerVPS posting says, they migrated us to a 'hot standby' server just to be safe/sure, and with no loss of data at all it was totally seamless to me and my users (apart from the initial outage).

Another related outage almost 24hours afterwards (a couple of hours ago) was very quickly addressed and fixed, with support saying they were still finishing the last of the migrations to the new/temporary server. So some 24 hours later, PowerVPS were still moving customers across to the hot standby server because they want to be safe/sure that our VPS's will be OK and so they can can investigate the reason for the node's original failure .... many, if not all, other hosting providers would have probably just rebooted/brought-back-up the failed node and hoped it will be alright, not spent many more hours being pro-active and investigative.

We moved from Shared Hosting 2.5 months ago because we had outgrown that model, and chose the PowerVPS solution after a concerted comparison of options (a non-profit and niche member community website has little funds) and reading about PowerVPS both in here and other 'webmaster' forums. This experience further cements and demonstrates why PowerVPS was an excellent choice!

We don't often hear much about the good work/service a company does or provides, and only hear when it's bad .. so I feel it's important to let others know when a job well done deserves praise. So once again ... applause and thanks to the PowerVPS support team

[david]

Netrider,

Thank you

(although I only sit around and bug Tony ;) someone has to crack the whip!).

[lundy]

Oh give yourself some credit David! You've certainly done more than that for me. :p

[ndndixie]

Netrider,

Thank you

(although I only sit around and bug Tony ;) someone has to crack the whip!).
Crack it an extra time for me......... :D

[elix]

VZ30 = down?

edit: nvm :-)

[Hvu]

I'm gettin 15% packetloss on eqash12

[elix]

Seems to be some random net outages again...

[elix]

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.1.1 - 0 | 211 | 211 | 0 | 0 | 16 | 0 |
| 10.9.0.1 - 0 | 211 | 211 | 0 | 8 | 32 | 15 |
| 167.206.32.33 - 0 | 210 | 210 | 0 | 8 | 32 | 15 |
| r3-ge10-1.mhe.hcvlny.cv.net - 0 | 210 | 210 | 0 | 8 | 31 | 15 |
| r2-srp0-0.wan.hcvlny.cv.net - 0 | 210 | 210 | 0 | 17 | 204 | 47 |
| r2-srp5-0.wan.prnynj.cv.net - 0 | 210 | 210 | 0 | 9 | 47 | 16 |
| r2-srp1-1.in.nwrknjmd.cv.net - 0 | 210 | 210 | 0 | 12 | 32 | 16 |
| 451be0d6.cst.lightpath.net - 0 | 210 | 210 | 0 | 13 | 32 | 0 |
| 0.ge-2-1-0.cr1.nyc3.us.nlayer.net - 0 | 210 | 210 | 0 | 12 | 32 | 16 |
| 0.so-2-1-0.cr1.iad1.us.nlayer.net - 0 | 210 | 210 | 0 | 17 | 94 | 15 |
| ge1-5.hr2.iad1.us.nlayer.net - 0 | 210 | 210 | 0 | 18 | 172 | 15 |
| dvlabs.ge1-8.hr2.iad1.us.nlayer.net - 0 | 210 | 210 | 0 | 24 | 407 | 16 |
| eqash30.defenderhosting.com - 13 | 210 | 183 | 0 | 16 | 63 | 16 |
| lancer.bizquality.com - 13 | 210 | 183 | 0 | 16 | 32 | 15 |
|________________________________________________| ______|______|______|______|______|______|
WinMTR - 0.8. Copyleft @2000-2002 Vasile Laurentiu Stanimir ( stanimir@cr.nivis.com )

[elix]

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.1.1 - 0 | 527 | 527 | 0 | 0 | 16 | 0 |
| 10.9.0.1 - 0 | 527 | 527 | 0 | 8 | 78 | 15 |
| 167.206.32.33 - 0 | 527 | 527 | 0 | 9 | 47 | 15 |
| r3-ge10-1.mhe.hcvlny.cv.net - 0 | 527 | 527 | 0 | 9 | 47 | 15 |
| r2-srp0-0.wan.hcvlny.cv.net - 1 | 527 | 526 | 0 | 17 | 204 | 15 |
| r2-srp5-0.wan.prnynj.cv.net - 0 | 526 | 526 | 0 | 10 | 125 | 31 |
| r2-srp1-1.in.nwrknjmd.cv.net - 0 | 526 | 526 | 0 | 12 | 63 | 16 |
| 451be0d6.cst.lightpath.net - 0 | 526 | 526 | 0 | 13 | 63 | 16 |
| 0.ge-2-1-0.cr1.nyc3.us.nlayer.net - 0 | 526 | 526 | 0 | 13 | 78 | 16 |
| 0.so-2-1-0.cr1.iad1.us.nlayer.net - 0 | 526 | 526 | 0 | 17 | 94 | 15 |
| ge1-5.hr2.iad1.us.nlayer.net - 0 | 526 | 526 | 0 | 18 | 187 | 16 |
| dvlabs.ge1-8.hr2.iad1.us.nlayer.net - 0 | 526 | 526 | 0 | 25 | 407 | 32 |
| eqash30.defenderhosting.com - 22 | 526 | 413 | 0 | 17 | 63 | 16 |
| lancer.bizquality.com - 22 | 526 | 413 | 0 | 17 | 47 | 16 |
|________________________________________________| ______|______|______|______|______|______|
WinMTR - 0.8. Copyleft @2000-2002 Vasile Laurentiu Stanimir ( stanimir@cr.nivis.com )

[elix]

Seems to be okay now..will continue to monitor.

[charles]

I am unable to trace back to you, but the packet loss your seeing on the last two hops will reveal itself if we could. Can you add your source IP to the ticket, or try it yourself from your VPS please? mtr/traceroute to 167.206.32.33 is getting me nowhere.

thanks
charles

[charles]

nm, I just saw your ticket update. I can trace back to your IP cleanly now. We saw packet loss in cv.net., level3.net and also nlayer at the time of the problem. We'll keep monitoring it, and if we can pin it down we'll try bias the routing around it. Right now would be a shotgun approach though.

charles

[Hvu]

Charles, my IP is mypc.xiphin.com

Also still getting occational ping spike.

[charles]

Henry, l'm seeing minor loss in sbcglobal.net, but its definately spiking.



Packets Pings
Hostname %Loss Rcv Snt Last Best Avg Worst
1. colo.aleron.com
0% 435 435 0 0 7 175 2. ge2-10.as.eqxashva.aleron.net
0% 435 435 44 0 15 253 3. ex1-g9-1-0.eqabva.sbcglobal.net 0% 435 435 0 0 10 199 4. bb1-p11-0.hrndva.sbcglobal.net
0% 435 435 0 0 9 204 5. core1-p5-0.crhnva.sbcglobal.net
0% 435 435 0 0 14 213 6. core1-p9-0.crcloh.sbcglobal.net
0% 435 435 8 8 24 235 7. core1-p3-0.crchil.sbcglobal.net
0% 435 435 22 21 38 242 8. core2-p8-0.crchil.sbcglobal.net
0% 435 435 25 24 41 259 9. core1-p4-0.crkcmo.sbcglobal.net
0% 435 435 36 36 45 314 10. core1-p3-0.crdnco.sbcglobal.net
0% 435 435 48 48 63 256 11. core1-p2-0.crskut.sbcglobal.net
1% 434 435 71 71 87 311 12. core1-p2-0.crsfca.sbcglobal.net
0% 435 435 71 71 87 311 13. bb1-p8-0.crsfca.sbcglobal.net
0% 435 435 71 71 79 254 14. bb1-p4-0.pltnca.sbcglobal.net
0% 435 435 72 72 84 280 15. dist3-10g9-2.pltnca.sbcglobal.net
0% 435 435 72 72 72 133 16. bras1-g9-0.pltnca.sbcglobal.net
1% 434 435 74 73 77 155 17. ???

charles

[Hvu]

Yeah its spiking, ever since they rolled out the 14.95 DSL deal the network gone bad


Target Name: xiphin.com
IP: 204.157.7.6
Date/Time: 7/29/2005 5:50:15 PM to 7/29/2005 5:51:40 PM

Hop Sent Err PL% Min Max Avg Host Name / [IP]
1 35 0 0.0 2 8 2 [192.168.1.1]
2 35 0 0.0 2 22 3 [192.168.0.1]
3 35 0 0.0 12 40 15 bras1-l0.pltnca.pbi.net [151.164.184.77]
4 35 0 0.0 11 473 25 dist2-vlan50.pltn13.pbi.net [64.164.97.67]
5 35 0 0.0 11 443 39 bb2-10g2-0.pltnca.sbcglobal.net [151.164.42.102]
6 35 0 0.0 12 405 37 bb1-p5-0.crscca.sbcglobal.net [151.164.240.137]
7 35 0 0.0 13 360 31 ex1-p13-0.pxpaca.sbcglobal.net [151.164.190.206]
8 35 0 0.0 14 312 25 asn4436-nlayer.pxpaca.sbcglobal.net [151.164.248.90]
9 35 0 0.0 79 331 88 0.so-2-1-0.cr1.iad1.us.nlayer.net [69.22.142.102]
10 35 0 0.0 80 292 90 ge1-1.hr1.iad1.us.nlayer.net [69.31.31.130]
11 35 0 0.0 80 247 93 dvlabs.ge2-2.hr1.iad1.us.nlayer.net [69.31.31.86]
12 35 0 0.0 80 202 85 eqash12.defenderhosting.com [204.157.0.74]
13 35 0 0.0 80 161 84 unknown6.7.157.204.defenderhosting.com [204.157.7.6]


We'll my users arnt complaing anymore and i'm moving to DSLextreme on monday so i'll just leave it at this. I'll holler if they complain again.

[Hvu]

Still Ping Spikes, not sure if its me or nLayer

My IP: mypc.xiphin.com

Target Name: xiphin.com
IP: 204.157.7.6
Date/Time: 7/30/2005 10:04:05 AM to 7/30/2005 10:05:30 AM

Hop Sent Err PL% Min Max Avg Host Name / [IP]
1 35 0 0.0 2 6 2 [192.168.1.1]
2 35 0 0.0 2 14 3 [192.168.0.1]
3 35 0 0.0 12 36 16 bras1-l0.pltnca.pbi.net [151.164.184.77]
4 35 0 0.0 11 116 16 dist2-vlan50.pltn13.pbi.net [64.164.97.67]
5 35 0 0.0 11 192 38 bb2-10g2-0.pltnca.sbcglobal.net [151.164.42.102]
6 35 0 0.0 13 132 21 bb1-p5-0.crscca.sbcglobal.net [151.164.240.137]
7 35 0 0.0 14 128 24 ex1-p13-0.pxpaca.sbcglobal.net [151.164.190.206]
8 35 0 0.0 14 387 26 asn4436-nlayer.pxpaca.sbcglobal.net [151.164.248.90]
9 35 0 0.0 80 409 91 0.so-2-1-0.cr1.iad1.us.nlayer.net [69.22.142.102]
10 35 0 0.0 79 366 95 ge1-1.hr1.iad1.us.nlayer.net [69.31.31.130]
11 35 0 0.0 80 387 99 dvlabs.ge2-2.hr1.iad1.us.nlayer.net [69.31.31.86]
12 35 0 0.0 80 282 87 eqash12.defenderhosting.com [204.157.0.74]
13 35 0 0.0 80 236 86 unknown6.7.157.204.defenderhosting.com [204.157.7.6]

[elix]

Looks like it's SBC..

[PvUtrix]

Damn... now mine is down....
http://forums.powervps.com/showthread.php?t=502

[TomK]

Yea, talk about that idiot "Murphy" - typical, no major issues for a long time and now 2 hardware issues in one week.

We'll know more shortly, please hang tight.

PS - Idiot as is Murphy's Law...

[PvUtrix]

Come on, prove this idiot wrong!

[TomK]

The raid array appears to be dead, even though it has redundant PS - we are going to swap the unit out now and move the drives over to the new array.

Sorry, wish the news was better, we should have the new array racked and start moving the physical HDD's over shortly, then it'll need to fsck, and assuming everything else is OK, should come back up.

[TomK]

Come on, prove this idiot wrong!

The unit has been swapped out and the array is functional again, and VPS servers are starting.

[PvUtrix]

looks like we're back in business! Well done!

[ndndixie]

Yea, talk about that idiot "Murphy" - typical, no major issues for a long time and now 2 hardware issues in one week.

We'll know more shortly, please hang tight.

PS - Idiot as is Murphy's Law...
Whatever shall break shall do it all in one week. That's the truth with anything electrical/electronic. If my dryer goes out, the ac will follow, if the fridge goes out, the dishwasher goes on the fritz......lol, you can't avoid it, it's a universal law!

[Robert]

Also... if you would like to trace back from your own VPS to your local connection, run the following command while logged in as root:

/usr/sbin/traceroute YOURIPHERE

[Hvu]

Having some fun

http://img.photobucket.com/albums/v86/pztup/trace.jpg

[elix]

Having some fun

http://img.photobucket.com/albums/v86/pztup/trace.jpg
lol. What program is that?

[Hvu]

http://www.d3tr.com/