Very slow network

**Ron** · 07-29-2006, 11:07 PM

Slow network to this forum (and my site)...

Anyone else seeing this problem for the last couple of hours?

Code:

-------------------------------------------------------
Tracing route xxxxxxxxxxxxxxxxxxxxxx
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.0.1
  2     2 ms     2 ms     2 ms  10.9.44.4
  3     2 ms     2 ms     1 ms  P2-2.LCR-02.BSTNMA.verizon-gni.net [130.81.32.182]
  4    12 ms    12 ms    11 ms  so-7-0-0-0.PEER-RTR1.NY111.verizon-gni.net [130.81.17.131]
  5    12 ms    12 ms    12 ms  so-6-0-0.pr2.lga1.us.above.net [64.125.13.33]
  6    12 ms    12 ms    11 ms  so-5-0-0.cr2.lga1.us.above.net [64.125.27.138]
  7    17 ms    17 ms    17 ms  so-1-0-0.mpr2.iad1.us.above.net [64.125.28.65]
  8    18 ms    18 ms    18 ms  so-6-0-0.cr2.dca2.us.above.net [64.125.27.210]
  9    49 ms    49 ms    49 ms  so-3-2-0.cr2.dfw2.us.above.net [64.125.28.205]
 10    54 ms    54 ms    54 ms  so-3-0-0.mpr2.iah1.us.above.net [64.125.31.41] 11   280 ms   140 ms    54 ms  209.133.90.85.available.above.net [209.133.90.85]
 12    54 ms    54 ms    54 ms  hou-rtr3-vl3.cyrusone.com [69.7.175.35]
 13    55 ms    54 ms    55 ms  vl2-rtr1.nocdirect.com [69.7.160.227]
 14    54 ms    54 ms    57 ms  xxxxxxxxxxxxxxx

Trace complete.
----------------------------------------------------
Tracing route to xxxxxxxxxxxxxxxxxxxxx
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.0.1
  2     2 ms     1 ms     1 ms  10.9.44.4
  3     2 ms     2 ms     1 ms  P2-2.LCR-02.BSTNMA.verizon-gni.net [130.81.32.182]
  4    12 ms    12 ms    11 ms  so-7-0-0-0.PEER-RTR1.NY111.verizon-gni.net [130.81.17.131]
  5    12 ms    12 ms    12 ms  so-6-0-0.pr2.lga1.us.above.net [64.125.13.33]
  6    12 ms    12 ms    12 ms  so-5-0-0.cr2.lga1.us.above.net [64.125.27.138]
  7    17 ms    18 ms    17 ms  so-1-0-0.mpr2.iad1.us.above.net [64.125.28.65]
  8    18 ms    18 ms    18 ms  so-6-0-0.cr2.dca2.us.above.net [64.125.27.210]
  9    49 ms    49 ms    49 ms  so-3-2-0.cr2.dfw2.us.above.net [64.125.28.205]
 10    54 ms    53 ms    53 ms  so-3-0-0.mpr2.iah1.us.above.net [64.125.31.41]
 11   204 ms   199 ms   199 ms  209.133.90.85.available.above.net [209.133.90.85]
 12    54 ms    54 ms    54 ms  hou-rtr3-vl3.cyrusone.com [69.7.175.35]
 13     *       56 ms    56 ms  vl2-rtr1.nocdirect.com [69.7.160.227]
 14    54 ms    55 ms    54 ms  xxxxxxxxxxxxxxxxxxxx

--------------------------------------------------------
Tracing route to xxxxxxxxxxxxxxxxxxxxxxx
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.0.1
  2     2 ms     2 ms     1 ms  10.9.44.4
  3     2 ms     2 ms     1 ms  P2-2.LCR-02.BSTNMA.verizon-gni.net [130.81.32.182]
  4    11 ms    12 ms    11 ms  so-7-0-0-0.PEER-RTR1.NY111.verizon-gni.net [130.81.17.131]
  5    12 ms    11 ms    11 ms  so-6-0-0.pr2.lga1.us.above.net [64.125.13.33]
  6    12 ms    12 ms    20 ms  so-5-0-0.cr2.lga1.us.above.net [64.125.27.138]
  7    17 ms    22 ms    17 ms  so-1-0-0.mpr2.iad1.us.above.net [64.125.28.65]
  8    18 ms    18 ms    17 ms  so-6-0-0.cr2.dca2.us.above.net [64.125.27.210]
  9    49 ms    49 ms    49 ms  so-3-2-0.cr2.dfw2.us.above.net [64.125.28.205]
 10    54 ms    53 ms    54 ms  so-3-0-0.mpr2.iah1.us.above.net [64.125.31.41]
 11   149 ms   247 ms   192 ms  209.133.90.85.available.above.net [209.133.90.85]
 12    54 ms    54 ms    54 ms  hou-rtr3-vl3.cyrusone.com [69.7.175.35]
 13    54 ms    54 ms    55 ms  vl2-rtr1.nocdirect.com [69.7.160.227]
 14     *       54 ms    54 ms  xxxxxxxxxxxxxxxxxx

Trace complete.
--------------------------------------------------------
Tracing route to xxxxxxxxxxxxxxxxxxxxxx
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.0.1
  2     2 ms     2 ms     1 ms  10.9.44.4
  3     2 ms     2 ms     2 ms  P2-2.LCR-02.BSTNMA.verizon-gni.net [130.81.32.182]
  4    12 ms    12 ms    11 ms  so-7-0-0-0.PEER-RTR1.NY111.verizon-gni.net [130.81.17.131]
  5    12 ms    11 ms    12 ms  so-6-0-0.pr2.lga1.us.above.net [64.125.13.33]
  6    12 ms    12 ms    12 ms  so-5-0-0.cr2.lga1.us.above.net [64.125.27.138]
  7    17 ms    17 ms    17 ms  so-1-0-0.mpr2.iad1.us.above.net [64.125.28.65]
  8    18 ms    18 ms    18 ms  so-6-0-0.cr2.dca2.us.above.net [64.125.27.210]
  9    49 ms    49 ms    49 ms  so-3-2-0.cr2.dfw2.us.above.net [64.125.28.205]
 10    54 ms    53 ms    54 ms  so-3-0-0.mpr2.iah1.us.above.net [64.125.31.41]
 11    54 ms    54 ms    54 ms  209.133.90.85.available.above.net [209.133.90.85]
 12    54 ms    54 ms    54 ms  hou-rtr3-vl3.cyrusone.com [69.7.175.35]
 13   382 ms   133 ms    56 ms  vl2-rtr1.nocdirect.com [69.7.160.227]
 14    54 ms    54 ms    56 ms  xxxxxxxxxxxxxxxxxxxx

Trace complete.
----------------------------------------------------------

Tracing route to xxxxxxxxxxxxxxxxxxxx
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.0.1
  2     2 ms     2 ms     1 ms  10.9.44.4
  3     2 ms     1 ms     1 ms  P2-2.LCR-02.BSTNMA.verizon-gni.net [130.81.32.182]
  4    12 ms    79 ms    12 ms  so-7-0-0-0.PEER-RTR1.NY111.verizon-gni.net [130.81.17.131]
  5    12 ms    12 ms    12 ms  so-6-0-0.pr2.lga1.us.above.net [64.125.13.33]
  6    12 ms    11 ms    12 ms  so-5-0-0.cr2.lga1.us.above.net [64.125.27.138]
  7    18 ms    17 ms    17 ms  so-1-0-0.mpr2.iad1.us.above.net [64.125.28.65]
  8    18 ms    17 ms    17 ms  so-6-0-0.cr2.dca2.us.above.net [64.125.27.210]
  9    81 ms    49 ms    49 ms  so-3-2-0.cr2.dfw2.us.above.net [64.125.28.205]
 10    53 ms    53 ms    63 ms  so-3-0-0.mpr2.iah1.us.above.net [64.125.31.41]
 11    54 ms    54 ms    56 ms  209.133.90.85.available.above.net [209.133.90.85]
 12    54 ms    54 ms    54 ms  hou-rtr3-vl3.cyrusone.com [69.7.175.35]
 13    54 ms    54 ms    54 ms  vl2-rtr1.nocdirect.com [69.7.160.227]
 14     *        *       55 ms  xxxxxxxxxxxxxxxxx

Trace complete.
----------------------------------

First was a good traceroute for comparison....

Seems like the problem has been shifting around between the server, the last router and the last outside router...
I've looked at the server and it's doing just fine; in fact I'm running stuff through SSH that's interactive and there is 0 lag, so this is HTTP and HTTPS (since I can just barely interact with the JAG ticket system) Also this forum is dreadfully slow (more than 3 tries and then 2 minutes to get to this text entry page).

Support says things are all better now...

**Vin DSL** · 07-29-2006, 11:37 PM

Looks like a bad edge router to me...

**Spathiphyllum** · 07-29-2006, 11:38 PM

Yes. The forum has been running slowly this evening.

Code:

  1     1 ms   <10 ms   <10 ms  firewall [192.168.xxx.xxx]
  2     *        *        *     Request timed out. [modem - ping response free]
  3     9 ms     *        9 ms  GE-1-2-ur01.somewhere3.tx.dallas.comcast.net [68.87.204.5]
  4    14 ms    11 ms    10 ms  te-8-1-ur01.somewhere2.tx.dallas.comcast.net [68.87.207.41]
  5     *        9 ms     9 ms  te-8-3-ar01.somewhere1.tx.dallas.comcast.net [68.87.207.29]
  6    11 ms     9 ms    12 ms  12.118.225.13
  7    12 ms    13 ms    14 ms  tbr2-p013801.dlstx.ip.att.net [12.123.16.102]
  8    10 ms    11 ms    12 ms  gar2-p370.dlstx.ip.att.net [12.123.16.93]
  9    12 ms    12 ms    11 ms  att-gw.dfw.above.net [192.205.32.162]
 10    16 ms    17 ms    16 ms  so-4-0-0.mpr1.iah1.us.above.net [64.125.31.37]
 11    17 ms    18 ms    17 ms  209.133.90.85.available.above.net [209.133.90.85]
 12    16 ms    16 ms    16 ms  hou-rtr3-vl3.cyrusone.com [69.7.175.35]
 13    19 ms    17 ms   304 ms  vl2-rtr1.nocdirect.com [69.7.160.227]
 14    17 ms    17 ms    17 ms  xxxxxxxxxxxxxxx [xxxxxxxxxxxxxxx]

Different node showing delay.

**Vin DSL** · 07-29-2006, 11:46 PM

OMG!!! Guess who?

Code:

Microsoft Windows 2000 [Version 5.00.2195]
(C) Copyright 1985-2000 Microsoft Corp.

C:\>tracert lenon.com

Tracing route to lenon.com [69.73.166.75]
over a maximum of 30 hops:

  1     1 ms   <10 ms   <10 ms  XX.X.X.X
  2    44 ms    43 ms    44 ms  lo0-ar1.phx.cybertrails.com [XXX.XX.XXX.X]
  3    44 ms    45 ms    45 ms  ge-7-0.1000M.gsr1.phx.cybertrails.com [XXX.XX.XXX.X]
  4    47 ms    45 ms    46 ms  phv-edge-02.inet.qwest.net [72.164.12.9]
  5    45 ms    46 ms    45 ms  phx-core-01.inet.qwest.net [205.171.129.113]
  6    71 ms    65 ms    64 ms  svx-core-02.inet.qwest.net [67.14.1.58]
  7    68 ms    64 ms    65 ms  sjp-brdr-01.inet.qwest.net [205.171.214.138]
  8    64 ms    68 ms    64 ms  so-2-0-0.edge1.SanJose1.Level3.net [209.245.146.229]
  9    65 ms    64 ms    64 ms  so-1-2-0.bbr2.SanJose1.Level3.net [209.244.3.141]
 10    96 ms    95 ms    95 ms  so-1-0-0.mp1.Houston1.Level3.net [209.247.11.169]
 11    94 ms    96 ms    94 ms  so-11-0.hsa2.Houston1.Level3.net [4.68.96.54]

 12   428 ms   433 ms   421 ms  unknown.Level3.net [209.247.109.114]

 13   314 ms   102 ms   101 ms  hou-rtr3-vl2.cyrusone.com [69.7.175.19]
 14   108 ms   293 ms   102 ms  vl2-rtr1.nocdirect.com [69.7.160.227]
 15     *      102 ms   100 ms  69.73.166.75

Trace complete.

C:\>

**Ron** · 07-29-2006, 11:47 PM

It's almost like different pieces of the network are getting swamped at different times.

I included 5 traceroutes. 2 & 3 show a bad router at hop 11, 4 shows hop 13 (the same one with which Spathi had a problem), and #5 shows my own server timing out.

Almost like there's a DoS going on that is sequentially overloading pieces of the network, or actually targeting different pieces.

Very strange.

PS Does unacceptably bad response time constitute "network downtime"?

**Vin DSL** · 07-30-2006, 12:05 AM

http://jaguarpc.com/forums/showthread.php?t=14544

**Vin DSL** · 07-30-2006, 12:32 AM

Originally Posted by Ron

I'm running stuff through SSH that's interactive and there is 0 lag...

LoL!

I just uploaded a 123kb file using SSH -- took 9 minutes exactly!

**Vin DSL** · 07-30-2006, 04:33 AM

Man, this sucks!

What's the old saying -- never buy stock that's cheaper than a pizza?

http://www.google.com/finance?q=LVLT

Latest...

Code:

Microsoft Windows 2000 [Version 5.00.2195]
(C) Copyright 1985-2000 Microsoft Corp.

C:\>tracert lenon.com

Tracing route to lenon.com [69.73.166.75]
over a maximum of 30 hops:

  1     1 ms     1 ms   <10 ms  10.0.0.1
  2    44 ms    44 ms    44 ms  lo0-ar1.phx.cybertrails.com
  3    45 ms    45 ms    53 ms  ge-7-0.1000M.gsr1.phx.cybertrails.com
  4    45 ms    45 ms    47 ms  phv-edge-02.inet.qwest.net [72.164.12.9]
  5   110 ms    46 ms    45 ms  phx-core-01.inet.qwest.net [205.171.129.113]
  6    65 ms    64 ms    65 ms  svx-core-02.inet.qwest.net [67.14.1.58]
  7    64 ms    64 ms    65 ms  sjp-brdr-01.inet.qwest.net [205.171.214.138]
  8    64 ms    64 ms    64 ms  so-2-0-0.edge1.SanJose1.Level3.net [209.245.146.229]
  9    70 ms    65 ms    69 ms  so-1-2-0.bbr2.SanJose1.Level3.net [209.244.3.141]
 10    97 ms   118 ms    95 ms  so-1-0-0.mp1.Houston1.Level3.net [209.247.11.169]
 11    95 ms   106 ms    96 ms  so-10-0.hsa2.Houston1.Level3.net [4.68.112.150]

 12   326 ms   439 ms   443 ms  unknown.Level3.net [209.247.109.114]

 13    95 ms    95 ms    96 ms  hou-rtr3-vl2.cyrusone.com [69.7.175.19]
 14   109 ms    95 ms    95 ms  vl2-rtr1.nocdirect.com [69.7.160.227]
 15    95 ms    96 ms   103 ms  69.73.166.75

Trace complete.

C:\>

**Ron** · 07-30-2006, 05:47 AM

This seems to have resolved while I was sitting online surfing my site and this one at about 7:15AM ET.

**Ron** · 07-30-2006, 03:03 PM

Well, it's 5PM now and it's bAAAAAaaaaaaack

**Aybara** · 07-30-2006, 03:39 PM

When can we expect an update on the 'Network Latency' issue that is in the announcement forum?

We have been unable to effectively use our site since prior to midnight, and there have been no sufficient updates since then.

Does anyone even have the slightest clue as to what is going on?

**Spathiphyllum** · 07-30-2006, 04:47 PM

Yep. Overall, things look OK.

"Locally",

Code:

  1     1 ms   <10 ms   <10 ms  firewall [192.168.xxx.xxx]
  2     *        *        *     Request timed out. [modem - ping response free]
  3    10 ms     9 ms     *     GE-1-2-ur01.somewhere3.tx.dallas.comcast.net [68.87.204.5]
  4     9 ms     *       14 ms  te-8-1-ur01.somewhere2.tx.dallas.comcast.net [68.87.207.41]
  5     *       13 ms     *     te-8-3-ar01.somewhere1.tx.dallas.comcast.net [68.87.207.29]
  6    11 ms    11 ms    12 ms  12.118.225.13
  7    13 ms    11 ms    11 ms  tbr1-p012301.dlstx.ip.att.net [12.123.16.98]
  8    10 ms    11 ms    17 ms  gar2-p360.dlstx.ip.att.net [12.123.16.89]
  9    14 ms    11 ms    15 ms  att-gw.dfw.above.net [192.205.32.162]
 10    20 ms    19 ms    16 ms  so-3-1-0.mpr1.iah1.us.above.net [64.125.30.229]
 11    17 ms    16 ms    15 ms  209.133.90.85.available.above.net [209.133.90.85]
 12    78 ms    18 ms    19 ms  hou-rtr3-vl3.cyrusone.com [69.7.175.35]
 13    18 ms    18 ms    18 ms  vl2-rtr1.nocdirect.com [69.7.160.227]
 14    18 ms    19 ms    20 ms  xxxxxxxxxxxxxxx [xxxxxxxxxxxxxxx]

Intermittently crappy response to the forum. Ranging from slower than average to pathetic. Mostly seems more local to me this time, but there's that hint of packet loss/latency, too.

**Vin DSL** · 07-30-2006, 06:05 PM

Originally Posted by Spathiphyllum

Intermittently crappy response to the forum. Ranging from slower than average to pathetic. Mostly seems more local to me this time, but there's that hint of packet loss/latency, too.

Agreed! Same here...

Uploads speeds are hideous!!!

**Ron** · 07-30-2006, 07:06 PM

As I'm typing this, the page hasn't fully... OK, it just finished loading.
About a minute or a minute and a half to load this page on the forum.

My forum is equally affected. This is certainly would have to qualify as "network downtime." Do you think anyone in off-hours tech support has notified the head honchos that a large part of their Houston operation is functionally unavailable?

**KaiserSoze** · 07-30-2006, 07:50 PM

This is embaressingly slow

LinkBack

Thread Tools

Display

Very slow network

The 'Network Latency' Issue

Bookmarks

Bookmarks

Posting Permissions