Jaguar Slows My Site for Using Web Services

[akreider2]

I started using webservices recently (SOAP), so I could split my software into a server and a client. It's all very web 2.0 =)

When I have the server and client installed on Jaguar the client is very slow.

When I have the server installed on Jaguar, and the client on my localhost the client is fast.

My other pages that don't use webservices on Jaguar are all operating at normal speed. So my php libraries aren't the problem.

By slow, I mean pages are taking 15 seconds to 20 minutes (for the client located on Jaguar), when they should be taking 3-5 seconds (as they do locally). They often time out. These are simple pages.

My server load is consistently low (1-2).

The only changing variable is whether the webservices are being access by my localhost, or by the Jaguar server. So there is a different IP.

My hypothesis is that Jaguar is penalizing me for having lots of requests coming from the same IP address (eg. I have lots of requests coming from Wolverine going to Wolverine). Thus it might look like some kind of Denial of Service attack, and the defense is to add a delay to requests. Really though it isn't that big, perhaps 4000 hits at the most per day from the same IP. These 4000 hits/day are from my regular site traffic.

(Googlebot regularly exceeds that level of traffic. Though maybe Googlebot gets away with it by using multiple IP addresses...)

On the other hand, traffic from my localhost isn't penalized, because I'm only doing several hundred requests per day. So my theory is that my localhost isn't triggering the DOS filter.


I filed a support ticket, but they haven't made any progress on this. They have twice hinted that my hypothesis might be correct.

[Vin DSL]

Which server are you on? I'm experiencing the same problem. Matter of fact, it seems like many ppl in Atlanta are -- but some are not...

Ron has an interesting theory. He thinks DNS lookups to external URIs are causing the problem -- and I think he might be right.

Since you're using SOAP (so am I) I assume you're sucking data from external sources. Ronny says using IPs instead on domain names will speed things up.

I haven't had a chance to try this yet, so...

This makes some sense, however, since not everyone is having this problem.

[akreider2]

I'm on Wolverine.

I'm running the webservices off of wolverine. Eg I'm pulling data from my mysql databases stored on wolverine. It's an internal (aka local) source.

(I'm using webservices, so other websites can access the data.)

I'm using NuSOAP (a SOAP library for PHP). It makes some references to other websites in its namespace:

var $namespaces = array(
'SOAP-ENV' => 'http://schemas.xmlsoap.org/soap/envelope/',
'xsd' => 'http://www.w3.org/2001/XMLSchema',
'xsi' => 'http://www.w3.org/2001/XMLSchema-instance',
'SOAP-ENC' => 'http://schemas.xmlsoap.org/soap/encoding/'
);


So are you saying that the namespaces might be trouble?

The NuSOAP library is 7000 lines, so I haven't closely examined it to see how it works.

[JonathanB]

Ron has an interesting theory. He thinks DNS lookups to external URIs are causing the problem -- and I think he might be right.

I've been experiencing DNS problems from here too. Probably for about the past several weeks. In fact, I was just trying to load a page from the forum, and I timed out.

I just did a DNS timing check for jaguarpc.com (from http://www.dnsstuff.com/) and I saw this:

Looking up at ns.jaguarpc.net.... Reports 1 A record(s). 51ms.
Looking up at ns2.jaguarpc.net.... Timed out.

My website (on centaur) uses a different name server. Just checked that and got the same results.

Looking up at ns.nocdirect.com.... Reports 1 A record(s). 310ms.
Looking up at ns2.nocdirect.com.... Timed out.

Something is definitely wrong.

[Vin DSL]

I've been experiencing DNS problems from here too. Probably for about the past several weeks. In fact, I was just trying to load a page from the forum, and I timed out...

Yeah, the JagPC Forums just timed out on me about 10 times (doing a search for new posts). Weird!

[Ron]

Yup. Definitely wrong.

Ever since the move of zeus to manticore I've been having DNS problems; I thought they were related to the move though. I thought they were related to bad/old entries in the old(for me) shared server DNS system. Well, there were some old entries, and the wrong DNS server(s) wer being pointed to and other stuff, but I think I may have been seeing this problem as well.

[akreider2]

Supports says they fixed the problem. So far my site is doing better.

[Vin DSL]

Supports says they fixed the problem. So far my site is doing better.

So much for the punishment theory, eh?

[JonathanB]

My site and Jaguar's are loading fine for me, but just for kicks I did another DNS timing check.

Looking up at ns.jaguarpc.net.... Timed out.
Looking up at ns2.jaguarpc.net.... Reports 1 A record(s). 47ms.

Looking up at ns.nocdirect.com.... Timed out.
Looking up at ns2.nocdirect.com.... Reports 1 A record(s). 46ms.

I don't know a whole lot about how the name-server system works, but wouldn't these problems affect all of the sites hosted at JaguarPC?

Should I open a ticket for this, or since it's a site wide issue, do the folks here know about it?

[Vin DSL]

Heh! Check it out...

Oops... BBL

[JonathanB]

I ran that DNS timing check a few times today, and each time I'm getting good results -- much better that yesterday.

[Gwaihir]

So much for the punishment theory, eh?

Well, it doesn't invalidate the theory. It is still quite possible that they have a rate limiter in use that limits the number of requests that are handled from the same IP, either per hour, or simultaneously, or both. It is in fact sort of netiquette for a client never to place more than two requests per server simultaneously. Most browsers respect this and will, for example, not request the third image on a page before either the first or the second has fully arrived.

Of course, the localhost / the machines own IPs should not be hindered. They should be on the (probably very short) white list for this protection. Perhaps adding them to that list is exactly what they did to fix it?

[JonathanB]

Here we go again...

Looking up at ns.jaguarpc.net.... Timed out.
Looking up at ns2.jaguarpc.net.... Reports 1 A record(s). 48ms.

Looking up at ns.nocdirect.com.... Timed out.
Looking up at ns2.nocdirect.com.... Reports 1 A record(s). 45ms.

Time for me to open a support ticket.

[Vin DSL]

Here we go again...

Yep!

I was going to report that my site got a 'A+' rating earlier today (Noonish local time) -- but now it's a 'D', with 'ns.nocdirect.com' consistently timing out...

Weird!

Guess I should try flipping the nameservers around at my ISP...

[JonathanB]

Ticket opened.

I just did a tracert, but I don't know enough about that to see if anything can be seen wrong there. Any ideas folks?

Tracing route to jaguarpc.com [69.73.138.32]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.0.1
2 7 ms 7 ms 7 ms 76.210.69.254
3 7 ms 7 ms 7 ms dist2-vlan60.mdsnwi.ameritech.net [65.42.115.243
]
4 7 ms 7 ms 7 ms bb1-g8-0.mdsnwi.ameritech.net [65.42.115.115]
5 11 ms 11 ms 10 ms bb1-p11-1.chcgil.ameritech.net [151.164.190.70]

6 12 ms 11 ms 10 ms bb2-p5-1.chcgil.ameritech.net [151.164.191.182]

7 12 ms 12 ms 11 ms ex1-p0-0.eqchil.sbcglobal.net [151.164.42.147]
8 12 ms 12 ms 12 ms asn3561-savvis.eqchil.sbcglobal.net [151.164.89.
182]
9 12 ms 13 ms 12 ms bpr1-ae2.ChicagoEquinix.savvis.net [208.175.9.13
4]
10 12 ms 12 ms 12 ms dcr2-so-4-3-0.Chicago.savvis.net [208.175.10.237
]
11 28 ms 59 ms 29 ms bcs1-so-3-0-0.Washington.savvis.net [204.70.192.
25]
12 48 ms 49 ms 48 ms dcr1-so-3-0-0.Atlanta.savvis.net [204.70.192.53]

13 48 ms 48 ms 50 ms dcr2-as0-0.Atlanta.savvis.net [204.70.192.42]
14 49 ms 52 ms 48 ms bpr2-so-0-0-0.AtlantaPaix.savvis.net [208.172.75
.238]
15 49 ms 49 ms 49 ms 65.254.43.35
16 49 ms 50 ms 48 ms 209.51.131.90
17 49 ms 49 ms 50 ms goliath.nocdirect.com [69.73.138.32]