slow filesystem access on VPS

[thisisit3]

We noticed that filesystem access is very slow on our VPS server, unusually slow.

For example:

# time find . -iname "*.html" |wc -l
16155
real    5m49.248s

we see that searching for about 16.000 html files it takes nearly 6 minutes!

while doing the same thing on a small laptop that i'm running right now:

# time find . -iname "*.html" |wc -l
112603
real    0m36.220s

shows that searching for about 112.000 files takes 36 seconds (searching for 16.000 files takes a few seconds).

anyone knows if Jag has implemented disk access restrictions on their VPS nodes or if the server uses remote networked file systems?

[JPC-Veena]

thisisit3,

All vps servers use local disks/fs on the node, no remote disks used. Please open a support ticket so techs could check what the problem may be and whether it is happening only at a certain time of the day etc.

[thisisit3]

I've opened a new ticket (12719580) for this issue.

Although i don't expect much, support will just say that its normal for a VPS to have such slow disk access so that one VPS doesn't affect the others...

We'll have to add "slow disk access" in the bad side effects of VPS's (along with low CPU power and the rest...)

[Pawel Kowalski]

Man, I wish I knew as much about linux as you do.

Although i don't expect much

Give them a chance, they are usually more than happy to offer help in this area. Let us know what they do say as I'm curious if I have the same issue on my servers.

[thisisit3]

JPC-Veena is very sneaky, he made me open the ticket and then he replied himself by complaining that my VPS limits are set too high and that i'm using too much cpu! (which is not the case, cpu usage goes up when i access the file system so thats an effect of the problem at hand)

he thinks my cpu usage causes my disk access to slow down, while the truth is the exact opposite, since cpu usage goes up only when i try to "find/locale" files in the file system!

he avoided to tell me if Jag is using disk access limits.

whats interesting is that different parts of the file system have different read access speeds, for example:

/usr reads 110352 records in nearly a minute
/home/client reads 66865 records in nearly eight minutes!

# time find /usr -iname "*" |wc -l
110352
real    0m57.752s

# time find /home/client -iname "*" |wc -l
66865
real    8m38.081s

Anyone got any explanations?

[thisisit3]

i think i made a mistake, i believe that Jag VPS nodes are not using any disk access limits, because on some paths like /usr i can have the same performance as on a dedicated (local) server.

So, i should have been more careful and should have looked at the differences between the paths.

I still don't understand why there is such a huge difference when traversing the two paths...

[Ron]

The answer is likely to be that someone else is also using a lot of disk on your VPS' hardware node.

[Ron]

However, the problem might be one of your memory limits as well. If you start to page because of a memory issue while you're performing disk I/O you might be thrashing your disk.

Can you check/have you checked where is your page file is located? If you have an option as to where to place it, try another physical disk. Try a different disk I/O intensive command that may not be using as much memory as find (I don't know if find is memory intensive, just a thought).

Is it possible for you to move your app to a different physical disk and try the I/O there?

When all is said and done and you can't isolate it within your own VPS, the answer becomes somewhat obvious, I'd think, that the problem is with the hardware node -- either an actual hard problem or you've got resource competition going on.

This can be a hard thing to track down in a dedicated environment, it might be impossible for you to figure it out without access to the node level.

GOOD LUCK

[thisisit3]

Ron, a virtuozzo (or OpenVZ) container (commonly known as a VPS ) is setup as a single disk, there are no partitions or real disks to move things around.

So anything that affects disk performance is a limitation or a problem that is inherited from the hardware node itself.

Thats why i wonder if the data are stored on network drives or there is something else going on...

[Ron]

So is /usr outside of your container?
Where is /tmp?

[thisisit3]

Its just one single (virtual) drive with a single partition in it:

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 88G 24G 66G 26% /

basically thats the "container" to the hardware node that runs it.

[Ron]

So that's your answer then, I suppose.

Perhaps it's not an I/O issue at all, it has to do with the amount of work you're trying to accomplish, likely in memory. Perhaps the find in one directory tree isn't particularly memory friendly compared to the other.

Do you get to see what is going on with memory and paging while this find is running? Can you run a utility (such as top or sar) in the background logging to a file while this is executing in both /usr and /home/client (or whatever)?

Or has it already been resolved and I'm just blowing in the wind?

[thisisit3]

No, the problem hasn't been fixed.

Its not a memory related issue, while "find" is running, memory consumption doesn't change.

Also, running other disk related utilities like "du" causes the same slow disk IO problem to appear.

[Ron]

Perhaps the physical array is corrupted/badly fragmented and is requiring on-the-fly correction?

If you have enough space, perhaps you could copy the directory tree then rename the top level and then rename the copy to the old name. Leave the old, renamed tree in place to prevent it from being reused.

Are some directories using some kind of encryption/compression/authentication?

I'm running out of ideas of why only some parts would run slow.
Lemme know if you don't wanna hear any more SWAGs.

[thisisit3]

Unfortunately, i still haven't resolved my problem. Even if i delete the files and recreate them won't solve the slow disk access. The file system is not encrypted and its all in one partition as seen by the VPS.

Here is an example, a simple "du" takes 21 minutes to complete

# time du -s -h .
705M	.

real	21m23.292s
user	0m0.483s
sys	0m3.281s

Any help would be appreciated.