Apache log access

[wgiese]

I've been working with support for the last few weeks, trying to do something one would think would be rather simple. That is, get complete, monthly raw Apache access logs for my domains.

According to support, complete monthly logs are not kept or provided to customers. I've been at numerous ISP's, and this is unheard of - every one of them tar'ed/gzipped them at the end of the month, and made them available (most for 2-3 months.)

The official options suggested are all flawed, as are work-arounds I've tried.

Download via CPanel - This works fine, if all you want are month-to-date logs. It's a little unreasonable to expect customers to login at 11:59PM on the last day of the month to get a complete monthly log, though.

So I asked support about direct shell access to the raw logs - figured I could set up a cron job that tar'ed/gzipped the logs to my home directory on the last day of the month. I was provided with the path to my logs, and thought all was fine - until I realized the path provided would not work, presumably due to permissions or the lock-down in place thanks to jailshell.

So...support creates a logs folder under my home directory, and configures Apache to write them there. Perfect! Or so I thought...then it dawned on me that home directories are subject to quotas, and raw log files can get huge - checked my November log file and it's already at 200MB. Needless to say, having 40% of my disk quota taken up by an access log ain't gonna cut it.

The solution is simple, but Jag seems unwilling or unable to do it; simply tar/gzip access logs at the end of the month, copy them to home directories for folks who want them, then delete the originals as they currently do.

How are the rest of you dealing with this? I can't be the only person here who wants their logs...

Bill

[jason]

I have a hunch that the cron job will run under bash instead of JailShell, so even though you can't access the logs directory through SSH a cron job might still work.

--Jason

[wgiese]

Good idea Jason, thanks. I'll have them move the log location back and give it a shot.

Bill

[jason]

You're welcome. I know it isn't a perfect solution, but...

--Jason

[stevenha]

Bill, your message sounds exactly like what I've been going through with Jag support. I agree with you completely, that a compressed log for the entire month, should be created and automatically stored at the end of the month... and available for download at leisure.

Jag support has made several attempts to create cron jobs for me, for which I'm grateful, but they just haven't worked. Sometimes I get emails saying a cron job didn't have permissions, but mostly, these cron jobs seem to fail, simply because they have low priority for CPU time, and even when they try to run at 3AM, they can't get enough CPU time to complete the job, so the job never completes.

I tried a variation of a cron job that would store the log, without compressing it, and when i realized it was creating a 600MB file size, I had to ask them to stop it.

On a positive note though, Jag support told me that they've heard us, and realized that the current situation isn't working. I've been told they are working on it. I hope so, because log analysis is very important to some people.

[Gwaihir]

Originally posted by stevenha
Jag support told me that they've heard us, and realized that the current situation isn't working. I've been told they are working on it. I hope so, because log analysis is very important to some people.

Hear, hear

I'm joining those waiting in anticipation for this handy feature.

[wgiese]

Originally posted by stevenha
...Jag support told me that they've heard us, and realized that the current situation isn't working. I've been told they are working on it.

'Tis good to hear they are working on it, although they made no such claim to me.

I'm at a loss to understand why this is such a huge task. logrotate has shipped with every version of Linux I've used for years, and can automatically rotate and compress log files, and is pretty simple to configure for Apache and virtual hosts.

It's not rocket science...

[mattsiegman]

You guys don't realize how many customers Jag has. With a large user base, zipping a 200mb-several GB file for each user needs a dedicated server of its own. Compression of data is not a speedy or a computationally simple process.

[Connie]

Back in the old days we could access the logs through the CP ( think that was a
different CP than we have today). The currant situation is one of the only
complaints I have about Jag.

I would really like to download logs for the month. However, I have to say the
other benefits at Jag are more important to me than the log files.

Hopefully Jag will do something about this in the future.

[stevenha]

Matt, you raise a very good point. I see how it would be impossible to do the log rotation, for say, 200 users per server, near midnight on the last day of the month.

Of course, maybe some users don't need this, especially if they'd end up paying extra for disk space, whereas other users might be willing to pay more, and even bid, for premier dates and times to have the log rotation occur.

[Connie]

Stevenha and Matt, you have both raised good points. From my prospective raw
log access was part of the original plains I signed up under. But then at that time so
was a toll free number to call support.

Due to the enhancements Jag has made over the years with their plans
disc space usage would not be a problem with me. However I would be
glad to pay a little more per month as an option to have access to the raw logs again.

Actually I think the problem may be more CP related than a Jag problem.
If my failing memory is correct access to the raw logs was not a problem until Jag
made their first move.

Jason or one of the other old timers at Jag may be able to elaborate on this
more.

[mattsiegman]

Well, the CPanel has two or three log analyzers built in to it. I know there is a way to access your log files, but I'm not sure if they are flushed every month or not. I believe they are in a special directory that you can only reach via SSH.

[Connie]

Originally posted by mattsiegman
Well, the CPanel has two or three log analyzers built in to it. I know there is a way to access your log files, but I'm not sure if they are flushed every month or not. I believe they are in a special directory that you can only reach via SSH.

The built in log analyzers are really very poor. They only provide very general
information.

I don't think there is any way to access the logs by SSH, which may be do to the
jail we are in.

The downloadable logs have to be done through CP. They are flushed at the end
of every month. Currently there is no way to access the raw logs without
loosing some information

I used to use an independentt analyzer. It did not provide all the information I
needed so I quite paying $40.00 per month for the independent analyzer.

I still think this is a CP issue. Before Jag moved us to CP we could access
the logs. That was a long time ago with an entirely different CP. This is an
area that Jag could improve on and this is not the first time this dissussion has come up.

[mattsiegman]

I remember the old CP... But I also remember support telling me a path to access logs... I made a symlink in my home directory Unfortunately, I deleted it some time ago... and I can't find a new path because jailshell hides directories you don't have permission to see... (normally 'whereis apache' gets the log directory too, but it doesn't show anything right now)

[Connie]

Originally posted by mattsiegman
I remember the old CP... But I also remember support telling me a path to access logs... I made a symlink in my home directory Unfortunately, I deleted it some time ago... and I can't find a new path because jailshell hides directories you don't have permission to see... (normally 'whereis apache' gets the log directory too, but it doesn't show anything right now)

I knew you were around in those days. I didn't have to make any symlink but support did have to tell me how to access the logs. In the old days the logs were not flushed at the end of the month. They were contentious. I don't think they ever were flushed.

I guess the answer is to download the logs daily. Of course on the last day of the month you are going to loose a few stats under the currant system.