Logs and logrotate - important fix

[thisisit3]

Another problem with cPanel servers is the log files and logrotate. By default, logrotate looks for the wrong log files in the wrong place (in addition, it can't even send a signal to apache because its looking at the wrong PID file).

As a result, you'll notice your "/etc/httpd/logs/suexec_log" growing to several gigs (after several months on a busy site). Once suexec_log reaches 2GB then your entire server will not be able to execute any CGI scripts (due to a size limit in the libraries). Here is how to make logrotate work with Jag's specific server configuration:

STEP 1: Edit your "/etc/logrotate.d/httpd" file and remove everything, then copy/paste this in it:

/etc/httpd/logs/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
}

STEP 2: If you are using the jag-specific custom BFD rules i've created (see my signature), then you may also create the file "/etc/logrotate.d/bfd" and add this in it: (make sure the bfd_log if the correct log file for BFD, for example i've renamed it as bfd.log in the BFD config file)

/var/log/bfd_log {
    missingok
    postrotate
    endscript
}

STEP 3: SSL-enabled domains create {domain address}-ssl_data_log files under /etc/httpd/domlogs/ which are useless SSL data logs. Unfortunately these logs are never rotated. In addition, they aren't used for any statistic purposes and they serve no real purpose.

Since these files aren't used for anything, you may delete them entirely and
remove the SSLLogFile entry from the apache template. Here is how to do it:

Edit your: /usr/local/cpanel/etc/httptemplates/apache1/ssldefault
Find the line: SSLLogFile %logdir%/%logname%-ssl_data_log
and remove it!

That should take care of all future domains, now all you need to do is edit
your /etc/httpd/conf/httpd.conf and remove all SSLLogFile lines from there.
Then restart your apache and remove all ssl log files with "rm -rf
/etc/httpd/domlogs/*ssl_data_log

Remember that a cPanel/WHM automatic update may overwrite your "ssldefault" template and revert the changes. There is already a bug report in cPanel's bugzilla to fix this.


STEP 4: Clean your exim statistics. These are located in a mysql database called "eximstats" which corresponds to the /var/log/exim* log files. Its a good idea to clean the database once it becomes too big, which in practice is good only for high-volume servers.

First check the size of your eximstats database:
du -s -h /var/lib/mysql/eximstats

If you want to clean it then connect to mysql as root and run:
use eximstats;
delete from sends;
delte from smtp;

[matzah]

Thanks for all of the helpful posts... any thoughts on these files getting too big?
access_log
error_log
ssl_engine_log
or any sort of auto cleaning?

[thisisit3]

Thanks for all of the helpful posts... any thoughts on these files getting too big?
access_log
error_log
ssl_engine_log
or any sort of auto cleaning?

Please be more specific, since there are multiple versions of those files, scattered around the server. For example, cPanel/WHM have their own access_log, while apache has a catch-all access_log and a unique one for every virtual domain.

The above change will solve the sizes of all /etc/httpd/logs/*log files, which covers the most important main logs.

Unfortunately there is still a problem with ssl_data_log files, which aren't rotated at all. I'll post a patch for that soon.

[matzah]

Sorry for the delay, been busy

In my /etc/httpd/logs I have a couple of files that are ****_log.1 and they seem to all have yesterday's date on them:
access_log
access_log.1
error_log
error_log.1
ssl_engine_log
ssl_engine_log.1

I am assuming the ".1" implies the last file... the error_log.1 is rather large, but may be I am missing the point.

I will admit I am no expert coder and I try and hack as little as possible (to not break things)... so thats why I am asking for info and help here.

Thanks!

[thisisit3]

They are all normal! Thats how its supposed to work, since before my changes the rotation of the logs wasn't working.

The .1 denotes the log file from the previous rotation, by default, once a week on my system. Some people have it set for once a day. You may change it from /etc/logrotate.conf:

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 1

I've also set my backlogs to 1, since i dont care to keep 4 week old backups.

[thisisit3]

Added a solution for the stale (and ever growing) SSL data logs, which are pretty useless.

[matzah]

Added a solution for the stale (and ever growing) SSL data logs, which are pretty useless.

Where can in find this info on the SSL cleanups? Thanks... I love all this help you are providing!

[thisisit3]

From the cpanel bugzilla report:

this directive (SSLLogFile) was renamed to SSLLog beginning with mod_ssl 2.1 (~1999). While it exists to handle logging for SSL specific problems, the manual also states:

"Error type messages are additionally duplicated to the general Apache error
log file (directive ErrorLog)."

Beginning with Apache 2, all logging is handled by the internal handlers,
making such directives as SSLLog and SSLLogFile obsolete.

Since:

1. For Apache 1.3.x errors are also logged to the standard log; and
2. Apache 2+ no longer uses this

[thisisit3]

Remember that a cPanel/WHM automatic update may overwrite your "ssldefault" template and revert the changes. There is already a bug report in cPanel's bugzilla to fix this.

[thisisit3]

Added the solution about the exim stats database.

[secondv]

Sorry to bump an old thread.. but..

STEP 1: Edit your "/etc/logrotate.d/httpd" file and remove everything, then copy/paste this in it:

What if the file doesn't exist? Should I create it? I don't have the httpd file, but I do have the one for BFD.

[thisisit3]

My instructions are sometimes based on a cPanel installation, if you are using another control panel like DirectAdmin then just ignore the parts that don't apply to you.

If you don't have an "/etc/logrotate.d/httpd" then don't create it, just do the rest only.

The specific 'httpd' file is useful for those running cPanel, because by default cPanel doesn't rotate all the log files and as a result, wastes lots of GIGs in old logs. The above change will make sure you don't have any stale logs. If you don't use cPanel then you probably don't need it.

[secondv]

Ah Ok, yeah I'm using DA.