Linux : Modsecurity concurrent log analysis

It’s been a crazy few months with work and family, but following on from the modgrep modsecurity serial log analysis article a few months ago,we’re back with another instalment for modsecurity users who use the concurrent logging format.

Anyone who uses modsecurity will know that concurrent logging creates a lot of files and directories. Essentially, instead of ramming all of mod security’s log events into a single file, it places each event into its own file. This provides for far greater performance in high load scenarios, and avoid the dreaded file locking errors that can cause apache to struggle to complete requests in a timely manner, or not at all.

A typical symptom of a server being too heavily loaded to cope with serial logging any more are messages like this filling up Apache’s error_log.

Enabling concurrent logging is quite straightforward, and requires this directive to be set in modsec2.conf (assuming cPanel Atomic Corp recommended setup).

and the following in modsec2.user.conf

The directory structure for /var/asl can be created using the shell commands highlighted on the Atomic Corp page linked above.

Modsecurity concurrent log analysis

As explained in my previous article, mod security only records limited information in the Apache error log. The real meat of any security event is contained in the mod security audit logs. Concurrent logging creates a directory structure a bit like this:

Under the nobody directory (named after the default user that apache uses to access files) – each day’s logs are stored inside a directory named in the format yyyymmdd – or %Y%m%d for posix fans.

Under each day’s directory there is another layer of directories named for each hour and minute of the day there was an event, named in the format yyyymmdd-hhmm – or %Y%m%d-%H%M in posix strftime format.

Under each hour/minute directory, there are the event log files themselves, which are named with all of the above, plus the number of seconds, plus a unique identifier e.g. the following event log filename was generated on 28th June 2014 at 1 minute and 25 seconds past midnight.

Modgrep reborn as Modgrepc

The original modgrep, intended for serial logs, required a bit of work to deal with concurrent logs, but the result is only slightly slower (mainly because it has to open so many files in order to search for hits). Here is the script – don’t forget to make it executable before trying to run it 🙂

This works pretty much in the same way as the original modgrep script does.

Dealing with those pesky modsecurity concurrent log files when they build up

Concurrent logging, as we have seen, can create an unholy number of files – if you leave a busy server for a few days you are going to find potentially millions of nodes being eaten up on your file system. This needs cleaning up!

The following script can be run from the command line, or placed in /etc/cron.daily (Centos/RH). Essentially, the script serialises the concurrent logs into a log file much like the old serial modsec audit log format.

By default, it will look for any directories under the /var/asl/data/audit/nobody directory that don’t match the current date (i.e. today), and build a sorted list of all event log entries, and then concatenate those files into/usr/local/apache/logs/modsec_serial.log – which you can then rotate using logrotate (Centos/RH) or other tool of choice. It then deletes the old concurrent logs, keeping the node count down.

Don’t be tempted to alter this script to write the output to /usr/local/apache/logs/modsec_audit.log because even in concurrent mode, the old modsec_audot log is still used to record the location of each concurrent audit log file.

I hope you find this handy – and please let me know if you do!

, ,

No comments yet.

Leave a Reply

Time limit is exhausted. Please reload CAPTCHA.