Learn how to - Perform Proper Rsyslog Logs Rotation using logrotate

Felix Williams How to Logging June 6, 2023

Learn how to – Perform Proper Rsyslog Logs Rotation using logrotate.

One of the challenges of managing computer systems is dealing with log files. They can be voluminous and difficult to parse, and sifting through them can be time-consuming and tedious. However, logs are an essential tool for troubleshooting and diagnosing issues that may arise in a system.

When something goes wrong, logs can provide crucial information about what happened, when it happened, and what system components were involved. This information can help system administrators and developers quickly pinpoint the root cause of a problem and take steps to correct it.

Log files are typically used to record events, messages, and other information related to the operation of a system. The size of log files can vary depending on the frequency and verbosity of the logging activity. When logs are updated frequently and with a lot of detail, the log files can quickly become large and unwieldy.

As the size of log files increases, managing and analyzing them becomes more challenging. It can be difficult to find information that is relevant to a specific problem, and the process of searching through large log files can be time-consuming.

However, there are techniques and tools available to help manage log files, such as log rotation and compression. Log rotation involves creating a new log file when the current file reaches a certain size or age so that the log files remain a manageable size. It involves managing log files by creating new log files and removing old ones. Compression techniques can also be used to reduce the size of log files, making them easier to store and analyze.

Log rotation offers several benefits, among them are:

Improved performance: small log files can be read more quickly and efficiently, improving system performance when analyzing log data.
Easier troubleshooting: with log rotation, the most recent logs are separated from the older logs. This can make it easier to identify when an issue started and narrow down the cause of the problem.
Reduced storage requirements: logs can be compressed or removed altogether, reducing storage requirements and freeing up disk space.
Compliance requirements: Most industries have regulatory compliance requirements that mandate the retention of logs for a certain period of time. Log rotation can help ensure that logs are retained in a manageable way that meets these requirements.

In this guide, we will learn how to perform a proper Rsyslog Logs Rotation using logrotate.

Install Rsyslog on Your System

Rsyslog exists in the default package managers on many Linux systems. This application works in conjunction with the systemd-journald service to provide local and remote logging support. The rsyslogddaemon constantly reads all the syslog messages from the systemd-journald service and stores them in rsyslog.

This package can be installed using the below commands:

##On Debian/Ubuntu
sudo apt update && sudo apt install rsyslog

##On RHEL/CentOS/Rocky Linux/Alma Linux
sudo yum install rsyslog

##On OpenSUSE
sudo zypper install rsyslog

Once installed, ensure that Rsyslog is started and enabled:

sudo systemctl start rsyslog
sudo systemctl enable rsyslog

Check the status of the service:

$ systemctl status rsyslog
● rsyslog.service - System Logging Service
     Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-04-26 14:25:29 CEST; 21s ago
       Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
   Main PID: 58333 (rsyslogd)
      Tasks: 3 (limit: 23441)
     Memory: 960.0K
        CPU: 6ms
     CGroup: /system.slice/rsyslog.service
             └─58333 /usr/sbin/rsyslogd -n

Understanding the Rsyslogd Configuration

Understanding the rsyslog.conf will be so important in this guide. The config file is divided into 3 main parts. These are:

Global: used to configure the global properties of the rsyslogd daemon
Modules: since Rsyslog follows a modular design, any modules can be declared here.
Rules: This defines what should be logged and where. This forms the focus of this article.

To view the configuration, open it at /etc/rsyslog.conf. Below is a sample output:

Perform Proper Rsyslog Logs Rotation using logrotate 1

The rules section has the below syntax:

Perform Proper Rsyslog Logs Rotation using logrotate 2

From the above images, we can see the rules section following some syntax. The first part may consist of one or more pairs of Facility.Priority(example mail.none;cron.none) separated by semicolons. The Facility part describes the type of message while the Priority part indicated the severity. The severity can be “debug, notice, warning, crit, err, info, alert, emerg, none” etc.

The priority provided indicates that all the messages of such priority should be logged. Therefore the above line tells Rsyslog to log all the messages with the priority info or higher apart from those that belong to mail, authpriv, and cron services

*.info;mail.none;authpriv.none;cron.none

The rule can also be written as:

*.info;mail,authpriv,cron.none                /var/log/messages

This means that mail, authpriv, and cron are grouped and the priority for all of them is none.

How to Log Custom Messages with Rsyslog

Aside from the messages defined in the default Rsyslog file, you can log any other messages. In this guide, I will show you how to log custom messages such as daemon messages to /var/log/learnhowto.log or any other desired path.

To do that, we can add the rule to rsyslog.conf or create a separate config at /etc/rsyslog.d. Create the config with the command:

sudo vim /etc/rsyslog.d/deamon.conf

In the file, add the below lines:

###RULE##
daemon.*    /var/log/learnhowto.log

This will log all the daemon messages. For the changes to apply, restart the service:

sudo systemctl restart rsyslog

Now we should have the messages stored in the specified path. To check that, restart any random daemons on your system.

sudo systemctl restart NetworkManager
sudo systemctl restart firewalld

View the contents of the log file:

sudo cat /var/log/learnhowto.log

Sample output:

Rsyslog Logs Rotation using logrotate

After collecting the logs, the files grow in size and make it hard for your to view the activities when troubleshooting. To solve this Rsyslog offers the logrotate utility that makes it easy to rotate, compress, remove, and also mail logs. This makes it easier to manage logs for organizations that generate large numbers of log files.

Logrotate runs as a scheduled task daily and reads all the configuration files at /etc/logrotate.conf and /etc/logrotate.d/.

To understand better, we need to look at the /etc/logrotate.conf file:

Perform Proper Rsyslog Logs Rotation using logrotate 4

These are the global settings for logrotate. They apply to all the services that are being logged. In case you want to perform a specific log rotation to a given file, you can add it below there or create a separate config file for it.

For this demonstration, we will see how to configure custom log rotation. Let’s start by viewing the available logrotate configs.

cat /etc/logrotate.d/wtmp

The file has the below lines:

/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    minsize 1M
    rotate 1
}

The above script instructs Logrotate to perform the actions;

/var/log/wtmp: This specifies the path of the wtmp log file.
missingok is used to tell logrotate not to produce an error message if the above path is missing and will continue to execute the other directives in the configuration file.
monthly: This tells logrotate to rotate the log file monthly. This means that a new log file will be created every month and the current log file will be compressed and/or removed as needed.
create 0664 root utmp: This specifies the permissions and ownership of the new log file that is created. In this case, the new log file will have permissions set to 0664 and will be owned by root with the group set to utmp.
minsize 1M: This tells logrotate to rotate the log file when it reaches a minimum size of 1 megabyte. If the log file is smaller than this, it will not be rotated.
rotate 1: This specifies that logrotate should keep only one previous version of the log file. This means that only the previous month’s log file will be kept, and older log files will be removed.

Another example is the /etc/logrotate.d/samba

cat /etc/logrotate.d/samba

It has the below lines:

/var/log/samba/log.* {
    compress
    dateext
    maxage 365
    rotate 99
    notifempty
    olddir /var/log/samba/old
    missingok
    copytruncate
}

Here is what each line means in the configuration file:

/var/log/samba/log.*: This specifies the path of the Samba log files that match the pattern “log.*”.
compress: This tells logrotate to compress the rotated log files using gzip.
dateext: This tells logrotate to use the date as a suffix for the rotated log files.
maxage 365: This specifies the maximum age (in days) of the rotated log files. Log files older than 365 days will be deleted.
rotate 99: This specifies that logrotate should keep up to 99 rotated log files before deleting the oldest log files.
notifempty: This tells logrotate not to rotate the log files if they are empty.
olddir /var/log/samba/old: This specifies the directory where the rotated log files will be moved to. In this case, the rotated log files will be moved to the directory “/var/log/samba/old”.
missingok: This parameter tells logrotate not to produce an error message if a log file is missing when logrotate is executed.
copytruncate: This tells logrotate to copy the current log file and then truncate it, rather than renaming the log file and creating a new one. This is useful for applications that keep a file descriptor open to the log file and cannot be notified to open a new log file.

Another final example is /etc/logrotate.d/rsyslog

$ cat /etc/logrotate.d/rsyslog
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true
    endscript
}

The above lines have the below meaning:

/var/log/cron /var/log/maillog /var/log/messages /var/log/secure /var/log/spooler: This specifies the path of the five log files that this configuration applies to.
missingok: This tells logrotate not to produce an error message if a log file is missing when logrotate is executed.
sharedscripts: This tells logrotate to run the postrotate script only once, after all the log files have been rotated.
postrotate: This specifies the script that should be executed after the log files have been rotated. In this case, the script runs the systemctl kill command to send a HUP signal to the rsyslog.service systemd unit. This tells rsyslog to reopen its log files, so it can start writing to the newly created log files.
/usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true: This is the command that is run after log files are rotated. It sends a HUP signal to the rsyslog.service systemd unit to tell it to reopen its log files. >/dev/null 2>&1 redirects all output to /dev/null to suppress any error messages that may be produced. || true ensures that the command returns a success status even if the rsyslog service is not running.

Create a Custom Rsyslog Logs Rotation Rule

After going through how the logrotate files look, we can create a custom one for the log file we created earlier in the guide.

Create the logrotate file:

sudo vim /etc/logrotate.d/learnhowto

In the file, add the below lines:

/var/log/learnhowto.log {
    missingok
    daily
    rotate 4
    compress
    delaycompress
    notifempty
    create 0644 root root
}

This is a logrotate configuration that applies to the /var/log/learnhowto.log log file, where:

/var/log/learnhowto.log specifies the path of the log file that this configuration applies to.
missingok tells logrotate not to produce an error message if the log file is missing when logrotate is executed.
daily tells logrotate to rotate the log file daily.
rotate 4 tells logrotate to keep up to 4 rotated log files before deleting the oldest one.
compress tells logrotate to compress the rotated log files using gzip.
delaycompress tells logrotate to delay compression of the rotated log file until the next rotation cycle. This is useful because it allows the log file to continue to be written to without being compressed, so applications can write to it without incurring the performance penalty of compressing the log file on every write.
notifempty tells logrotate not to rotate the log file if it is empty.
create 0644 root root tells logrotate to create the log file with the specified permissions if it does not already exist. In this case, the log file is created with permissions of 0644 and owned by the root user and root group.

As of today, we have a log file created, this will be rotated after a day.

$ ls -al /var/log/learnhowto.log
-rw-------. 1 root root 13692 Apr 26 14:28 /var/log/learnhowto.log

With these examples, you should be able to configure a proper Rsyslog Logs Rotation using logrotate.

Saving Rsyslog Logs to a SQL Database

It is also possible to save the logs to a database server such as MySQL, MariaDB, PostgreSQL, and Oracle. To demonstrate this, we will install the MariaDB server using any of the guides below:

Once installed, download the createDB.sql schema:

wget https://raw.githubusercontent.com/sematext/rsyslog/master/plugins/ommysql/createDB.sql

Import the schema to your database server:

mysql -u root -p < createDB.sql
##OR
mariadb  -u root -p < createDB.sql

Now access the shell:

mariadb -u root -p

View the available tables:

use Syslog;
show tables;

Sample Output:

Perform Proper Rsyslog Logs Rotation using logrotate 5

Exit the MySQL shell and modify your rsyslog configuration file:

sudo vim /etc/rsyslog.conf

Add the below lines to the modules section:

#### MODULES ####
module(load="ommysql")
action(type="ommysql" server="localhost" serverport="3306"
       db="Syslog" uid="root" pwd="Your_MySQL-root-PasswordHere")

Ensure that the ommysql Rsyslog module is installed. This can be done using the commands:

##On Debian/Ubuntu
sudo apt install rsyslog-mysql

##On Red Hat/CentOS/Rocky Linux/Alma Linux
sudo yum install rsyslog-mysql

##On Arch
sudo pacman -S rsyslog-mysql

##On OpenSUSE
sudo zypper install rsyslog-module-mysql

Once installed restart the services:

sudo systemctl restart rsyslog 
sudo systemctl restart mariadb

You can also try restarting a few services to generate logs to be viewed.

Querying Rsyslog Logs from the Database

You can now start to view the Logs stored in the DB using the SQL syntax. Access your database:

mariadb -u root -p

View the events with a command with the below syntax:

USE Syslog;
SELECT ReceivedAt, Message FROM SystemEvents;

Sample Output:

Perform Proper Rsyslog Logs Rotation using logrotate 6

Final Thoughts

Today, we have learned a lot about rsyslog. We have seen how the rsyslog and logrotate configs look, learned how to configure both. By now, you should be able to perform a proper Rsyslog Logs Rotation using logrotate. I hope this was informative.

See more:

Cart

Learn how to – Perform Proper Rsyslog Logs Rotation using logrotate