Learn how to - VMware vSphere and vCenter Central Logs Management with Rsyslog

Learn how to – VMware vSphere and vCenter Central Logs Management with Rsyslog.

For the past one week, I’ve been working on a centralized Logs management system for VMware vSphere and vCenter environment. Having searched on the web for a simple, better open-source solution for this, the ideal solution seemed to be Rsyslog collector.

So I ended up installing Rsyslog server on CentOS 7 system and configuring it as a central logs server for vCenter Server Appliance(vCSA) logs and all logs generated by vSphere servers. In this guide, we will cover both installation/configuration of Rsyslog server and configuration of vCSA/vSphere to send logs to remote Rsysog server we configured.

Setting up vSphere and vCenter Central Logs Management with Rsyslog

Before we get started, let me state the assumptions here:

You have installed CentOS 7 server; Debian based system should be fine
Rsyslog is installed
Your user account has sudo privileges

If any of the above is not satisfied, make sure you have all set. When done, use steps below to install and configure Rsyslog server to receive logs from your VMware vSphere and vCenter infrastructure.

Step 1: Update your system

Let’s update our system packages to the latest releases:

sudo yum update

For Ubuntu, do the same with apt:

sudo apt-get update && sudo apt-get upgrade

Reboot the system if possible for updates which require an update to take effect, e.g kernel updates:

sudo reboot

Step 2: Configure the Rsyslog server

Both CentOS and Ubuntu/Debian systems come with rsyslog installed and running. We will need to create an additional configuration file for our VMware setup.

For basic configuration of Rsyslog on Ubuntu/Debian, refer to How to Configure Rsyslog Centralized Log Server on Ubuntu 18.04 LTS

The default configuration file is./etc/rsyslog.conf. Any additional configuration can be placed under the directory/etc/rsyslog.d/.

Create a directory for VMware vSphere and vCSA appliance logs:

$ sudo mkdir -p /data/logs/{vcenter,esxi}

Then add the following configuration file to /etc/rsyslog.d/vmware.conf.This is a config for vSphere Infrastructure logging. It has templates which dictate where and how to store logs, match patterns, and log timestamp definition.

# Provides TCP syslog reception
$ModLoad imtcp

#### Create Templates for Log parsing
#### Log store is created folder previously

template(name="TIMESTAMP" type="string" string="%timegenerated:8:25%||%HOSTNAME%||%syslogtag%||%msg%n")
template(name="ESXI" type="string" string="/data/logs/esxi/%fromhost-ip%/%$YEAR%-%$MONTH%/%$DAY%.%$MONTH%-messages.log")
template(name="VCENTER" type="string" string="/data/logs/vcenter/%fromhost-ip%/%$YEAR%-%$MONTH%/%$DAY%.%$MONTH%-messages.log")

#### Define remote logging RuleSet
$RuleSet remote

### The following rule is created to store logs coming from vCenter with different parameters
### If log is coming from vcenter with IP 192.168.10.50 it will be stored using VCENTER template.
### Replace 192.168.10.50 with your vCSA IP address

if $fromhost == '192.168.10.50' then {
*.* ?VCENTER
}
else {
# Store all other logs using TIMESTAMP and ESXI templates.
*.* ?ESXI;TIMESTAMP
}

### Tell rsyslog to listen TCP port 514.
### All events coming to this port will be parsed using remote RuleSet.

$InputTCPServerBindRuleset remote
$InputTCPServerRun 514

Here we defined three templates:

TIMESTAMP: Used to format all logs coming from certain IP Address. This is to make logs more readable by having proper time formats.
ESXI: This tells rsyslog to store logs from each host in a separate folder and logs from each day be stored in a folder corresponding to that day. Logs from vSphere hosts will be stored in /data/logs/esxi
VCENTER: This tells rsyslog to store logs from a vCenter appliance in /data/logs/vcenterusing timestamp and structure similar to the one used on ESXI template.

Restart rsyslog service after making the changes:

sudo systemctl restart rsyslog

Check status, it should be in running state:

$ systemctl status rsyslog
● rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2018-08-02 11:54:03 EAT; 1min ago
Docs: man:rsyslogd(8)
Main PID: 12489 (rsyslogd)
Tasks: 8
Memory: 7.6M
CGroup: /system.slice/rsyslog.service
└─12489 /usr/sbin/rsyslogd -n

Aug 02 11:54:03 rsyslog.example.com systemd[1]: Starting System Logging Service...
Aug 02 11:54:03 rsyslog.example.com rsyslogd[12489]: [origin software="rsyslogd" swVersion="8.24.0" x-pid="12489" x-info="http://www.r...] start
Aug 02 11:54:03 rsyslog.example.com systemd[1]: Started System Logging Service.
Hint: Some lines were ellipsized, use -l to show in full.

Open Rsyslog firewall ports

Rsyslog listens on port 514, we configured it to user TCP, open it using your firewalld administration tool.

On Ubuntu / Debian with ufw:

sudo ufw allow 514/tcp

On CentOS 7:

sudo firewall-cmd --add-port=514/tcp --permanent
sudo firewall-cmd --reload

Step 3: Configure vSphere hosts and vCSA Appliance

The only pending piece is to configure vSphere and vCSA appliance to push logs to our rsyslog server.

How to Configure vSphere remote Syslog

Open your vCenter and navigate to:

Hosts and Clusters > Select Host > Configure > Advanced System Settings > Edit

on Filter box, search “syslog”. You should see “Syslog.global.logHost“. Fill rsyslog server address inside this box:

tcp://192.168.10.35:514

If you want to configure both tcp and udp, separate them using a comma:

tcp://192.168.10.35:514,udp://192.168.10.35:514

It should look like below:

VMware vSphere and vCenter Central Logs Management with Rsyslog 1

How to Configure vCenter (vCSA) remote Syslog

For vCSA, configure remote rsyslog by opening vCSA administration UI:

http://vCSA-IP:5480

This will give login dashboard:

VMware vSphere and vCenter Central Logs Management with Rsyslog 2

Enter username and password provided during installation. The initial default username is root, and the default password is vmware.

Procedure:

In the vCenter Server Appliance Management Interface, select Syslog.
In the Forwarding Configuration section, click Configure if you have not configured any remote syslog hosts. Click Edit if you already have configured hosts.
In the Create Forwarding Configuration page, enter the server address of the destination host. The maximum number of supported destination hosts is three.
From the Protocol drop-down menu, select the protocol to use.
In the Port text box, enter the port number to use for communication with the destination host.
In the Create Forwarding Configuration pane, click Add to enter another remote syslog server.
Click Save.
Verify that the remote syslog server is receiving messages.
In the Forwarding Configuration section, click Send Test Message.
Verify on the remote syslog server that the test message was received.

The new configuration settings are shown in the Forwarding Configuration section.

If you go back to the rsyslog server, there should be two folders created under logs directory configured earlier:

$ ls /data/logs/
esx vcenter
$ ls /data/logs/vcenter/192.168.10.50/2018-08/02.08-messages.log 
/data/logs/vcenter/192.168.10.50/2018-08/02.08-messages.log

And for ESXI hosts:

$ file /data/logs/esx/192.168.10.31/2018-08/02.08-messages.log 
/data/logs/esx/192.168.10.31/2018-08/02.08-messages.log: ASCII text, with very long lines

You can consider configuring log rotation for the same.

$ cat  /etc/logrotate.d/vmware
/data/logs/esxi/*/*/*.log {
rotate 90
daily
notifempty
compress
}

/data/logs/vcenter/*/*/*.log {
rotate 90
daily
notifempty
compress
}

You now have a working rsyslog server for your vSphere Infrastructure. Have a happy logging.

VMware Learning Materials:

Also, read:

Cart

Learn how to – VMware vSphere and vCenter Central Logs Management with Rsyslog