如何將Rsyslog日誌轉發到Logstash

最新要做一個對Linux系統日誌採集的需求，當然除了Linux的系統日誌採集外，還需要轉發Tomcat日誌，或者Nginx日誌等。所以就使用了rsyslog這個比較常用並且功能比較強大的工具。

版本：

• Rsyslog V5
• Logstash 5.2.2

配置文件

就不做過多的介紹了直接貼測試通過的rsyslog.conf配置文件該配置文件的目錄為：/etc/rsyslog.conf

# rsyslog v5 configuration file# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html#### MODULES ####$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)$ModLoad imklog   # provides kernel logging support (previously done by rklogd)#$ModLoad immark  # provides --MARK-- message capability# Provides UDP syslog reception#$ModLoad imudp#$UDPServerRun 514# Provides TCP syslog reception#$ModLoad imtcp#$InputTCPServerRun 514#### GLOBAL DIRECTIVES ##### Use default timestamp format$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat# File syncing capability is disabled by default. This feature is usually not required,# not useful and an extreme performance hit#$ActionFileEnableSync on# Include all config files in /etc/rsyslog.d/$IncludeConfig /etc/rsyslog.d/*.conf#### RULES ##### Log all kernel messages to the console.# Logging much else clutters up the screen.(內核)kern.*                                                 /dev/console# Log anything (except mail) of level info or higher.# Don't log private authentication messages!(記錄的內核消息、各種服務的公共消息，報錯信息等)*.info;mail.none;authpriv.none;cron.none                /var/log/messages# The authpriv file has restricted access.(包含驗證和授權方面信息)authpriv.*                                              /var/log/secure# Log all the mail messages in one place.(包含來着系統運行電子郵件服務器的日誌信息)mail.*                                                  -/var/log/maillog# Log cron stuff(每當cron進程開始一個工作時，就會將相關信息記錄在這個文件中)cron.*                                                  /var/log/cron# Everybody gets emergency messages*.emerg                                                 *# Save news errors of level crit and higher in a special file.uucp,news.crit                                          /var/log/spooler# Save boot messages also to boot.log(自定義的消息)local7.*                                                /var/log/boot.log$ModLoad imfile #裝載imfile模塊$InputFileName /opt/tomcat/apache-tomcat-8.5.15/logs/catalina.out #讀取日誌文件$InputFileTag catalina: #日誌寫入日誌附加標籤字符串$InputFileFacility local5 #日誌類型$InputFileSeverity info #日誌等級$InputFileStateFile ssologs.log_state #定義記錄偏移量數據文件名$InputFilePollInterval 1 #檢查日誌文件間隔（秒）$InputFilePersistStateInterval 1 #回寫偏移量數據到文件間隔時間（秒）$InputRunFileMonitor #激活讀取，可以設置多組日誌讀取，每組結束時設置本參數。以示生效。$InputFileName /opt/tomcat/apache-tomcat-8.5.15/logs/localhost_access_log.%$year%-%$month%-%$day%.txt #讀取日誌文件$InputFileTag access: #日誌寫入日誌附加標籤字符串$InputFileFacility local6 #日誌類型$InputFileSeverity info #日誌等級$InputFileStateFile sssologs.log_state #定義記錄偏移量數據文件名$InputFilePollInterval 1 #檢查日誌文件間隔（秒）$InputFilePersistStateInterval 1 #回寫偏移量數據到文件間隔時間（秒）$InputRunFileMonitor #激活讀取，可以設置多組日誌讀取，每組結束時設置本參數。以示生效。# ### begin forwarding rule #### The statement between the begin ... end define a SINGLE forwarding# rule. They belong together, do NOT split them. If you create multiple# forwarding rules, duplicate the whole block!# Remote Logging (we use TCP for reliable delivery)## An on-disk queue is created for this action. If the remote host is# down, messages are spooled to disk and sent when it is up again.#$WorkDirectory /var/lib/rsyslog # where to place spool files#$ActionQueueFileName fwdRule1 # unique name prefix for spool files#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown#$ActionQueueType LinkedList   # run asynchronously#$ActionResumeRetryCount -1    # infinite retries if host is down# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional*.* @10.255.0.167:514# ### end of the forwarding rule #### A template to for higher precision timestamps + severity logging$template SpiceTmpl,"%TIMESTAMP%.%TIMESTAMP:::date-subseconds% %syslogtag% %syslogseverity-text%:%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n":programname, startswith, "spice-vdagent"   /var/log/spice-vdagent.log;SpiceTmpl

上面文件中：

#*.* @remote-host:514 *.*即表示轉發所有設備的日誌信息 @表示使用UDP協議傳輸 @@表示使用TCP協議傳輸 找到上面這句去掉前面的#號然後添加對應的IP和端口即可。 例： *.* @10.255.0.165:514

如果你只想要轉發服務器上的指定設備的日誌消息，比如說內核設備，那麼你可以在rsyslog配置文件中使用以下聲明。 kern.* @10.255.0.165:514

修改完成後執行service rsyslog restart 重新啟動rsyslong 即可。

Logstash配置

input {    udp {        port => 514        type => syslog    }}filter {    if [type] == "syslog" {            grok {              patterns_dir => "/opt/logstash/logstash-5.2.2/patterns"              match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }            }    }}output{    elasticsearch {         hosts => ["10.255.0.167"]        index => "rsyslog_test"    }    stdout{        codec => rubydebug    }}

最後更新配置：/etc/init.d/rsyslog restart 附：Logstash的配置很簡單只是監聽514端口就可以了，但是使用grok切日誌才是麻煩的，畢竟那麼多種的日誌每種都要寫對應的正則。