使用Rsyslog将Nginx Access Log写入Kafka

发表于 更新于 分类于 ， 阅读次数： Changyan：

环境说明

  • CentOS Linux release 7.3.1611
  • kafka_2.12-0.10.2.2
  • nginx/1.12.2
  • rsyslog-8.24.0-34.el7.x86_64.rpm

创建测试Topic

1
$ ./kafka-topics.sh --zookeeper 192.168.72.25:2181/kafka --create --topic develop-test-topic --partitions 10 --replication-factor 3

RSyslog安装

一般系统自带RSyslog服务无需另外安装。但是因为数据需要通过RSyslog的omkafka模块写入到Kafka，而omkafka在RSyslog的v8.7.0+版本才支持，所以要看当前系统中RSyslog的版本是否需要升级。使用以下命令查看：

1
2
3
4
5
6
7
8
9
10
11
12
# rsyslogd -v
rsyslogd 7.4.7, compiled with:
	FEATURE_REGEXP:				Yes
	FEATURE_LARGEFILE:			No
	GSSAPI Kerberos 5 support:		Yes
	FEATURE_DEBUG (debug build, slow code):	No
	32bit Atomic operations supported:	Yes
	64bit Atomic operations supported:	Yes
	Runtime Instrumentation (slow code):	No
	uuid support:				Yes

See http://www.rsyslog.com for more information.

执行以下命令安装：

1
# sudo yum install rsyslog

安装依赖关系如下：

添加omkafka模块

1
# yum install rsyslog-kafka

RSyslog Client Nginx配置

注意，Nginx 1.7.1之后才支持syslog的方式处理日志。具体配置项参见官网文档Logging to syslog。

Nginx配置主要是日志格式和Access Log配置项：

1
2
3
4
5
6
7
8
9
10
11
12
log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

server {
    listen 11000;

    location / {
        proxy_pass http://10.16.0.144:11000;
        access_log syslog:server=localhost,facility=local7,tag=nginx11000Root,severity=info main;
    }
}

RSyslog Server端配置

RSyslog的主配置文件/etc/rsyslog.conf，其中会包含引入/etc/rsyslog.d下扩展名为conf的配置文件。

修改配置文件/etc/rsyslog.conf将下面两行前面的注释去掉：

1
2
$ ModLoad imudp
$ UDPServerRun 514

在/etc/rsyslog.d目录下创建rsyslog_nginx_kafka_cluster.conf，配置内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
module(load="imudp")
input(type="imudp" port="514")

# nginx access log ==> rsyslog server(local) ==> kafka
module(load="omkafka")

template(name="nginx-11000-root" type="string" string="%msg%")

if $inputname == "imudp" then {
    if ($programname == "nginx11000Root") then
        action(type="omkafka"
            template="nginx-11000-root"
            broker=["192.168.72.10:9092","192.168.72.20:9092","192.168.72.25:9092","192.168.72.26:9092","192.168.72.27:9092","192.168.72.48:9092","192.168.72.55:9092","192.168.72.80:9092","192.168.72.81:9092","192.168.72.97:9092"]
            topic="develop-test-topic"
            partitions.auto="on"
            confParam=[
                "socket.keepalive.enable=true"
            ]
        )
}

:rawmsg, contains, "nginx11000Root" ~

联调测试

启动RSyslog服务：

1
2
# service rsyslog start
Redirecting to /bin/systemctl start  rsyslog.service

遇到的问题

syslog tag 只能包含字母和数字

1
2
3
# nginx -t
nginx: [emerg] syslog "tag" only allows alphanumeric characters and underscore in     /etc/nginx/conf.d/jx-11000-jenkins149-36-144.conf:7
nginx: configuration file /etc/nginx/nginx.conf test failed

‘omkafka’ is unknown

RSyslog中没有包含omkafka模块，需要另外安装。查看/var/log/messages日志信息会有以下提示：

1
2
3
4
5
6
# tail -f messages
Mar 15 15:13:40 192-168-72-29 systemd: Started System Logging Service.
Mar 15 15:13:40 192-168-72-29 rsyslogd: could not load module '/usr/lib64/rsyslog/omkafka.so', dlopen:     /usr/lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory  [v8.24.0-34.el7 try     http://www.rsyslog.com/e/2066 ]
Mar 15 15:13:40 192-168-72-29 rsyslogd: could not load module '/usr/lib64/rsyslog/omkafka.so', dlopen:     /usr/lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory  [v8.24.0-34.el7 try     http://www.rsyslog.com/e/2066 ]
Mar 15 15:13:40 192-168-72-29 rsyslogd: module name 'omkafka' is unknown [v8.24.0-34.el7 try     http://www.rsyslog.com/e/2209 ]
Mar 15 15:13:40 192-168-72-29 rsyslogd: module name 'omkafka' is unknown [v8.24.0-34.el7 try     http://www.rsyslog.com/e/2209 ]

CentOS 6.5升级Rsyslog

CentOS 6.5自带的RSyslog版本是rsyslogd 5.8.10。按照以下方式安装新版本：

1
2
3
# cd /etc/yum.repos.d/
# wget http://rpms.adiscon.com/v8-stable/rsyslog.repo
# yum install rsyslog