ELK5.5+Filebeat分布式日志系统

k8sntfs4elklogstashjavagolangfilebeatfilebeatrediselklogstashrediselasticsearchkibana

此图是盗用别人的, 但能够很好的展示流程, 所以自己就不画了

一. 日志服务器配置

/mnt

# 进入挂载目录  
cd /mnt
# 创建elk文件夹
mkdir elk

# 下载elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.tar.gz
# 下载kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.1-linux-x86_64.tar.gz
# 下载logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.1.tar.gz

# 分别解压
tar -zxvf elasticsearch-5.5.1.tar.gz
tar -zxvf kibana-5.5.1.tar.gz
tar -zxvf logstash-5.5.1.tar.gz

# 移动到elk目录
mv elasticsearch-5.5.1 elk/elasticsearch
mv kibana-5.5.1 elk/kibana
mv logstash-5.5.1 elk/logstash

Elasticsearch

# 进入目录(`ES_HOME`所在的目录)
cd /mnt/elk/elasticsearch
# 安装X-Pack
bin/elasticsearch-plugin install x-pack

# 编辑配置文件
vi config/elasticsearch.yml
# 头部添加
#################################################
cluster.name: yinnote-elastic
network.host: 127.0.0.1
#################################################

cluster.namenetwork.hosthttp.port9200path.logsES_HOME/logspath.dataES_HOME/data

# 启动(daemon方式)
bin/elasticsearch -d

5.5ShieldX-Pack

# 配置elastic账号的密码
curl -XPUT -u elastic 'localhost:9200/_xpack/security/user/elastic/_password'
-H "Content-Type: application/json" -d '{
   "password" : "123456"
}'

# 配置kibana账号的密码
curl -XPUT -u elastic 'localhost:9200/_xpack/security/user/kibana/_password'
-H "Content-Type: application/json" -d '{
   "password" : "123456"
}'

# 配置elastic账号的密码
curl -XPUT -u elastic 'localhost:9200/_xpack/security/user/logstash_system/_password'
-H "Content-Type: application/json" -d '{
   "password" : "123456"
}'

# 生成密钥
bin/x-pack/syskeygen

# 修改配置文件, 重启
vi config/elasticsearch.yml
# 添加密钥支持
#################################################
xpack.security.audit.enabled: true
#################################################
# 重启Elasticsearch
kill -9 [pid]
bin/elasticsearch -d

Kibana

# 进入目录(KIBABA_HOME所在的目录)
cd /mnt/elk/kibana
# 安装X-Pack
bin/kibana-plugin install x-pack

# 配置文件
vi config
# 头部添加
#################################################
server.host: "0.0.0.0."
elasticsearch.username: "elastic"
elasticsearch.password: "123456"
#################################################

server.port5601server.hostipnginx127.0.0.1elasticsearch.urlhttp://127.0.0.1:9200elasticsearch.usernameelasticsearch.password

# 启动(daemon方式)
bin/kibana -d

Logstash

filebeat

# 进入目录(LOGSTASH_HOME所在的目录)
cd /mnt/elk/logstash
# 创建配置文件
vi client-http.conf
# 添加
#################################################
input {
    beats {
        port => 5044
        codec => "json"
    }
}
output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "logstash-nginx-error-%{+YYYY.MM.dd}"
        user => "elastic"
        password => "123456"
    }
    stdout {codec => rubydebug}
}
#################################################

# 启动服务(监听模式, 便于查看数据)
bin/logstash -e -f client-http.conf

redis

# 创建配置文件
vi client-redis.conf
# 添加
#################################################
input {
    redis {
        host => "127.0.0.1"
        port => "6379"
        key => "filebeat"
        data_type => "list"
        password => "redis的密码"
        threads => 50
    }
}
output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "logstash-nginx-error-%{+YYYY.MM.dd}"
        user => "elastic"
        password => "123456"
    }
    stdout {codec => rubydebug}
}
#################################################

# 启动服务(监听模式, 便于查看数据)
bin/logstash -e -f client-http.conf

二. 客户端数据采集配置

# 下载安装
cd /mnt
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.1-linux-x86_64.tar.gz
tar -zxvf filebeat-5.5.1-linux-x86_64.tar.gz
mv filebeat-5.5.1-linux-x86_64 filebeat
# 进入目录
cd filebeat

RESTfulLogstash

# 创建配置文件
vi client-http.yml
# 添加
#################################################
filebeat.prospectors:

- input_type: log
  paths:
    - /mnt/logs/nginx/error.log
  fields:
    feature: nginx-err

output.logstash:
  hosts: ["yinnote.com:5044"]
#################################################
# 启动服务(监听模式, 便于查看数据)
./filebeat -c client-http.yml

RedisLogstash

# 创建配置文件
vi client-redis.yml
# 添加
#################################################
filebeat.prospectors:

- input_type: log
  paths:
    - /mnt/logs/nginx/error.log
  fields:
    feature: nginx-err

output.redis:
  hosts: ["yinnote.com"]
  port: 6379
  password: "redis的密码"
#################################################
# 启动服务(监听模式, 便于查看数据)
./filebeat -c client-redis.yml

访问后台

1. 浏览器访问  http://yinnote.com:5601

2. 输入elastic用户和密码即可登录

3. 点击左侧的 discover 菜单, 即可查看日志采集情况

Elasticsearch

---