1. ELK Stack概述
ELK Stack是Elasticsearch、Logstash和Kibana的组合，用于日志收集、存储、分析和可视化。它是一种强大的日志管理解决方案，可以帮助企业实时分析和监控系统日志，提高故障排查效率。更多学习教程www.fgedu.net.cn
# uname -a
Linux server 3.10.0-1160.el7.x86_64 #1 SMP Wed Aug 26 15:27:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
# 检查Java版本
# java -version
openjdk version “1.8.0_292”
OpenJDK Runtime Environment (build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)
2. Elasticsearch安装
Elasticsearch是一种分布式搜索引擎，用于存储和索引日志数据。学习交流加群风哥微信: itpux-com
# 导入GPG密钥
# rpm –import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# 添加Elasticsearch仓库
# cat > /etc/yum.repos.d/elasticsearch.repo << EOF
[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
# 安装Elasticsearch
# yum install -y elasticsearch
# 启动Elasticsearch服务
# systemctl start elasticsearch
# systemctl enable elasticsearch
# 查看Elasticsearch状态
# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2026-03-30 10:00:00 CST; 1h ago
Docs: https://www.elastic.co
Main PID: 1234 (java)
CGroup: /system.slice/elasticsearch.service
└─1234 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-1234567890 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xlog:gc*:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.security.policy=file:/usr/share/elasticsearch/config/jvm.options.d/java.policy -Djna.tmpdir=/tmp -XX:MaxDirectMemorySize=536870912 -XX:G1HeapRegionSize=4m -XX:MinMetaspaceFreeRatio=40 -XX:MaxMetaspaceFreeRatio=50 -XX:MetaspaceSize=100m -XX:MetaspaceMaxSize=300m -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-1234567890 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xlog:gc*:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.security.policy=file:/usr/share/elasticsearch/config/jvm.options.d/java.policy -Djna.tmpdir=/tmp -XX:MaxDirectMemorySize=536870912 -XX:G1HeapRegionSize=4m -XX:MinMetaspaceFreeRatio=40 -XX:MaxMetaspaceFreeRatio=50 -XX:MetaspaceSize=100m -XX:MetaspaceMaxSize=300m org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
# 验证Elasticsearch
# curl -X GET http://fgedudb:9200/
{
"name" : "server",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "1234567890abcdef1234567890abcdef",
"version" : {
"number" : "7.15.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1234567890abcdef1234567890abcdef12345678",
"build_date" : "2021-11-04T14:04:42.515624022Z",
"build_snapshot" : false,
"lucene_version" : "8.9.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
3. Logstash安装
Logstash是一种日志收集和处理工具，用于从各种来源收集日志，进行处理后发送到Elasticsearch。
# yum install -y logstash
# 启动Logstash服务
# systemctl start logstash
# systemctl enable logstash
# 查看Logstash状态
# systemctl status logstash
● logstash.service – logstash
Loaded: loaded (/usr/lib/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2026-03-30 10:00:00 CST; 1h ago
Main PID: 1234 (java)
CGroup: /system.slice/logstash.service
└─1234 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -cp /usr/share/logstash/logstash-core/lib/jars/* org.logstash.Logstash -f /etc/logstash/conf.d/logstash.conf
4. Kibana安装
Kibana是一种数据可视化平台，用于展示Elasticsearch中的数据，支持创建仪表盘和可视化图表。
# yum install -y kibana
# 启动Kibana服务
# systemctl start kibana
# systemctl enable kibana
# 查看Kibana状态
# systemctl status kibana
● kibana.service – Kibana
Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2026-03-30 10:00:00 CST; 1h ago
Main PID: 1234 (node)
CGroup: /system.slice/kibana.service
└─1234 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli
# 访问Kibana UI
# 打开浏览器，访问 http://fgedudb:5601
5. Filebeat安装
Filebeat是一种轻量级的日志收集工具，用于从服务器收集日志并发送到Logstash或Elasticsearch。
# yum install -y filebeat
# 启动Filebeat服务
# systemctl start filebeat
# systemctl enable filebeat
# 查看Filebeat状态
# systemctl status filebeat
● filebeat.service – Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2026-03-30 10:00:00 CST; 1h ago
Main PID: 1234 (filebeat)
CGroup: /system.slice/filebeat.service
└─1234 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
6. Logstash配置
Logstash配置文件定义了输入、过滤和输出插件，用于处理日志数据。
# cat /etc/logstash/conf.d/logstash.conf
input {
beats {
port => 5044
}
}
filter {
if [message] =~ /^\d{4}-\d{2}-\d{2}/ {
grok {
match => { “message” => “%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} \[%{DATA:service}\] %{GREEDYDATA:message}” }
}
date {
match => [ “timestamp”, “yyyy-MM-dd HH:mm:ss.SSS” ]
target => “@timestamp”
}
}
}
output {
elasticsearch {
hosts => [“fgedudb:9200”]
index => “logstash-%{+YYYY.MM.dd}”
}
stdout {
codec => rubydebug
}
}
# 测试Logstash配置
# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf –config.test_and_exit
# 重新加载Logstash配置
# systemctl restart logstash
7. Elasticsearch配置
Elasticsearch配置文件定义了集群设置、内存配置、网络设置等。
# vi /etc/elasticsearch/elasticsearch.yml
# 集群名称
cluster.name: my-cluster
# 节点名称
node.name: node-1
# 数据存储路径
path.data: /var/lib/elasticsearch
# 日志存储路径
path.logs: /var/log/elasticsearch
# 网络设置
network.host: 0.0.0.0
http.port: 9200
# 集群发现
cluster.initial_master_nodes: [“node-1”]
# 重新启动Elasticsearch
# systemctl restart elasticsearch
# 查看Elasticsearch集群状态
# curl -X GET http://fgedudb:9200/_cluster/health
{
“cluster_name” : “my-cluster”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 0,
“active_shards” : 0,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0
}
8. Kibana配置
Kibana配置文件定义了Elasticsearch连接、端口设置等。
# vi /etc/kibana/kibana.yml
# 服务器设置
server.host: “0.0.0.0”
server.port: 5601
# Elasticsearch连接设置
elasticsearch.hosts: [“http://fgedudb:9200”]
# 索引模式设置
kibana.index: “.kibana”
# 重新启动Kibana
# systemctl restart kibana
# 访问Kibana UI
# 打开浏览器，访问 http://fgedudb:5601
# 在Kibana UI中，创建索引模式：logstash-*
9. 日志管理最佳实践
遵循日志管理最佳实践可以提高日志收集和分析的效率。
# 使用JSON格式或结构化日志，便于解析和分析
# 日志级别管理
# 合理设置日志级别，避免过多的 debug 日志
# 日志轮转
# 配置日志轮转，避免日志文件过大
# 日志保留策略
# 根据业务需求设置日志保留时间
# 索引管理
# 在Elasticsearch中设置索引生命周期策略
# 性能优化
# 调整Elasticsearch内存配置
# 合理设置Logstash管道
# 使用Filebeat的批量发送功能
# 安全配置
# 启用Elasticsearch安全功能
# 配置Kibana访问控制
# 加密日志传输
10. ELK故障排查
ELK故障排查包括日志分析、配置检查、性能优化等。
# tail -f /var/log/elasticsearch/elasticsearch.log
# 查看Logstash日志
# tail -f /var/log/logstash/logstash-plain.log
# 查看Kibana日志
# tail -f /var/log/kibana/kibana.log
# 查看Filebeat日志
# tail -f /var/log/filebeat/filebeat
# 检查Elasticsearch健康状态
# curl -X GET http://fgedudb:9200/_cluster/health
# 检查Elasticsearch索引
# curl -X GET http://fgedudb:9200/_cat/indices
# 检查Logstash管道
# curl -X GET http://fgedudb:9600/_node/pipelines
# 常见问题排查
# 连接问题：检查网络连接、防火墙设置
# 性能问题：检查内存使用、磁盘空间、CPU使用率
# 索引问题：检查索引状态、分片分配
# 配置问题：检查配置文件语法、参数设置
本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html
