本文档风哥主要介绍Rancher日志收集与ELK集成实战,包括Rancher数据库日志概念、Rancher数据库ELK概念、Rancher数据库Fluentd概念、Rancher数据库日志准备、Rancher数据库日志要求、Rancher数据库日志规划、Rancher数据库安装ELK、Rancher数据库配置Fluentd、Rancher数据库收集日志、Rancher数据库查询日志、Rancher数据库分析日志、Rancher数据库优化日志等内容,风哥教程参考Rancher官方文档日志、ELK、Fluentd等内容,适合运维人员在学习和测试中使用,如果要应用于生产环境则需要自行确认。
Part01-基础概念与理论知识
1.1 Rancher数据库日志概念
Rancher数据库日志是指Rancher集群和应用程序产生的日志信息,包括系统日志、应用日志、审计日志等。日志收集是指将这些日志信息集中收集到一个统一的日志系统中,便于查询和分析。Rancher集成了Fluentd,可以方便地收集和转发日志。更多视频教程www.fgedu.net.cn
- 集中收集:统一收集日志
- 实时转发:实时转发日志
- 结构化存储:结构化存储日志
- 灵活查询:灵活查询日志
- 可视化展示:可视化展示日志
1.2 Rancher数据库ELK概念
Rancher数据库ELK是指Elasticsearch、Logstash、Kibana三个开源工具的组合,用于日志收集、存储、分析和可视化。Elasticsearch是一个分布式搜索引擎,用于存储和搜索日志。Logstash是一个数据收集引擎,用于收集、过滤和转换日志。Kibana是一个数据可视化平台,用于展示和分析日志。学习交流加群风哥微信: itpux-com
- Elasticsearch:分布式搜索引擎
- Logstash:数据收集引擎
- Kibana:数据可视化平台
- 实时搜索:实时搜索日志
- 可视化展示:可视化展示日志
1.3 Rancher数据库Fluentd概念
Rancher数据库Fluentd是一个开源的数据收集器,用于统一收集和处理日志。Fluentd支持多种输入和输出插件,可以灵活地配置日志收集和转发。Fluentd具有轻量级、高性能、可扩展等特点,是Kubernetes集群中常用的日志收集工具。学习交流加群风哥QQ113257174
- 轻量级:资源占用少
- 高性能:处理速度快
- 可扩展:支持插件扩展
- 灵活配置:灵活配置规则
- 多格式:支持多种日志格式
Part02-生产环境规划与建议
2.1 Rancher数据库日志准备
Rancher数据库日志准备:
# 1. Rancher Server准备
– Rancher Server已部署
– Rancher Server可访问
– Rancher Server配置正确
# 2. 日志需求分析
– 确定日志类型
– 确定日志格式
– 确定日志保留时间
– 确定日志查询需求
# 3. 存储准备
– Elasticsearch存储:>= 200GB
– Logstash存储:>= 10GB
– Kibana存储:>= 5GB
# 4. 网络准备
– 网络带宽:>= 100Mbps
– 网络延迟:< 100ms
- 端口开放:9200、5601等
# 5. 资源准备
- Elasticsearch:>= 4核CPU,>= 8GB内存
– Logstash:>= 2核CPU,>= 4GB内存
– Kibana:>= 1核CPU,>= 2GB内存
2.2 Rancher数据库日志要求
Rancher数据库日志要求:
# Elasticsearch要求
– 版本:>= v8.0.0
– 存储:>= 200GB
– 内存:>= 8GB
– CPU:>= 4核
# Logstash要求
– 版本:>= v8.0.0
– 存储:>= 10GB
– 内存:>= 4GB
– CPU:>= 2核
# Kibana要求
– 版本:>= v8.0.0
– 存储:>= 5GB
– 内存:>= 2GB
– CPU:>= 1核
# Fluentd要求
– 版本:>= v1.15.0
– 内存:>= 512MB
– CPU:>= 0.5核
# 日志保留要求
– Elasticsearch数据保留:30天
– Logstash数据保留:7天
– Kibana数据保留:30天
– 原始日志保留:90天
# 日志格式要求
– JSON格式:推荐JSON格式
– 结构化:结构化日志
– 时间戳:包含时间戳
– 日志级别:包含日志级别
– 上下文信息:包含上下文信息
2.3 Rancher数据库日志规划
Rancher数据库日志规划:
# 日志类型规划
系统日志:/var/log/messages
应用日志:/var/log/apps/
审计日志:/var/log/audit/
容器日志:/var/log/containers/
# 日志格式规划
JSON格式:
{
“timestamp”: “2026-04-10T10:00:00Z”,
“level”: “INFO”,
“message”: “Application started”,
“service”: “fgedu-nginx”,
“host”: “fgedu-node1”
}
# 日志保留规划
Elasticsearch:30天
Logstash:7天
Kibana:30天
原始日志:90天
# 日志查询规划
实时查询:支持实时查询
历史查询:支持历史查询
全文搜索:支持全文搜索
聚合分析:支持聚合分析
# 日志告警规划
错误日志:错误日志告警
异常日志:异常日志告警
性能日志:性能日志告警
安全日志:安全日志告警
Part03-生产环境项目实施方案
3.1 Rancher数据库安装ELK
3.1.1 Rancher数据库通过Helm安装Elasticsearch
[root@rancher ~]# helm repo add elastic https://helm.elastic.co
“elastic” has been added to your repositories
[root@rancher ~]# helm repo update
Hang tight while we grab the latest from your chart repositories…
…Successfully got an update from the “elastic” chart repository
Update Complete. ⎈Happy Helming!⎈
# 安装Elasticsearch
[root@rancher ~]# helm install elasticsearch elastic/elasticsearch \
–namespace logging \
–create-namespace \
–set replicas=3 \
–set volumeClaimTemplate.resources.requests.storage=200Gi \
–set resources.requests.cpu=4 \
–set resources.requests.memory=8Gi \
–set resources.limits.cpu=8 \
–set resources.limits.memory=16Gi \
–set esJavaOpts=”-Xms8g -Xmx8g”
NAME: elasticsearch
LAST DEPLOYED: Fri Apr 10 10:00:00 2026
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The elasticsearch cluster has been installed.
Elasticsearch can be accessed:
* From inside the cluster by running `kubectl exec -it -n logging elasticsearch-0 — curl http://localhost:9200`
* From outside the cluster by running `kubectl port-forward -n logging svc/elasticsearch-master 9200:9200`
# 查看Elasticsearch状态
[root@rancher ~]# kubectl get pods -n logging
NAME READY STATUS RESTARTS AGE
elasticsearch-0 1/1 Running 0 2m
elasticsearch-1 1/1 Running 0 2m
elasticsearch-2 1/1 Running 0 2m
# 查看Elasticsearch集群状态
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s http://localhost:9200/_cluster/health?pretty
{
“cluster_name” : “elasticsearch”,
“status” : “green”,
“number_of_nodes” : 3,
“number_of_data_nodes” : 3,
“active_primary_shards” : 10,
“active_shards” : 20,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0
}
3.1.2 Rancher数据库通过Helm安装Kibana
[root@rancher ~]# helm install kibana elastic/kibana \
–namespace logging \
–set service.type=LoadBalancer \
–set resources.requests.cpu=1 \
–set resources.requests.memory=2Gi \
–set resources.limits.cpu=2 \
–set resources.limits.memory=4Gi
NAME: kibana
LAST DEPLOYED: Fri Apr 10 10:00:00 2026
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Kibana can be accessed:
* From outside the cluster by running `kubectl port-forward -n logging svc/kibana 5601:5601`
# 查看Kibana状态
[root@rancher ~]# kubectl get pods -n logging
NAME READY STATUS RESTARTS AGE
elasticsearch-0 1/1 Running 0 5m
elasticsearch-1 1/1 Running 0 5m
elasticsearch-2 1/1 Running 0 5m
kibana-1234567890-abcde 1/1 Running 0 2m
# 端口转发访问Kibana
[root@rancher ~]# kubectl -n logging port-forward svc/kibana 5601:5601 &
Forwarding from 127.0.0.1:5601 -> 5601
Forwarding from [::1]:5601 -> 5601
# 访问Kibana
# URL: http://localhost:5601
# 用户名:elastic
# 密码:查看Elasticsearch密码
3.2 Rancher数据库配置Fluentd
3.2.1 Rancher数据库安装Fluentd
[root@rancher ~]# helm repo add fluent https://fluent.github.io/helm-charts
“fluent” has been added to your repositories
[root@rancher ~]# helm repo update
Hang tight while we grab the latest from your chart repositories…
…Successfully got an update from the “fluent” chart repository
Update Complete. ⎈Happy Helming!⎈
# 安装Fluentd
[root@rancher ~]# helm install fluentd fluent/fluentd \
–namespace logging \
–set image.tag=v1.15.3 \
–set resources.requests.cpu=0.5 \
–set resources.requests.memory=512Mi \
–set resources.limits.cpu=1 \
–set resources.limits.memory=1Gi \
–set tolerations[0].key=node-role.kubernetes.io/master \
–set tolerations[0].effect=NoSchedule
NAME: fluentd
LAST DEPLOYED: Fri Apr 10 10:00:00 2026
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Fluentd has been installed.
Fluentd can be accessed:
* From inside the cluster by running `kubectl exec -it -n logging fluentd-1234567890-abcde — fluentd –version`
# 查看Fluentd状态
[root@rancher ~]# kubectl get pods -n logging
NAME READY STATUS RESTARTS AGE
elasticsearch-0 1/1 Running 0 10m
elasticsearch-1 1/1 Running 0 10m
elasticsearch-2 1/1 Running 0 10m
kibana-1234567890-abcde 1/1 Running 0 7m
fluentd-1234567890-abcde 1/1 Running 0 2m
fluentd-2345678901-fghij 1/1 Running 0 2m
fluentd-3456789012-klmno 1/1 Running 0 2m
3.2.2 Rancher数据库配置Fluentd输出到Elasticsearch
[root@rancher ~]# cat <
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
time_format %Y-%m-%dT%H:%M:%S.%NZ
@type kubernetes_metadata
@type elasticsearch
host elasticsearch-master.logging.svc.cluster.local
port 9200
logstash_format true
logstash_prefix kubernetes
logstash_dateformat %Y.%m.%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s
EOF
configmap/fluentd-config created
# 更新Fluentd配置
[root@rancher ~]# kubectl set env daemonset/fluentd -n logging \
–from=env/FLUENTD_CONF=fluent.conf
daemonset.apps/fluentd env updated
# 重启Fluentd
[root@rancher ~]# kubectl rollout restart daemonset/fluentd -n logging
daemonset.apps/fluentd restarted
# 查看Fluentd状态
[root@rancher ~]# kubectl get pods -n logging -l app=fluentd
NAME READY STATUS RESTARTS AGE
fluentd-1234567890-abcde 1/1 Running 0 1m
fluentd-2345678901-fghij 1/1 Running 0 1m
fluentd-3456789012-klmno 1/1 Running 0 1m
3.3 Rancher数据库收集日志
3.3.1 Rancher数据库验证日志收集
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s http://localhost:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open kubernetes-2026.04.10 12345678-90ab-cdef 5 1 12345 0 10.5mb 10.5mb
# 查看日志数据
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s http://localhost:9200/kubernetes-2026.04.10/_search?pretty | head -50
{
“took” : 5,
“timed_out” : false,
“_shards” : {
“total” : 5,
“successful” : 5,
“skipped” : 0,
“failed” : 0
},
“hits” : {
“total” : {
“value” : 12345,
“relation” : “eq”
},
“max_score” : 1.0,
“hits” : [
{
“_index” : “kubernetes-2026.04.10”,
“_type” : “_doc”,
“_id” : “1234567890abcdef”,
“_score” : 1.0,
“_source” : {
“@timestamp” : “2026-04-10T10:00:00.000Z”,
“log” : “2026/04/10 10:00:00 [info] Application started”,
“stream” : “stdout”,
“kubernetes” : {
“pod_name” : “fgedu-nginx-1234567890-abcde”,
“namespace_name” : “fgedu-dev”,
“pod_id” : “12345678-90ab-cdef-1234-567890abcdef”,
“labels” : {
“app” : “fgedu-nginx”
}
}
}
}
]
}
}
# 通过Kibana查看日志
# 步骤1:登录Kibana界面
# 步骤2:点击”Management” – “Stack Management” – “Index Patterns”
# 步骤3:点击”Create index pattern”
# 步骤4:输入索引模式:kubernetes-*
# 步骤5:选择时间字段:@timestamp
# 步骤6:点击”Create index pattern”
# 步骤7:点击”Discover”查看日志
Part04-生产案例与实战讲解
4.1 Rancher数据库查询日志
4.1.1 Rancher数据库通过Kibana查询日志
# 步骤1:登录Kibana界面
# 步骤2:点击”Discover”
# 步骤3:选择索引模式:kubernetes-*
# 步骤4:设置时间范围:Last 1 hour
# 步骤5:输入查询条件:kubernetes.pod_name: “fgedu-nginx*”
# 步骤6:查看查询结果
# 通过Elasticsearch API查询日志
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s ‘http://localhost:9200/kubernetes-2026.04.10/_search?q=kubernetes.pod_name:fgedu-nginx*&pretty’ | head -50
{
“took” : 3,
“timed_out” : false,
“_shards” : {
“total” : 5,
“successful” : 5,
“skipped” : 0,
“failed” : 0
},
“hits” : {
“total” : {
“value” : 5678,
“relation” : “eq”
},
“max_score” : 1.0,
“hits” : [
{
“_index” : “kubernetes-2026.04.10”,
“_type” : “_doc”,
“_id” : “1234567890abcdef”,
“_score” : 1.0,
“_source” : {
“@timestamp” : “2026-04-10T10:00:00.000Z”,
“log” : “2026/04/10 10:00:00 [info] Application started”,
“stream” : “stdout”,
“kubernetes” : {
“pod_name” : “fgedu-nginx-1234567890-abcde”,
“namespace_name” : “fgedu-dev”
}
}
}
]
}
}
# 通过Elasticsearch DSL查询日志
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s -X POST ‘http://localhost:9200/kubernetes-2026.04.10/_search?pretty’ -H ‘Content-Type: application/json’ -d’
{
“query”: {
“bool”: {
“must”: [
{
“match”: {
“kubernetes.pod_name”: “fgedu-nginx”
}
},
{
“range”: {
“@timestamp”: {
“gte”: “now-1h”
}
}
}
]
}
}
}
‘ | head -50
{
“took” : 5,
“timed_out” : false,
“_shards” : {
“total” : 5,
“successful” : 5,
“skipped” : 0,
“failed” : 0
},
“hits” : {
“total” : {
“value” : 1234,
“relation” : “eq”
},
“max_score” : 2.0,
“hits” : [
{
“_index” : “kubernetes-2026.04.10”,
“_type” : “_doc”,
“_id” : “1234567890abcdef”,
“_score” : 2.0,
“_source” : {
“@timestamp” : “2026-04-10T10:00:00.000Z”,
“log” : “2026/04/10 10:00:00 [info] Application started”,
“stream” : “stdout”,
“kubernetes” : {
“pod_name” : “fgedu-nginx-1234567890-abcde”,
“namespace_name” : “fgedu-dev”
}
}
}
]
}
}
4.2 Rancher数据库分析日志
4.2.1 Rancher数据库通过Kibana分析日志
# 步骤1:登录Kibana界面
# 步骤2:点击”Visualize Library”
# 步骤3:点击”Create visualization”
# 步骤4:选择可视化类型:Line chart
# 步骤5:选择索引模式:kubernetes-*
# 步骤6:配置Y轴:Count
# 步骤7:配置X轴:Date Histogram (@timestamp)
# 步骤8:点击”Save visualization”
# 通过Elasticsearch聚合分析日志
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s -X POST ‘http://localhost:9200/kubernetes-2026.04.10/_search?pretty’ -H ‘Content-Type: application/json’ -d’
{
“size”: 0,
“aggs”: {
“by_pod”: {
“terms”: {
“field”: “kubernetes.pod_name.keyword”,
“size”: 10
}
}
}
}
‘ | head -50
{
“took” : 5,
“timed_out” : false,
“_shards” : {
“total” : 5,
“successful” : 5,
“skipped” : 0,
“failed” : 0
},
“hits” : {
“total” : {
“value” : 12345,
“relation” : “eq”
},
“max_score” : null,
“hits” : []
},
“aggregations” : {
“by_pod” : {
“doc_count_error_upper_bound” : 0,
“sum_other_doc_count” : 0,
“buckets” : [
{
“key” : “fgedu-nginx-1234567890-abcde”,
“doc_count” : 3456
},
{
“key” : “fgedu-nginx-1234567890-fghij”,
“doc_count” : 2345
},
{
“key” : “fgedu-nginx-1234567890-klmno”,
“doc_count” : 1234
}
]
}
}
}
4.3 Rancher数据库优化日志
4.3.1 Rancher数据库优化Elasticsearch性能
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s -X PUT ‘http://localhost:9200/_template/fgedu-template?pretty’ -H ‘Content-Type: application/json’ -d’
{
“index_patterns”: [“kubernetes-*”],
“settings”: {
“number_of_shards”: 5,
“number_of_replicas”: 1,
“index.lifecycle.name”: “fgedu-policy”,
“index.lifecycle.rollover_alias”: “kubernetes”
},
“mappings”: {
“properties”: {
“@timestamp”: {
“type”: “date”
},
“log”: {
“type”: “text”,
“fields”: {
“keyword”: {
“type”: “keyword”
}
}
}
}
}
}
‘
{
“acknowledged” : true
}
# 配置索引生命周期管理
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s -X PUT ‘http://localhost:9200/_ilm/policy/fgedu-policy?pretty’ -H ‘Content-Type: application/json’ -d’
{
“policy”: {
“phases”: {
“hot”: {
“actions”: {
“rollover”: {
“max_size”: “50GB”,
“max_age”: “1d”
}
}
},
“delete”: {
“min_age”: “30d”,
“actions”: {
“delete”: {}
}
}
}
}
}
‘
{
“acknowledged” : true
}
# 查看索引状态
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s http://localhost:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open kubernetes-2026.04.10 12345678-90ab-cdef 5 1 12345 0 10.5mb 10.5mb
green open kubernetes-2026.04.11 23456789-0123-efgh 5 1 23456 0 20.5mb 20.5mb
Part05-风哥经验总结与分享
5.1 Rancher数据库日志最佳实践
Rancher数据库日志最佳实践:
- 集中管理:使用ELK栈集中管理日志
- 结构化日志:使用结构化日志格式
- 日志分级:合理设置日志级别
- 性能优化:优化Elasticsearch性能
- 定期清理:定期清理过期日志
- 日志告警:配置日志告警规则
- 文档记录:记录日志配置和变更
5.2 Rancher数据库日志问题排查
Rancher数据库日志问题排查:
# 问题1:Fluentd无法收集日志
# 现象:Elasticsearch中没有日志数据
# 原因:Fluentd配置错误、权限不足、网络不通
# 解决:
[root@rancher ~]# kubectl get pods -n logging -l app=fluentd
[root@rancher ~]# kubectl logs -n logging fluentd-1234567890-abcde
[root@rancher ~]# kubectl describe pod -n logging fluentd-1234567890-abcde
[root@rancher ~]# kubectl exec -n logging fluentd-1234567890-abcde — fluentd –version
# 问题2:Elasticsearch无法启动
# 现象:Elasticsearch Pod状态为CrashLoopBackOff
# 原因:配置错误、存储不足、资源不足
# 解决:
[root@rancher ~]# kubectl get pods -n logging -l app=elasticsearch
[root@rancher ~]# kubectl logs -n logging elasticsearch-0
[root@rancher ~]# kubectl describe pod -n logging elasticsearch-0
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — df -h
# 问题3:Kibana无法访问
# 现象:无法访问Kibana界面
# 原因:网络不通、端口未开放、认证失败
# 解决:
[root@rancher ~]# kubectl get svc -n logging kibana
[root@rancher ~]# kubectl port-forward -n logging svc/kibana 5601:5601
[root@rancher ~]# curl http://localhost:5601
[root@rancher ~]# kubectl logs -n logging kibana-1234567890-abcde
# 问题4:日志查询慢
# 现象:查询日志时响应很慢
# 原因:索引配置不当、数据量大、资源不足
# 解决:
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s http://localhost:9200/_cat/indices?v
[root@rancher ~]# kubectl exec -n logging elasticsearch-0 — curl -s http://localhost:9200/_cat/nodes?v
[root@rancher ~]# kubectl top pod -n logging elasticsearch-0
5.3 Rancher数据库日志维护
Rancher数据库日志维护:
# 1. 定期检查
– 检查Fluentd状态
– 检查Elasticsearch状态
– 检查Kibana状态
– 检查日志收集情况
# 2. 定期优化
– 优化Elasticsearch配置
– 优化索引配置
– 优化查询性能
– 优化存储空间
# 3. 定期备份
– 备份Elasticsearch数据
– 备份Kibana配置
– 备份Fluentd配置
– 备份索引模板
# 4. 定期清理
– 清理过期日志
– 清理无用索引
– 清理过期快照
– 清理过期日志文件
# 5. 定期审计
– 审计日志配置
– 审计日志访问
– 审计日志查询
– 审计操作日志
本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html
