Kubernetes教程FG028-Kubernetes故障处理与调试指南实战解析
本文档风哥主要介绍Kubernetes中的故障处理与调试，包括故障处理概述、调试概念、Kubernetes故障处理、故障处理策略、调试工具、最佳实践规划、控制平面故障处理、节点故障处理、应用故障处理、网络故障处理、故障处理案例、调试案例、恢复案例等内容，风哥教程参考Kubernetes官方文档和故障处理相关文档，适合DevOps工程师和系统管理员在学习和测试中使用，如果要应用于生产环境则需要自行确认。
Part01-基础概念与理论知识
1.1 故障处理概述
故障处理是指识别、分析和解决系统故障的过程，包括故障检测、故障定位、故障分析、故障修复和故障预防等步骤。Kubernetes故障处理需要综合考虑控制平面、节点、应用、网络等多个方面的问题。
1.2 调试概念
调试是指通过各种工具和方法，找出并解决系统中的问题，包括日志分析、网络诊断、资源监控、性能分析等。Kubernetes调试需要使用kubectl、logs、describe、exec等命令，以及Prometheus、Grafana等监控工具。
1.3 Kubernetes故障处理
Kubernetes故障处理是指在Kubernetes集群中识别、分析和解决故障的过程，包括控制平面故障、节点故障、应用故障、网络故障等。Kubernetes提供了多种工具和方法来帮助故障处理，如kubectl命令、事件查看、日志分析等。
Part02-生产环境规划与建议
2.1 故障处理策略
生产环境Kubernetes故障处理策略：
# 故障处理策略
– 预防为主：
– 定期检查集群状态，及时发现潜在问题
– 实施监控和告警，及时通知异常情况
– 备份关键数据，确保数据安全
– 制定应急预案，提高故障处理效率
– 快速响应：
– 建立故障响应机制，明确责任和流程
– 快速定位故障原因，减少故障影响范围
– 优先恢复服务，确保业务连续性
– 记录故障处理过程，积累经验
– 根因分析：
– 深入分析故障原因，避免类似问题再次发生
– 实施修复措施，确保系统稳定性
– 优化系统配置，提高系统可靠性
– 定期回顾故障处理经验，持续改进
– 团队协作：
– 建立跨团队协作机制，共同处理故障
– 明确沟通渠道，确保信息及时传递
– 分享故障处理经验，提高团队能力
– 定期培训，提高团队故障处理技能
– 工具支持：
– 使用监控工具，及时发现异常
– 使用日志分析工具，快速定位问题
– 使用诊断工具，深入分析故障原因
– 使用自动化工具，提高故障处理效率
– 预防为主：
– 定期检查集群状态，及时发现潜在问题
– 实施监控和告警，及时通知异常情况
– 备份关键数据，确保数据安全
– 制定应急预案，提高故障处理效率
– 快速响应：
– 建立故障响应机制，明确责任和流程
– 快速定位故障原因，减少故障影响范围
– 优先恢复服务，确保业务连续性
– 记录故障处理过程，积累经验
– 根因分析：
– 深入分析故障原因，避免类似问题再次发生
– 实施修复措施，确保系统稳定性
– 优化系统配置，提高系统可靠性
– 定期回顾故障处理经验，持续改进
– 团队协作：
– 建立跨团队协作机制，共同处理故障
– 明确沟通渠道，确保信息及时传递
– 分享故障处理经验，提高团队能力
– 定期培训，提高团队故障处理技能
– 工具支持：
– 使用监控工具，及时发现异常
– 使用日志分析工具，快速定位问题
– 使用诊断工具，深入分析故障原因
– 使用自动化工具，提高故障处理效率
2.2 调试工具
生产环境Kubernetes调试工具：
# 调试工具
– kubectl命令：
– kubectl get：查看资源状态
– kubectl describe：查看资源详细信息
– kubectl logs：查看容器日志
– kubectl exec：进入容器执行命令
– kubectl port-forward：端口转发
– kubectl proxy：代理API服务器
– kubectl cp：复制文件
– kubectl drain：排空节点
– 监控工具：
– Prometheus：监控集群和应用性能
– Grafana：可视化监控数据
– Alertmanager：管理告警
– kube-state-metrics：收集Kubernetes资源状态
– 日志工具：
– ELK Stack：收集、存储和分析日志
– Loki：轻量级日志聚合系统
– Fluentd：日志收集和转发
– Promtail：日志收集代理
– 网络工具：
– ping：测试网络连通性
– curl：测试HTTP服务
– netstat：查看网络连接
– tcpdump：网络抓包
– traceroute：跟踪网络路径
– 诊断工具：
– kubeadm：集群诊断
– kubelet：节点诊断
– crictl：容器运行时诊断
– etcdctl：etcd诊断
– kubernetes-dashboard：集群可视化
– kubectl命令：
– kubectl get：查看资源状态
– kubectl describe：查看资源详细信息
– kubectl logs：查看容器日志
– kubectl exec：进入容器执行命令
– kubectl port-forward：端口转发
– kubectl proxy：代理API服务器
– kubectl cp：复制文件
– kubectl drain：排空节点
– 监控工具：
– Prometheus：监控集群和应用性能
– Grafana：可视化监控数据
– Alertmanager：管理告警
– kube-state-metrics：收集Kubernetes资源状态
– 日志工具：
– ELK Stack：收集、存储和分析日志
– Loki：轻量级日志聚合系统
– Fluentd：日志收集和转发
– Promtail：日志收集代理
– 网络工具：
– ping：测试网络连通性
– curl：测试HTTP服务
– netstat：查看网络连接
– tcpdump：网络抓包
– traceroute：跟踪网络路径
– 诊断工具：
– kubeadm：集群诊断
– kubelet：节点诊断
– crictl：容器运行时诊断
– etcdctl：etcd诊断
– kubernetes-dashboard：集群可视化
2.3 最佳实践规划
生产环境Kubernetes故障处理与调试的最佳实践规划：
# 最佳实践规划
– 监控和告警：
– 部署Prometheus和Grafana，监控集群和应用
– 配置告警规则，及时通知异常情况
– 建立监控仪表盘，直观查看系统状态
– 定期检查监控配置，确保监控有效性
– 日志管理：
– 配置集中式日志管理系统，如ELK Stack或Loki
– 标准化日志格式，便于分析和查询
– 设置日志保留策略，平衡存储成本和查询需求
– 定期清理过期日志，避免存储溢出
– 网络管理：
– 配置网络策略，限制不必要的网络通信
– 监控网络性能，及时发现网络问题
– 实施网络隔离，提高网络安全性
– 定期测试网络连通性，确保网络可靠性
– 存储管理：
– 监控存储使用情况，及时扩容
– 备份关键数据，确保数据安全
– 测试存储恢复流程，确保数据可恢复
– 定期检查存储性能，优化存储配置
– 安全管理：
– 配置Pod安全策略，限制Pod特权
– 使用RBAC控制访问权限
– 定期扫描安全漏洞，及时修复
– 监控安全事件，及时响应
– 文档和流程：
– 建立故障处理流程，明确责任和步骤
– 记录故障处理过程，积累经验
– 编写故障处理文档，指导团队成员
– 定期培训，提高团队故障处理能力
– 监控和告警：
– 部署Prometheus和Grafana，监控集群和应用
– 配置告警规则，及时通知异常情况
– 建立监控仪表盘，直观查看系统状态
– 定期检查监控配置，确保监控有效性
– 日志管理：
– 配置集中式日志管理系统，如ELK Stack或Loki
– 标准化日志格式，便于分析和查询
– 设置日志保留策略，平衡存储成本和查询需求
– 定期清理过期日志，避免存储溢出
– 网络管理：
– 配置网络策略，限制不必要的网络通信
– 监控网络性能，及时发现网络问题
– 实施网络隔离，提高网络安全性
– 定期测试网络连通性，确保网络可靠性
– 存储管理：
– 监控存储使用情况，及时扩容
– 备份关键数据，确保数据安全
– 测试存储恢复流程，确保数据可恢复
– 定期检查存储性能，优化存储配置
– 安全管理：
– 配置Pod安全策略，限制Pod特权
– 使用RBAC控制访问权限
– 定期扫描安全漏洞，及时修复
– 监控安全事件，及时响应
– 文档和流程：
– 建立故障处理流程，明确责任和步骤
– 记录故障处理过程，积累经验
– 编写故障处理文档，指导团队成员
– 定期培训，提高团队故障处理能力
Part03-生产环境项目实施方案
3.1 控制平面故障处理
生产环境Kubernetes控制平面故障处理的实施：
# 控制平面故障处理
– 检查控制平面组件状态：
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
etcd-fgedu-master 1/1 Running 0 1d
kube-apiserver-fgedu-master 1/1 Running 0 1d
kube-controller-manager-fgedu-master 1/1 Running 0 1d
kube-scheduler-fgedu-master 1/1 Running 0 1d
– 检查etcd状态：
$ kubectl exec -it etcd-fgedu-master -n kube-system — etcdctl –endpoints=https://127.0.0.1:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key endpoint health
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.234ms
– 检查API Server状态：
$ kubectl get –raw=/healthz
ok
– 检查控制器管理器状态：
$ kubectl get pods -n kube-system | grep controller-manager
kube-controller-manager-fgedu-master 1/1 Running 0 1d
– 检查调度器状态：
$ kubectl get pods -n kube-system | grep scheduler
kube-scheduler-fgedu-master 1/1 Running 0 1d
– 控制平面故障修复：
– etcd故障：
$ kubectl exec -it etcd-fgedu-master -n kube-system — etcdctl –endpoints=https://127.0.0.1:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcd-snapshot.db
$ kubectl exec -it etcd-fgedu-master -n kube-system — etcdctl –endpoints=https://127.0.0.1:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot restore /tmp/etcd-snapshot.db
– API Server故障：
$ systemctl restart kube-apiserver
– 控制器管理器故障：
$ systemctl restart kube-controller-manager
– 调度器故障：
$ systemctl restart kube-scheduler
– 检查控制平面组件状态：
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
etcd-fgedu-master 1/1 Running 0 1d
kube-apiserver-fgedu-master 1/1 Running 0 1d
kube-controller-manager-fgedu-master 1/1 Running 0 1d
kube-scheduler-fgedu-master 1/1 Running 0 1d
– 检查etcd状态：
$ kubectl exec -it etcd-fgedu-master -n kube-system — etcdctl –endpoints=https://127.0.0.1:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key endpoint health
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.234ms
– 检查API Server状态：
$ kubectl get –raw=/healthz
ok
– 检查控制器管理器状态：
$ kubectl get pods -n kube-system | grep controller-manager
kube-controller-manager-fgedu-master 1/1 Running 0 1d
– 检查调度器状态：
$ kubectl get pods -n kube-system | grep scheduler
kube-scheduler-fgedu-master 1/1 Running 0 1d
– 控制平面故障修复：
– etcd故障：
$ kubectl exec -it etcd-fgedu-master -n kube-system — etcdctl –endpoints=https://127.0.0.1:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcd-snapshot.db
$ kubectl exec -it etcd-fgedu-master -n kube-system — etcdctl –endpoints=https://127.0.0.1:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot restore /tmp/etcd-snapshot.db
– API Server故障：
$ systemctl restart kube-apiserver
– 控制器管理器故障：
$ systemctl restart kube-controller-manager
– 调度器故障：
$ systemctl restart kube-scheduler
3.2 节点故障处理
生产环境Kubernetes节点故障处理的实施：
# 节点故障处理
– 检查节点状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready 1d v1.24.0
fgedu-node2 NotReady 1d v1.24.0
– 检查节点详细信息：
$ kubectl describe node fgedu-node2
Name: fgedu-node2
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=fgedu-node2
kubernetes.io/os=linux
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 01 Jan 2024 00:00:00 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message，风哥提示：。
—- —— —————– —————— —— ——-
Ready False Fri, 01 Jan 2024 01:00:00 +0000 Fri, 01 Jan 2024 01:00:00 +0000 KubeletNotReady Container runtime is down
…
– 检查kubelet状态：
$ ssh fgedu-node2 systemctl status kubelet
● kubelet.service – kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 01:00:00 UTC; 10s ago
Docs: https://kubernetes.io/docs/
Process: 1234 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
Main PID: 1234 (code=exited, status=255)
– 检查容器运行时状态：
$ ssh fgedu-node2 systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 01:00:00 UTC; 20s ago
Docs: https://docs.docker.com
Process: 5678 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock (code=exited, status=255)
Main PID: 5678 (code=exited, status=255)
– 节点故障修复：
– 重启容器运行时：
$ ssh fgedu-node2 systemctl restart docker
$ ssh fgedu-node2 systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 01:05:00 UTC; 5s ago
Docs: https://docs.docker.com
Process: 5678 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock (code=exited, status=0/SUCCESS)
Main PID: 5678 (dockerd)
– 重启kubelet：
$ ssh fgedu-node2 systemctl restart kubelet
$ ssh fgedu-node2 systemctl status kubelet
● kubelet.service – kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 01:10:00 UTC; 5s ago
Docs: https://kubernetes.io/docs/
Process: 1234 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=0/SUCCESS)
Main PID: 1234 (kubelet)
– 验证节点状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready 1d v1.24.0
fgedu-node2 Ready 1d v1.24.0
– 检查节点状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready
fgedu-node2 NotReady
– 检查节点详细信息：
$ kubectl describe node fgedu-node2
Name: fgedu-node2
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=fgedu-node2
kubernetes.io/os=linux
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 01 Jan 2024 00:00:00 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message，风哥提示：。
—- —— —————– —————— —— ——-
Ready False Fri, 01 Jan 2024 01:00:00 +0000 Fri, 01 Jan 2024 01:00:00 +0000 KubeletNotReady Container runtime is down
…
– 检查kubelet状态：
$ ssh fgedu-node2 systemctl status kubelet
● kubelet.service – kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 01:00:00 UTC; 10s ago
Docs: https://kubernetes.io/docs/
Process: 1234 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
Main PID: 1234 (code=exited, status=255)
– 检查容器运行时状态：
$ ssh fgedu-node2 systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 01:00:00 UTC; 20s ago
Docs: https://docs.docker.com
Process: 5678 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock (code=exited, status=255)
Main PID: 5678 (code=exited, status=255)
– 节点故障修复：
– 重启容器运行时：
$ ssh fgedu-node2 systemctl restart docker
$ ssh fgedu-node2 systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 01:05:00 UTC; 5s ago
Docs: https://docs.docker.com
Process: 5678 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock (code=exited, status=0/SUCCESS)
Main PID: 5678 (dockerd)
– 重启kubelet：
$ ssh fgedu-node2 systemctl restart kubelet
$ ssh fgedu-node2 systemctl status kubelet
● kubelet.service – kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 01:10:00 UTC; 5s ago
Docs: https://kubernetes.io/docs/
Process: 1234 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=0/SUCCESS)
Main PID: 1234 (kubelet)
– 验证节点状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready
fgedu-node2 Ready
3.3 应用故障处理
生产环境Kubernetes应用故障处理的实施。，风哥提示：。
# 应用故障处理
– 检查应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 0/1 CrashLoopBackOff 5 10m
fgedu-app-6d6f58987b-8d2k3 1/1 Running 0 10m
fgedu-app-6d6f58987b-9f5g7 1/1 Running 0 10m
– 检查应用详细信息：
$ kubectl describe pod fgedu-app-6d6f58987b-7f5f8
Name: fgedu-app-6d6f58987b-7f5f8
Namespace: default
Priority: 0
Node: fgedu-node1/192.168.1.101
Start Time: Fri, 01 Jan 2024 00:00:00 +0000
Labels: app=fgedu-app
pod-template-hash=6d6f58987b
Annotations:
Status: Running
IP: 10.244.1.2
IPs:
IP: 10.244.1.2
Controlled By: ReplicaSet/fgedu-app-6d6f58987b
Containers:
fgedu-app:
Container ID: docker://1234567890ab
Image: nginx:latest
Image ID: docker-pullable://nginx@sha256:1234567890ab
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 01 Jan 2024 00:05:00 +0000
Finished: Fri, 01 Jan 2024 00:05:30 +0000
Ready: False
Restart Count: 5
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xyz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-xyz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xyz
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal Scheduled 10m default-scheduler Successfully assigned default/fgedu-app-6d6f58987b-7f5f8 to fgedu-node1
Normal Pulling 10m kubelet Pulling image “nginx:latest”
Normal Pulled 10m kubelet Successfully pulled image “nginx:latest”
Normal Created 10m kubelet Created container fgedu-app
Normal Started 10m kubelet Started container fgedu-app
Normal Killing 9m (x5 over 10m) kubelet Container fgedu-app failed liveness probe, will be restarted
Warning Unhealthy 9m (x6 over 10m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 500
Warning BackOff 2m (x22 over 9m) kubelet Back-off restarting failed container
– 检查应用日志：
$ kubectl logs fgedu-app-6d6f58987b-7f5f8
2024/01/01 00:00:00 [emerg] 1#1: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
2024/01/01 00:00:00 [emerg] 1#1: bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
2024/01/01 00:00:00 [emerg] 1#1: still could not bind()
nginx: [emerg] still could not bind()
– 进入容器检查：
$ kubectl exec -it fgedu-app-6d6f58987b-8d2k3 — bash
root@fgedu-app-6d6f58987b-8d2k3:/# netstat -tuln
Active Internet connections (only servers)，学习交流加群风哥微信: itpux-com。
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp6 0 0 :::80 :::* LISTEN
– 应用故障修复：
– 检查端口占用：
$ kubectl exec -it fgedu-app-6d6f58987b-7f5f8 — lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 1234 root 6u IPv4 12345 0t0 TCP *:80 (LISTEN)
– 修复应用配置：
$ kubectl edit deployment fgedu-app
# 修改端口配置
ports:
– containerPort: 8080
– 验证应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 1/1 Running 0 15m
fgedu-app-6d6f58987b-8d2k3 1/1 Running 0 15m
fgedu-app-6d6f58987b-9f5g7 1/1 Running 0 15m
– 检查应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 0/1 CrashLoopBackOff 5 10m
fgedu-app-6d6f58987b-8d2k3 1/1 Running 0 10m
fgedu-app-6d6f58987b-9f5g7 1/1 Running 0 10m
– 检查应用详细信息：
$ kubectl describe pod fgedu-app-6d6f58987b-7f5f8
Name: fgedu-app-6d6f58987b-7f5f8
Namespace: default
Priority: 0
Node: fgedu-node1/192.168.1.101
Start Time: Fri, 01 Jan 2024 00:00:00 +0000
Labels: app=fgedu-app
pod-template-hash=6d6f58987b
Annotations:
Status: Running
IP: 10.244.1.2
IPs:
IP: 10.244.1.2
Controlled By: ReplicaSet/fgedu-app-6d6f58987b
Containers:
fgedu-app:
Container ID: docker://1234567890ab
Image: nginx:latest
Image ID: docker-pullable://nginx@sha256:1234567890ab
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 01 Jan 2024 00:05:00 +0000
Finished: Fri, 01 Jan 2024 00:05:30 +0000
Ready: False
Restart Count: 5
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xyz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-xyz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xyz
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal Scheduled 10m default-scheduler Successfully assigned default/fgedu-app-6d6f58987b-7f5f8 to fgedu-node1
Normal Pulling 10m kubelet Pulling image “nginx:latest”
Normal Pulled 10m kubelet Successfully pulled image “nginx:latest”
Normal Created 10m kubelet Created container fgedu-app
Normal Started 10m kubelet Started container fgedu-app
Normal Killing 9m (x5 over 10m) kubelet Container fgedu-app failed liveness probe, will be restarted
Warning Unhealthy 9m (x6 over 10m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 500
Warning BackOff 2m (x22 over 9m) kubelet Back-off restarting failed container
– 检查应用日志：
$ kubectl logs fgedu-app-6d6f58987b-7f5f8
2024/01/01 00:00:00 [emerg] 1#1: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
2024/01/01 00:00:00 [emerg] 1#1: bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
2024/01/01 00:00:00 [emerg] 1#1: still could not bind()
nginx: [emerg] still could not bind()
– 进入容器检查：
$ kubectl exec -it fgedu-app-6d6f58987b-8d2k3 — bash
root@fgedu-app-6d6f58987b-8d2k3:/# netstat -tuln
Active Internet connections (only servers)，学习交流加群风哥微信: itpux-com。
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp6 0 0 :::80 :::* LISTEN
– 应用故障修复：
– 检查端口占用：
$ kubectl exec -it fgedu-app-6d6f58987b-7f5f8 — lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 1234 root 6u IPv4 12345 0t0 TCP *:80 (LISTEN)
– 修复应用配置：
$ kubectl edit deployment fgedu-app
# 修改端口配置
ports:
– containerPort: 8080
– 验证应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 1/1 Running 0 15m
fgedu-app-6d6f58987b-8d2k3 1/1 Running 0 15m
fgedu-app-6d6f58987b-9f5g7 1/1 Running 0 15m
3.4 网络故障处理
生产环境Kubernetes网络故障处理的实施：
# 网络故障处理
– 检查网络插件状态：
$ kubectl get pods -n kube-system | grep calico
calico-node-7f5f8 1/1 Running 0 1d
calico-node-8d2k3 1/1 Running 0 1d
calico-node-9f5g7 1/1 Running 0 1d
– 检查Pod网络连通性：
$ kubectl run -it –rm –image=busybox:1.28 busybox — ping -c 4 10.244.1.2
PING 10.244.1.2 (10.244.1.2): 56 data bytes
64 bytes from 10.244.1.2: seq=0 ttl=64 time=0.500 ms
64 bytes from 10.244.1.2: seq=1 ttl=64 time=0.400 ms
64 bytes from 10.244.1.2: seq=2 ttl=64 time=0.300 ms
64 bytes from 10.244.1.2: seq=3 ttl=64 time=0.200 ms
— 10.244.1.2 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.200/0.350/0.500 ms
– 检查Service网络连通性：
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fgedu-app LoadBalancer 10.96.123.45 192.168.1.200 80:30080/TCP 1d
kubernetes ClusterIP 10.96.0.1 443/TCP 1d
$ kubectl run -it –rm –image=busybox:1.28 busybox — wget -O- http://fgedu-app.default.svc.cluster.local
Connecting to fgedu-app.default.svc.cluster.local (10.96.123.45:80)
wget: server returned error: HTTP/1.1 503 Service Unavailable
– 检查网络策略：
$ kubectl get networkpolicies
NAME AGE
fgedu-app-network-policy 1d
$ kubectl describe networkpolicy fgedu-app-network-policy
Name: fgedu-app-network-policy
Namespace: default
Created on: 2024-01-01 00:00:00 +0000 UTC
Spec:
PodSelector: app=fgedu-app
Allowing ingress traffic:
To Port: 80/TCP
From:
PodSelector: app=fgedu-app
Allowing egress traffic:
To Port: 80/TCP
To:
PodSelector: app=fgedu-app
– 网络故障修复：
– 修改网络策略：
$ kubectl edit networkpolicy fgedu-app-network-policy
# 添加允许来自busybox的流量
ingress:
– from:
– podSelector: {}
ports:
– protocol: TCP
port: 80
– 验证网络连通性：
$ kubectl run -it –rm –image=busybox:1.28 busybox — wget -O- http://fgedu-app.default.svc.cluster.local
Connecting to fgedu-app.default.svc.cluster.local (10.96.123.45:80)
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Fri, 01 Jan 2024 00:00:00 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 14 Dec 2021 14:49:29 GMT
Connection: keep-alive
ETag: “61b8a129-267”
Accept-Ranges: bytes
– 检查网络插件状态：
$ kubectl get pods -n kube-system | grep calico
calico-node-7f5f8 1/1 Running 0 1d
calico-node-8d2k3 1/1 Running 0 1d
calico-node-9f5g7 1/1 Running 0 1d
– 检查Pod网络连通性：
$ kubectl run -it –rm –image=busybox:1.28 busybox — ping -c 4 10.244.1.2
PING 10.244.1.2 (10.244.1.2): 56 data bytes
64 bytes from 10.244.1.2: seq=0 ttl=64 time=0.500 ms
64 bytes from 10.244.1.2: seq=1 ttl=64 time=0.400 ms
64 bytes from 10.244.1.2: seq=2 ttl=64 time=0.300 ms
64 bytes from 10.244.1.2: seq=3 ttl=64 time=0.200 ms
— 10.244.1.2 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.200/0.350/0.500 ms
– 检查Service网络连通性：
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fgedu-app LoadBalancer 10.96.123.45 192.168.1.200 80:30080/TCP 1d
kubernetes ClusterIP 10.96.0.1
$ kubectl run -it –rm –image=busybox:1.28 busybox — wget -O- http://fgedu-app.default.svc.cluster.local
Connecting to fgedu-app.default.svc.cluster.local (10.96.123.45:80)
wget: server returned error: HTTP/1.1 503 Service Unavailable
– 检查网络策略：
$ kubectl get networkpolicies
NAME AGE
fgedu-app-network-policy 1d
$ kubectl describe networkpolicy fgedu-app-network-policy
Name: fgedu-app-network-policy
Namespace: default
Created on: 2024-01-01 00:00:00 +0000 UTC
Spec:
PodSelector: app=fgedu-app
Allowing ingress traffic:
To Port: 80/TCP
From:
PodSelector: app=fgedu-app
Allowing egress traffic:
To Port: 80/TCP
To:
PodSelector: app=fgedu-app
– 网络故障修复：
– 修改网络策略：
$ kubectl edit networkpolicy fgedu-app-network-policy
# 添加允许来自busybox的流量
ingress:
– from:
– podSelector: {}
ports:
– protocol: TCP
port: 80
– 验证网络连通性：
$ kubectl run -it –rm –image=busybox:1.28 busybox — wget -O- http://fgedu-app.default.svc.cluster.local
Connecting to fgedu-app.default.svc.cluster.local (10.96.123.45:80)
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Fri, 01 Jan 2024 00:00:00 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 14 Dec 2021 14:49:29 GMT
Connection: keep-alive
ETag: “61b8a129-267”
Accept-Ranges: bytes
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
Part04-生产案例与实战讲解
4.1 故障处理案例
生产环境Kubernetes故障处理的案例。
# 案例：控制平面故障
# 场景：API Server无响应，集群无法正常工作
# 问题：
– kubectl命令执行超时
– 集群状态无法查看
– 应用无法部署和管理
# 解决方案：
1. 检查API Server状态：
$ systemctl status kube-apiserver，学习交流加群风哥QQ113257174。
● kube-apiserver.service – Kubernetes API Server
Loaded: loaded (/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 00:00:00 UTC; 10s ago
Docs: https://kubernetes.io/docs/
Process: 1234 ExecStart=/usr/local/bin/kube-apiserver $KUBE_API_ARGS (code=exited, status=255)
Main PID: 1234 (code=exited, status=255)
2. 检查API Server日志：
$ journalctl -u kube-apiserver
Jan 01 00:00:00 fgedu-master kube-apiserver[1234]: E0101 00:00:00.123456 1234 storage_decorator.go:114] Unable to create storage backend: config (&{etcd3 {https://127.0.0.1:2379} /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/etcd/ca.crt true false 10s 1m0s 10s}), err (dial tcp 127.0.0.1:2379: connect: connection refused)
3. 检查etcd状态：
$ systemctl status etcd
● etcd.service – etcd
Loaded: loaded (/lib/systemd/system/etcd.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 00:00:00 UTC; 20s ago
Docs: https://github.com/coreos/etcd
Process: 5678 ExecStart=/usr/local/bin/etcd $ETCD_ARGS (code=exited, status=255)
Main PID: 5678 (code=exited, status=255)
4. 启动etcd：
$ systemctl start etcd
$ systemctl status etcd
● etcd.service – etcd
Loaded: loaded (/lib/systemd/system/etcd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 00:05:00 UTC; 5s ago
Docs: https://github.com/coreos/etcd
Process: 5678 ExecStart=/usr/local/bin/etcd $ETCD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 5678 (etcd)
5. 启动API Server：
$ systemctl start kube-apiserver
$ systemctl status kube-apiserver
● kube-apiserver.service – Kubernetes API Server
Loaded: loaded (/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 00:10:00 UTC; 5s ago
Docs: https://kubernetes.io/docs/
Process: 1234 ExecStart=/usr/local/bin/kube-apiserver $KUBE_API_ARGS (code=exited, status=0/SUCCESS)
Main PID: 1234 (kube-apiserver)
6. 验证集群状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready 1d v1.24.0
fgedu-node2 Ready 1d v1.24.0
# 案例：节点故障
# 场景：节点NotReady，Pod无法调度到该节点
# 问题：
– 节点状态为NotReady
– Pod无法调度到该节点
– 该节点上的Pod状态异常
# 解决方案：
1. 检查节点状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 NotReady 1d v1.24.0
fgedu-node2 Ready 1d v1.24.0
2. 检查节点详细信息：
$ kubectl describe node fgedu-node1
Name: fgedu-node1
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=fgedu-node1
kubernetes.io/os=linux
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 01 Jan 2024 00:00:00 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
—- —— —————– —————— —— ——-
Ready False Fri, 01 Jan 2024 01:00:00 +0000 Fri, 01 Jan 2024 01:00:00 +0000 KubeletNotReady PLEG is not healthy: pleg was last seen active 3m ago; threshold is 3m
3. 检查kubelet状态：
$ ssh fgedu-node1 systemctl status kubelet
● kubelet.service – kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 00:00:00 UTC; 1d ago
Docs: https://kubernetes.io/docs/
Main PID: 1234 (kubelet)
Tasks: 20
Memory: 100.0M
CPU: 10.0%
CGroup: /system.slice/kubelet.service
└─1234 /usr/bin/kubelet –bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf –kubeconfig=/etc/kubernetes/kubelet.conf –config=/var/lib/kubelet/config.yaml
4. 检查容器运行时状态：
$ ssh fgedu-node1 systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 01:00:00 UTC; 10s ago
Docs: https://docs.docker.com
Process: 5678 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock (code=exited, status=255)
Main PID: 5678 (code=exited, status=255)
5. 启动容器运行时：
$ ssh fgedu-node1 systemctl start docker
$ ssh fgedu-node1 systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 01:05:00 UTC; 5s ago
Docs: https://docs.docker.com
Process: 5678 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock (code=exited, status=0/SUCCESS)
Main PID: 5678 (dockerd)
6. 验证节点状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready 1d v1.24.0
fgedu-node2 Ready 1d v1.24.0
# 场景：API Server无响应，集群无法正常工作
# 问题：
– kubectl命令执行超时
– 集群状态无法查看
– 应用无法部署和管理
# 解决方案：
1. 检查API Server状态：
$ systemctl status kube-apiserver，学习交流加群风哥QQ113257174。
● kube-apiserver.service – Kubernetes API Server
Loaded: loaded (/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 00:00:00 UTC; 10s ago
Docs: https://kubernetes.io/docs/
Process: 1234 ExecStart=/usr/local/bin/kube-apiserver $KUBE_API_ARGS (code=exited, status=255)
Main PID: 1234 (code=exited, status=255)
2. 检查API Server日志：
$ journalctl -u kube-apiserver
Jan 01 00:00:00 fgedu-master kube-apiserver[1234]: E0101 00:00:00.123456 1234 storage_decorator.go:114] Unable to create storage backend: config (&{etcd3 {https://127.0.0.1:2379} /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/etcd/ca.crt true false 10s 1m0s 10s}), err (dial tcp 127.0.0.1:2379: connect: connection refused)
3. 检查etcd状态：
$ systemctl status etcd
● etcd.service – etcd
Loaded: loaded (/lib/systemd/system/etcd.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 00:00:00 UTC; 20s ago
Docs: https://github.com/coreos/etcd
Process: 5678 ExecStart=/usr/local/bin/etcd $ETCD_ARGS (code=exited, status=255)
Main PID: 5678 (code=exited, status=255)
4. 启动etcd：
$ systemctl start etcd
$ systemctl status etcd
● etcd.service – etcd
Loaded: loaded (/lib/systemd/system/etcd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 00:05:00 UTC; 5s ago
Docs: https://github.com/coreos/etcd
Process: 5678 ExecStart=/usr/local/bin/etcd $ETCD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 5678 (etcd)
5. 启动API Server：
$ systemctl start kube-apiserver
$ systemctl status kube-apiserver
● kube-apiserver.service – Kubernetes API Server
Loaded: loaded (/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 00:10:00 UTC; 5s ago
Docs: https://kubernetes.io/docs/
Process: 1234 ExecStart=/usr/local/bin/kube-apiserver $KUBE_API_ARGS (code=exited, status=0/SUCCESS)
Main PID: 1234 (kube-apiserver)
6. 验证集群状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready
fgedu-node2 Ready
# 案例：节点故障
# 场景：节点NotReady，Pod无法调度到该节点
# 问题：
– 节点状态为NotReady
– Pod无法调度到该节点
– 该节点上的Pod状态异常
# 解决方案：
1. 检查节点状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 NotReady
fgedu-node2 Ready
2. 检查节点详细信息：
$ kubectl describe node fgedu-node1
Name: fgedu-node1
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=fgedu-node1
kubernetes.io/os=linux
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 01 Jan 2024 00:00:00 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
—- —— —————– —————— —— ——-
Ready False Fri, 01 Jan 2024 01:00:00 +0000 Fri, 01 Jan 2024 01:00:00 +0000 KubeletNotReady PLEG is not healthy: pleg was last seen active 3m ago; threshold is 3m
3. 检查kubelet状态：
$ ssh fgedu-node1 systemctl status kubelet
● kubelet.service – kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 00:00:00 UTC; 1d ago
Docs: https://kubernetes.io/docs/
Main PID: 1234 (kubelet)
Tasks: 20
Memory: 100.0M
CPU: 10.0%
CGroup: /system.slice/kubelet.service
└─1234 /usr/bin/kubelet –bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf –kubeconfig=/etc/kubernetes/kubelet.conf –config=/var/lib/kubelet/config.yaml
4. 检查容器运行时状态：
$ ssh fgedu-node1 systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Fri 2024-01-01 01:00:00 UTC; 10s ago
Docs: https://docs.docker.com
Process: 5678 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock (code=exited, status=255)
Main PID: 5678 (code=exited, status=255)
5. 启动容器运行时：
$ ssh fgedu-node1 systemctl start docker
$ ssh fgedu-node1 systemctl status docker
● docker.service – Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-01-01 01:05:00 UTC; 5s ago
Docs: https://docs.docker.com
Process: 5678 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock (code=exited, status=0/SUCCESS)
Main PID: 5678 (dockerd)
6. 验证节点状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready
fgedu-node2 Ready
4.2 调试案例
生产环境Kubernetes调试的案例。。
# 案例：应用崩溃调试
# 场景：应用Pod频繁崩溃，状态为CrashLoopBackOff
# 问题：
– Pod状态为CrashLoopBackOff
– 应用日志显示错误信息
– 应用无法正常运行
# 解决方案：
1. 检查Pod状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 0/1 CrashLoopBackOff 5 10m
2. 检查Pod详细信息：
$ kubectl describe pod fgedu-app-6d6f58987b-7f5f8
Name: fgedu-app-6d6f58987b-7f5f8
Namespace: default
Priority: 0
Node: fgedu-node1/192.168.1.101
Start Time: Fri, 01 Jan 2024 00:00:00 +0000，更多视频教程www.fgedu.net.cn。
Labels: app=fgedu-app
pod-template-hash=6d6f58987b
Annotations:
Status: Running
IP: 10.244.1.2
IPs:
IP: 10.244.1.2
Controlled By: ReplicaSet/fgedu-app-6d6f58987b
Containers:
fgedu-app:
Container ID: docker://1234567890ab
Image: fgedu/app:latest
Image ID: docker-pullable://fgedu/app@sha256:1234567890ab
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 01 Jan 2024 00:05:00 +0000
Finished: Fri, 01 Jan 2024 00:05:30 +0000
Ready: False
Restart Count: 5
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xyz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-xyz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xyz
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal Scheduled 10m default-scheduler Successfully assigned default/fgedu-app-6d6f58987b-7f5f8 to fgedu-node1
Normal Pulling 10m kubelet Pulling image “fgedu/app:latest”
Normal Pulled 10m kubelet Successfully pulled image “fgedu/app:latest”
Normal Created 10m kubelet Created container fgedu-app
Normal Started 10m kubelet Started container fgedu-app
Normal Killing 9m (x5 over 10m) kubelet Container fgedu-app failed liveness probe, will be restarted
Warning Unhealthy 9m (x6 over 10m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 500
Warning BackOff 2m (x22 over 9m) kubelet Back-off restarting failed container
3. 检查应用日志：
$ kubectl logs fgedu-app-6d6f58987b-7f5f8
Error: Cannot connect to database: dial tcp 10.96.123.45:3306: connect: connection refused
panic: Cannot connect to database
goroutine 1 [running]:
main.main()
/app/main.go:100 +0x123
4. 检查数据库服务：
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fgedu-app LoadBalancer 10.96.123.45 192.168.1.200 80:30080/TCP 1d
kubernetes ClusterIP 10.96.0.1 443/TCP 1d
$ kubectl get pods -l app=mysql
No resources found in default namespace.
5. 部署数据库服务：
$ cat > mysql-deployment.yaml << 'EOF' apiVersion: apps/v1 kind: Deployment metadata: name: mysql namespace: default spec: replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0 ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD value: "fgedu123" - name: MYSQL_DATABASE value: "fgedudb" - name: MYSQL_USER value: "fgedu" - name: MYSQL_PASSWORD value: "fgedu123" --- apiVersion: v1 kind: Service metadata: name: mysql namespace: default spec: selector: app: mysql ports: - port: 3306 targetPort: 3306 EOF $ kubectl apply -f mysql-deployment.yaml 6. 验证数据库服务： $ kubectl get pods -l app=mysql NAME READY STATUS RESTARTS AGE mysql-6d6f58987b-7f5f8 1/1 Running 0 5m $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE fgedu-app LoadBalancer 10.96.123.45 192.168.1.200 80:30080/TCP 1d kubernetes ClusterIP 10.96.0.1 443/TCP 1d
mysql ClusterIP 10.96.123.46 3306/TCP 5m
7. 验证应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 1/1 Running 0 15m
mysql-6d6f58987b-7f5f8 1/1 Running 0 5m
# 案例：网络故障调试
# 场景：Pod之间无法通信，服务不可用，更多学习教程公众号风哥教程itpux_com。
# 问题：
– Pod之间无法ping通
– 服务无法访问
– 网络策略可能配置错误
# 解决方案：
1. 检查网络插件状态：
$ kubectl get pods -n kube-system | grep calico
calico-node-7f5f8 1/1 Running 0 1d
calico-node-8d2k3 1/1 Running 0 1d
calico-node-9f5g7 1/1 Running 0 1d
2. 检查Pod网络连通性：
$ kubectl run -it –rm –image=busybox:1.28 busybox — ping -c 4 10.244.1.2
PING 10.244.1.2 (10.244.1.2): 56 data bytes
— 10.244.1.2 ping statistics —
4 packets transmitted, 0 packets received, 100% packet loss
3. 检查网络策略：
$ kubectl get networkpolicies
NAME AGE
fgedu-app-network-policy 1d
$ kubectl describe networkpolicy fgedu-app-network-policy
Name: fgedu-app-network-policy
Namespace: default
Created on: 2024-01-01 00:00:00 +0000 UTC
Spec:
PodSelector: app=fgedu-app
Allowing ingress traffic:
To Port: 80/TCP
From:
PodSelector: app=fgedu-app
Allowing egress traffic:
To Port: 80/TCP
To:
PodSelector: app=fgedu-app
4. 修改网络策略：
$ kubectl edit networkpolicy fgedu-app-network-policy
# 添加允许来自所有Pod的流量
ingress:
– from:
– podSelector: {}
ports:
– protocol: TCP
port: 80
5. 验证网络连通性：
$ kubectl run -it –rm –image=busybox:1.28 busybox — ping -c 4 10.244.1.2
PING 10.244.1.2 (10.244.1.2): 56 data bytes
64 bytes from 10.244.1.2: seq=0 ttl=64 time=0.500 ms
64 bytes from 10.244.1.2: seq=1 ttl=64 time=0.400 ms
64 bytes from 10.244.1.2: seq=2 ttl=64 time=0.300 ms
64 bytes from 10.244.1.2: seq=3 ttl=64 time=0.200 ms
— 10.244.1.2 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.200/0.350/0.500 ms
$ kubectl run -it –rm –image=busybox:1.28 busybox — wget -O- http://fgedu-app.default.svc.cluster.local
Connecting to fgedu-app.default.svc.cluster.local (10.96.123.45:80)
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Fri, 01 Jan 2024 00:00:00 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 14 Dec 2021 14:49:29 GMT
Connection: keep-alive
ETag: “61b8a129-267”
Accept-Ranges: bytes
# 场景：应用Pod频繁崩溃，状态为CrashLoopBackOff
# 问题：
– Pod状态为CrashLoopBackOff
– 应用日志显示错误信息
– 应用无法正常运行
# 解决方案：
1. 检查Pod状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 0/1 CrashLoopBackOff 5 10m
2. 检查Pod详细信息：
$ kubectl describe pod fgedu-app-6d6f58987b-7f5f8
Name: fgedu-app-6d6f58987b-7f5f8
Namespace: default
Priority: 0
Node: fgedu-node1/192.168.1.101
Start Time: Fri, 01 Jan 2024 00:00:00 +0000，更多视频教程www.fgedu.net.cn。
Labels: app=fgedu-app
pod-template-hash=6d6f58987b
Annotations:
Status: Running
IP: 10.244.1.2
IPs:
IP: 10.244.1.2
Controlled By: ReplicaSet/fgedu-app-6d6f58987b
Containers:
fgedu-app:
Container ID: docker://1234567890ab
Image: fgedu/app:latest
Image ID: docker-pullable://fgedu/app@sha256:1234567890ab
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 01 Jan 2024 00:05:00 +0000
Finished: Fri, 01 Jan 2024 00:05:30 +0000
Ready: False
Restart Count: 5
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xyz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-xyz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xyz
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal Scheduled 10m default-scheduler Successfully assigned default/fgedu-app-6d6f58987b-7f5f8 to fgedu-node1
Normal Pulling 10m kubelet Pulling image “fgedu/app:latest”
Normal Pulled 10m kubelet Successfully pulled image “fgedu/app:latest”
Normal Created 10m kubelet Created container fgedu-app
Normal Started 10m kubelet Started container fgedu-app
Normal Killing 9m (x5 over 10m) kubelet Container fgedu-app failed liveness probe, will be restarted
Warning Unhealthy 9m (x6 over 10m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 500
Warning BackOff 2m (x22 over 9m) kubelet Back-off restarting failed container
3. 检查应用日志：
$ kubectl logs fgedu-app-6d6f58987b-7f5f8
Error: Cannot connect to database: dial tcp 10.96.123.45:3306: connect: connection refused
panic: Cannot connect to database
goroutine 1 [running]:
main.main()
/app/main.go:100 +0x123
4. 检查数据库服务：
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fgedu-app LoadBalancer 10.96.123.45 192.168.1.200 80:30080/TCP 1d
kubernetes ClusterIP 10.96.0.1
$ kubectl get pods -l app=mysql
No resources found in default namespace.
5. 部署数据库服务：
$ cat > mysql-deployment.yaml << 'EOF' apiVersion: apps/v1 kind: Deployment metadata: name: mysql namespace: default spec: replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:8.0 ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD value: "fgedu123" - name: MYSQL_DATABASE value: "fgedudb" - name: MYSQL_USER value: "fgedu" - name: MYSQL_PASSWORD value: "fgedu123" --- apiVersion: v1 kind: Service metadata: name: mysql namespace: default spec: selector: app: mysql ports: - port: 3306 targetPort: 3306 EOF $ kubectl apply -f mysql-deployment.yaml 6. 验证数据库服务： $ kubectl get pods -l app=mysql NAME READY STATUS RESTARTS AGE mysql-6d6f58987b-7f5f8 1/1 Running 0 5m $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE fgedu-app LoadBalancer 10.96.123.45 192.168.1.200 80:30080/TCP 1d kubernetes ClusterIP 10.96.0.1
mysql ClusterIP 10.96.123.46
7. 验证应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 1/1 Running 0 15m
mysql-6d6f58987b-7f5f8 1/1 Running 0 5m
# 案例：网络故障调试
# 场景：Pod之间无法通信，服务不可用，更多学习教程公众号风哥教程itpux_com。
# 问题：
– Pod之间无法ping通
– 服务无法访问
– 网络策略可能配置错误
# 解决方案：
1. 检查网络插件状态：
$ kubectl get pods -n kube-system | grep calico
calico-node-7f5f8 1/1 Running 0 1d
calico-node-8d2k3 1/1 Running 0 1d
calico-node-9f5g7 1/1 Running 0 1d
2. 检查Pod网络连通性：
$ kubectl run -it –rm –image=busybox:1.28 busybox — ping -c 4 10.244.1.2
PING 10.244.1.2 (10.244.1.2): 56 data bytes
— 10.244.1.2 ping statistics —
4 packets transmitted, 0 packets received, 100% packet loss
3. 检查网络策略：
$ kubectl get networkpolicies
NAME AGE
fgedu-app-network-policy 1d
$ kubectl describe networkpolicy fgedu-app-network-policy
Name: fgedu-app-network-policy
Namespace: default
Created on: 2024-01-01 00:00:00 +0000 UTC
Spec:
PodSelector: app=fgedu-app
Allowing ingress traffic:
To Port: 80/TCP
From:
PodSelector: app=fgedu-app
Allowing egress traffic:
To Port: 80/TCP
To:
PodSelector: app=fgedu-app
4. 修改网络策略：
$ kubectl edit networkpolicy fgedu-app-network-policy
# 添加允许来自所有Pod的流量
ingress:
– from:
– podSelector: {}
ports:
– protocol: TCP
port: 80
5. 验证网络连通性：
$ kubectl run -it –rm –image=busybox:1.28 busybox — ping -c 4 10.244.1.2
PING 10.244.1.2 (10.244.1.2): 56 data bytes
64 bytes from 10.244.1.2: seq=0 ttl=64 time=0.500 ms
64 bytes from 10.244.1.2: seq=1 ttl=64 time=0.400 ms
64 bytes from 10.244.1.2: seq=2 ttl=64 time=0.300 ms
64 bytes from 10.244.1.2: seq=3 ttl=64 time=0.200 ms
— 10.244.1.2 ping statistics —
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.200/0.350/0.500 ms
$ kubectl run -it –rm –image=busybox:1.28 busybox — wget -O- http://fgedu-app.default.svc.cluster.local
Connecting to fgedu-app.default.svc.cluster.local (10.96.123.45:80)
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Fri, 01 Jan 2024 00:00:00 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 14 Dec 2021 14:49:29 GMT
Connection: keep-alive
ETag: “61b8a129-267”
Accept-Ranges: bytes
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
4.3 恢复案例
生产环境Kubernetes恢复的案例。
# 案例：集群恢复
# 场景：集群崩溃，需要恢复到之前的状态
# 问题：
– 集群无法正常工作
– 控制平面组件故障
– 数据可能丢失
# 解决方案：
1. 备份etcd数据：
$ kubectl exec -it etcd-fgedu-master -n kube-system — etcdctl –endpoints=https://127.0.0.1:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcd-snapshot.db
2. 停止控制平面组件：
$ systemctl stop kube-apiserver kube-controller-manager kube-scheduler etcd
3. 恢复etcd数据：
$ etcdctl snapshot restore /tmp/etcd-snapshot.db –data-dir=/var/lib/etcd
4. 启动控制平面组件：
$ systemctl start etcd kube-apiserver kube-controller-manager kube-scheduler
5. 验证集群状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready 1d v1.24.0
fgedu-node2 Ready 1d v1.24.0
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 1/1 Running 0 1d
fgedu-app-6d6f58987b-8d2k3 1/1 Running 0 1d
fgedu-app-6d6f58987b-9f5g7 1/1 Running 0 1d
# 案例：应用恢复
# 场景：应用崩溃，需要恢复到之前的状态
# 问题：
– 应用Pod崩溃
– 应用数据可能丢失
– 应用服务不可用
# 解决方案：
1. 检查应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 0/1 CrashLoopBackOff 5 10m，from K8S+DB视频:www.itpux.com。
2. 检查应用日志：
$ kubectl logs fgedu-app-6d6f58987b-7f5f8
Error: Cannot connect to database: dial tcp 10.96.123.45:3306: connect: connection refused
3. 检查数据库服务：
$ kubectl get pods -l app=mysql
NAME READY STATUS RESTARTS AGE
mysql-6d6f58987b-7f5f8 1/1 Running 0 1d
4. 检查数据库连接：
$ kubectl exec -it mysql-6d6f58987b-7f5f8 — mysql -u fgedu -pfgedu123 fgedudb -e “SELECT 1;”
+—+。
| 1 |
+—+
| 1 |
+—+
5. 检查应用配置：
$ kubectl get configmap fgedu-app-config -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fgedu-app-config
namespace: default
data:
database_url: mysql://fgedu:fgedu123@mysql:3306/fgedudb
6. 修复应用配置：
$ kubectl edit configmap fgedu-app-config
# 修改database_url为正确的地址
database_url: mysql://fgedu:fgedu123@mysql.default.svc.cluster.local:3306/fgedudb
7. 重启应用：
$ kubectl rollout restart deployment fgedu-app
8. 验证应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 1/1 Running 0 15m
mysql-6d6f58987b-7f5f8 1/1 Running 0 1d
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fgedu-app LoadBalancer 10.96.123.45 192.168.1.200 80:30080/TCP 1d
kubernetes ClusterIP 10.96.0.1 443/TCP 1d
mysql ClusterIP 10.96.123.46 3306/TCP 1d
$ curl http://192.168.1.200:80
# 场景：集群崩溃，需要恢复到之前的状态
# 问题：
– 集群无法正常工作
– 控制平面组件故障
– 数据可能丢失
# 解决方案：
1. 备份etcd数据：
$ kubectl exec -it etcd-fgedu-master -n kube-system — etcdctl –endpoints=https://127.0.0.1:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcd-snapshot.db
2. 停止控制平面组件：
$ systemctl stop kube-apiserver kube-controller-manager kube-scheduler etcd
3. 恢复etcd数据：
$ etcdctl snapshot restore /tmp/etcd-snapshot.db –data-dir=/var/lib/etcd
4. 启动控制平面组件：
$ systemctl start etcd kube-apiserver kube-controller-manager kube-scheduler
5. 验证集群状态：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fgedu-master Ready control-plane,master 1d v1.24.0
fgedu-node1 Ready
fgedu-node2 Ready
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 1/1 Running 0 1d
fgedu-app-6d6f58987b-8d2k3 1/1 Running 0 1d
fgedu-app-6d6f58987b-9f5g7 1/1 Running 0 1d
# 案例：应用恢复
# 场景：应用崩溃，需要恢复到之前的状态
# 问题：
– 应用Pod崩溃
– 应用数据可能丢失
– 应用服务不可用
# 解决方案：
1. 检查应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 0/1 CrashLoopBackOff 5 10m，from K8S+DB视频:www.itpux.com。
2. 检查应用日志：
$ kubectl logs fgedu-app-6d6f58987b-7f5f8
Error: Cannot connect to database: dial tcp 10.96.123.45:3306: connect: connection refused
3. 检查数据库服务：
$ kubectl get pods -l app=mysql
NAME READY STATUS RESTARTS AGE
mysql-6d6f58987b-7f5f8 1/1 Running 0 1d
4. 检查数据库连接：
$ kubectl exec -it mysql-6d6f58987b-7f5f8 — mysql -u fgedu -pfgedu123 fgedudb -e “SELECT 1;”
+—+。
| 1 |
+—+
| 1 |
+—+
5. 检查应用配置：
$ kubectl get configmap fgedu-app-config -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fgedu-app-config
namespace: default
data:
database_url: mysql://fgedu:fgedu123@mysql:3306/fgedudb
6. 修复应用配置：
$ kubectl edit configmap fgedu-app-config
# 修改database_url为正确的地址
database_url: mysql://fgedu:fgedu123@mysql.default.svc.cluster.local:3306/fgedudb
7. 重启应用：
$ kubectl rollout restart deployment fgedu-app
8. 验证应用状态：
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
fgedu-app-6d6f58987b-7f5f8 1/1 Running 0 15m
mysql-6d6f58987b-7f5f8 1/1 Running 0 1d
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fgedu-app LoadBalancer 10.96.123.45 192.168.1.200 80:30080/TCP 1d
kubernetes ClusterIP 10.96.0.1
mysql ClusterIP 10.96.123.46
$ curl http://192.168.1.200:80
Welcome to fgedu app!
If you see this page, the fgedu app is successfully installed and
working. Further configuration is required.
Part05-风哥经验总结与分享
5.1 故障处理最佳实践
Kubernetes故障处理的最佳实践。
- 预防为主：定期检查集群状态，及时发现潜在问题，实施监控和告警
- 快速响应：建立故障响应机制，快速定位故障原因，优先恢复服务
- 根因分析：深入分析故障原因，避免类似问题再次发生，实施修复措施
- 团队协作：建立跨团队协作机制，明确沟通渠道，分享故障处理经验
- 工具支持：使用监控工具、日志分析工具、诊断工具等，提高故障处理效率
- 文档和流程：建立故障处理流程，记录故障处理过程，编写故障处理文档
- 持续改进：定期回顾故障处理经验，持续改进故障处理流程和方法
- 培训和知识共享：定期培训团队成员，共享故障处理知识和经验
5.2 调试最佳实践
Kubernetes调试的最佳实践：
- 系统排查：从控制平面、节点、应用、网络等多个方面进行排查
- 日志分析：查看容器日志、系统日志、应用日志，寻找错误信息
- 网络诊断：使用ping、curl、netstat等工具测试网络连通性
- 资源监控：检查CPU、内存、存储等资源使用情况，寻找资源瓶颈
- 配置检查：检查应用配置、网络配置、存储配置等，寻找配置错误
- 版本兼容性：检查Kubernetes版本、容器镜像版本等，确保兼容性
- 安全检查：检查RBAC权限、网络策略、Pod安全策略等，确保安全性
- 测试验证：在修复后进行测试验证，确保故障已解决
5.3 未来趋势
Kubernetes故障处理与调试的未来趋势：
- 自动化故障处理：使用AI和机器学习技术，实现自动化故障检测和处理
- 智能诊断：基于历史数据和模式识别，智能诊断故障原因
- 预测性维护：通过分析系统状态和趋势，预测可能的故障，提前进行维护
- 边缘计算故障处理：将故障处理扩展到边缘设备，支持边缘计算场景
- 多云故障处理：支持跨云平台的故障处理，实现统一的故障处理机制
- 服务网格集成：集成服务网格，提供更细粒度的故障处理和调试能力
- 零信任架构：基于零信任架构，提高故障处理的安全性和可靠性
- 可视化调试：使用更先进的可视化工具，直观展示系统状态和故障原因
持续学习：故障处理与调试技术在不断发展，需要持续学习和掌握新的技术和方法，以适应业务需求的变化。
本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html
