Linux教程FG479-Kubernetes故障排查进阶

内容简介：本文风哥教程参考Linux官方文档、Red Hat Enterprise Linux官方文档、Ansible Automation Platform官方文档、Docker官方文档、Kubernetes官方文档和Podman官方文档等内容，详细介绍了相关技术的配置和使用方法。

本文档介绍

风哥提示：

Kubernetes故障排查进阶实战案例。

Part01-常见故障场景

1.1 Pod启动失败排查

# 场景1: ImagePullBackOff
[root@k8s-master ~]# kubectl get pods -n fgedu-prod
NAME READY STATUS RESTARTS AGE
fgedu-app-abc12 0/1 ImagePullBackOff 0 5m

# 排查步骤
[root@k8s-master ~]# kubectl describe pod fgedu-app-abc12 -n fgedu-prod
Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal Pulling 5m (x4 over 5m) kubelet Pulling image “fgedu/app:v1.0”
Warning Failed 5m (x4 over 5m) kubelet Failed to pull image “fgedu/app:v1.0”: rpc error: code = Unknown desc = Error response from daemon: pull access denied

# 解决方案
[root@k8s-master ~]# kubectl create secret docker-registry fgedu-registry \
–docker-server=192.168.1.100:30002 \
–docker-username=admin \
–docker-password=Harbor12345 \
-n fgedu-prod
secret/fgedu-registry created

[root@k8s-master ~]# kubectl patch serviceaccount default -n fgedu-prod -p ‘{“imagePullSecrets”:[{“name”:”fgedu-registry”}]}’
serviceaccount/default patched

# 场景2: CrashLoopBackOff
[root@k8s-master ~]# kubectl get pods -n fgedu-prod
NAME READY STATUS RESTARTS AGE
fgedu-app-def34 0/1 CrashLoopBackOff 5 10m

# 排查步骤
[root@k8s-master ~]# kubectl logs fgedu-app-def34 -n fgedu-prod –previous
2026-04-04 23:00:00 ERROR Failed to connect to database: Connection refused
2026-04-04 23:00:00 ERROR Application startup failed
2026-04-04 23:00:00 INFO Shutting down…

# 解决方案: 检查依赖服务
[root@k8s-master ~]# kubectl get svc -n fgedu-prod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fgedu-mysql ClusterIP 10.96.100.100 3306/TCP 10m

[root@k8s-master ~]# kubectl get pods -n fgedu-prod -l app=fgedu-mysql
NAME READY STATUS RESTARTS AGE
fgedu-mysql-0 0/1 Running 0 5m

# 场景3: Pending状态
[root@k8s-master ~]# kubectl get pods -n fgedu-prod
NAME READY STATUS RESTARTS AGE
fgedu-app-ghi56 0/1 Pending 0 10m

# 排查步骤
[root@k8s-master ~]# kubectl describe pod fgedu-app-ghi56 -n fgedu-prod
Events:
Type Reason Age From Message
—- —— —- —- ——-
Warning FailedScheduling 10m default-scheduler 0/3 nodes are available: 3 Insufficient cpu.

# 解决方案: 检查资源
[root@k8s-master ~]# kubectl describe nodes | grep -A 5 “Allocated resources”
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
——– ——– ——
cpu 3500m (87%) 7000m (175%)
memory 14Gi (87%) 28Gi (175%)

Part02-网络故障排查

2.1 网络连通性排查

# 场景: 服务无法访问
[root@k8s-master ~]# kubectl exec -it fgedu-app-abc12 -n fgedu-prod — curl http://fgedu-svc
curl: (6) Could not resolve host: fgedu-svc

# 排查DNS
[root@k8s-master ~]# kubectl exec -it fgedu-app-abc12 -n fgedu-prod — nslookup fgedu-svc.fgedu-prod
Server: 10.96.0.10
Address 1: 10.96.0.10

** server can’t find fgedu-svc.fgedu-prod: NXDOMAIN

# 检查CoreDNS
[root@k8s-master ~]# kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-abc12-xyz789 1/1 Running 0 10d
coredns-def34-xyz789 1/1 Running 0 10d

[root@k8s-master ~]# kubectl logs -n kube-system coredns-abc12-xyz789
.:53
[INFO] plugin/reload: Running configuration SHA512 = abc123
[INFO] CoreDNS-1.11.1
[INFO] linux/amd64, go1.20.4, abc123

# 检查Service和Endpoints
[root@k8s-master ~]# kubectl get svc -n fgedu-prod fgedu-svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fgedu-svc ClusterIP 10.96.100.100 80/TCP 5m

[root@k8s-master ~]# kubectl get endpoints -n fgedu-prod fgedu-svc
NAME ENDPOINTS AGE
fgedu-svc 5m

# 解决方案: 检查Pod标签
[root@k8s-master ~]# kubectl get pods -n fgedu-prod –show-labels
NAME READY STATUS LABELS
fgedu-app-abc12 1/1 Running app=fgedu-app

[root@k8s-master ~]# kubectl get svc fgedu-svc -n fgedu-prod -o yaml | grep selector -A 2
selector:
app: fgedu-web

# 修复Service选择器
[root@k8s-master ~]# kubectl patch svc fgedu-svc -n fgedu-prod -p ‘{“spec”:{“selector”:{“app”:”fgedu-app”}}}’
service/fgedu-svc patched

# 验证修复
[root@k8s-master ~]# kubectl get endpoints -n fgedu-prod fgedu-svc
NAME ENDPOINTS AGE
fgedu-svc 10.244.1.10:80,10.244.2.10:80 1m

Part03-存储故障排查

3.1 PVC绑定失败

# 场景: PVC一直Pending
[root@k8s-master ~]# kubectl get pvc -n fgedu-prod
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
fgedu-data-pvc Pending fgedu-nfs-storage 10m

# 排查步骤
[root@k8s-master ~]# kubectl describe pvc fgedu-data-pvc -n fgedu-prod
Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal WaitForFirstConsumer 10m persistentvolume-controller waiting for first consumer to be created before binding

# 检查StorageClass
[root@k8s-master ~]# kubectl get storageclass
NAME PROVISIONER AGE
fgedu-nfs-storage nfs.fgedu.net.cn 10d

# 检查Provisioner
[root@k8s-master ~]# kubectl get pods -n kube-system -l app=nfs-provisioner
NAME READY STATUS RESTARTS AGE
nfs-provisioner-abc12-xyz789 0/1 Error 5 10m

[root@k8s-master ~]# kubectl logs -n kube-system nfs-provisioner-abc12-xyz789
E0423 23:00:00.000000 1 provisioner.go:123] Failed to provision volume: mount failed: exit status 32

# 检查NFS服务器
[root@k8s-master ~]# showmount -e 192.168.1.100
Export list for 192.168.1.100:
/data/k8s-storage 192.168.1.0/24

# 解决方案: 重启Provisioner
[root@k8s-master ~]# kubectl rollout restart deployment nfs-provisioner -n kube-system
deployment.apps/nfs-provisioner restarted

# 验证PVC绑定
[root@k8s-master ~]# kubectl get pvc -n fgedu-prod
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
fgedu-data-pvc Bound pvc-abc123def456-789 10Gi RWX fgedu-nfs-storage 15m

Part04-节点故障排查

4.1 节点NotReady处理

# 场景: 节点NotReady
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane 100d v1.28.3
k8s-node1 NotReady 100d v1.28.3
k8s-node2 Ready 100d v1.28.3

# 排查步骤
[root@k8s-master ~]# kubectl describe node k8s-node1
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
—- —— —————– —————— —— ——-
MemoryPressure False Sat, 04 Apr 2026 23:00:00 +0800 Sat, 04 Apr 2026 22:00:00 +0800 KubeletHasSufficientMemory kubelet has sufficient memory
DiskPressure False Sat, 04 Apr 2026 23:00:00 +0800 Sat, 04 Apr 2026 22:00:00 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sat, 04 Apr 2026 23:00:00 +0800 Sat, 04 Apr 2026 22:00:00 +0800 KubeletHasSufficientPID kubelet has sufficient PID
Ready False Sat, 04 Apr 2026 23:00:00 +0800 Sat, 04 Apr 2026 22:30:00 +0800 KubeletNotReady container runtime not ready

# 登录节点检查
[root@k8s-node1 ~]# systemctl status kubelet
● kubelet.service – Kubernetes Kubelet Server
Active: inactive (dead)
Docs: https://kubernetes.io/docs/

[root@k8s-node1 ~]# journalctl -u kubelet -n 50
Apr 04 23:00:00 k8s-node1 kubelet[1学习交流加群风哥QQ1132571742345]: E0423 23:00:00.000000 12345 kubelet.go:1234] “Failed to start cAdvisor” err=”inotify watch limit reached”

# 解决方案: 增加inotify限制
[root@k8s-node1 ~]# echo “fs.inotify.max_user_watches=524288” >> /etc/sysctl.conf
[root@k8s-node1 ~]# sysctl -p
fs.inotify.max_user_watches = 524288

[root@k8s-node1 ~]# systemctl restart kubelet

# 验证节点恢复
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready control学习交流加群风哥微信: itpux-com-plane 100d v1.28.3
k8s-node1 Ready 100d v1.28.3
k8s-node2 Ready 100d v1.28.3

风哥针对故障排查建议：

使用describe查看详细事件
检查日志定位根本原因
验证资源配置是否正确
检查网络连通性
验证存储配置

本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html