IT教程FG307-Kubernetes集群管理

1. Kubernetes集群概述

Kubernetes是一个开源的容器编排平台，用于自动化部署、扩展和管理容器化应用程序。更多学习教程www.fgedu.net.cn

# 查看Kubernetes版本
# kubectl version
Client Version: version.Info{Major:”1″, Minor:”28″, GitVersion:”v1.28.0″, GitCommit:”8f94981b5ce1c0c89a20d69b9e1e8fc7c19b553a”, GitTreeState:”clean”, BuildDate:”2026-03-15T12:00:00Z”, GoVersion:”go1.21.0″, Compiler:”gc”, Platform:”linux/amd64″}
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: version.Info{Major:”1″, Minor:”28″, GitVersion:”v1.28.0″, GitCommit:”8f94981b5ce1c0c89a20d69b9e1e8fc7c19b553a”, GitTreeState:”clean”, BuildDate:”2026-03-15T12:00:00Z”, GoVersion:”go1.21.0″, Compiler:”gc”, Platform:”linux/amd64″}

# 查看集群信息
# kubectl cluster-info
Kubernetes control plane is running at https://fgedu.net.cn:6443
CoreDNS is running at https://fgedu.net.cn:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump’.

# 查看集群详细信息
# kubectl cluster-info dump | head -50
{
“kind”: “NodeList”,
“apiVersion”: “v1”,
“metadata”: {
“resourceVersion”: “1234567”
},
“items”: [
{
“metadata”: {
“name”: “master01.fgedu.net.cn”,
“uid”: “abc123-def456-ghi789”,
“resourceVersion”: “1234560”,
“creationTimestamp”: “2026-01-01T00:00:00Z”,
“labels”: {
“beta.kubernetes.io/arch”: “amd64”,
“beta.kubernetes.io/os”: “linux”,
“kubernetes.io/arch”: “amd64”,
“kubernetes.io/hostname”: “master01”,
“kubernetes.io/os”: “linux”,
“node-role.kubernetes.io/control-plane”: “”,
“node-role.kubernetes.io/master”: “”
}
}
}
]
}

生产环境风哥建议：生产环境建议使用3个及以上Master节点实现高可用，etcd数据需要定期备份，确保集群版本一致性。

2. 集群状态检查

定期检查集群状态是确保服务稳定运行的关键。学习交流加群风哥微信: itpux-com

# 查看所有节点状态
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master01.fgedu.net.cn Ready control-plane 90d v1.28.0 192.168.1.10 Oracle Linux Server 8.9 5.4.17-2136.302.7.2.el7uek.x86_64 containerd://1.7.0
master02.fgedu.net.cn Ready control-plane 90d v1.28.0 192.168.1.11 Oracle Linux Server 8.9 5.4.17-2136.302.7.2.el7uek.x86_64 containerd://1.7.0
master03.fgedu.net.cn Ready control-plane 90d v1.28.0 192.168.1.12 Oracle Linux Server 8.9 5.4.17-2136.302.7.2.el7uek.x86_64 containerd://1.7.0
worker01.fgedu.net.cn Ready 90d v1.28.0 192.168.1.20 Oracle Linux Server 8.9 5.4.17-2136.302.7.2.el7uek.x86_64 containerd://1.7.0
worker02.fgedu.net.cn Ready 90d v1.28.0 192.168.1.21 Oracle Linux Server 8.9 5.4.17-2136.302.7.2.el7uek.x86_64 containerd://1.7.0
worker03.fgedu.net.cn Ready 90d v1.28.0 192.168.1.22 Oracle Linux Server 8.9 5.4.17-2136.302.7.2.el7uek.x86_64 containerd://1.7.0

# 查看节点详细信息
# kubectl describe node master01.fgedu.net.cn
Name: master01.fgedu.net.cn
Roles: control-plane
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=master01
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 01 Jan 2026 00:00:00 +0800
Taints: node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable: false

# 查看组件状态
# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {“health”:”true”,”reason”:””}
etcd-1 Healthy {“health”:”true”,”reason”:””}
etcd-2 Healthy {“health”:”true”,”reason”:””}

# 查看所有命名空间的Pod
# kubectl get pods –all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5d78c9869d-abc12 1/1 Running 0 90d
kube-system coredns-5d78c9869d-def34 1/1 Running 0 90d
kube-system etcd-master01.fgedu.net.cn 1/1 Running 0 90d
kube-system etcd-master02.fgedu.net.cn 1/1 Running 0 90d
kube-system etcd-master03.fgedu.net.cn 1/1 Running 0 90d
kube-system kube-apiserver-master01.fgedu.net.cn 1/1 Running 0 90d
kube-system kube-apiserver-master02.fgedu.net.cn 1/1 Running 0 90d
kube-system kube-apiserver-master03.fgedu.net.cn 1/1 Running 0 90d
kube-system kube-controller-manager-master01.fgedu.net.cn 1/1 Running 1 90d
kube-system kube-controller-manager-master02.fgedu.net.cn 1/1 Running 0 90d
kube-system kube-controller-manager-master03.fgedu.net.cn 1/1 Running 0 90d
kube-system kube-proxy-abcde 1/1 Running 0 90d
kube-system kube-scheduler-master01.fgedu.net.cn 1/1 Running 0 90d
kube-system kube-scheduler-master02.fgedu.net.cn 1/1 Running 0 90d
kube-system kube-scheduler-master03.fgedu.net.cn 1/1 Running 0 90d

# 查看集群资源使用情况
# kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master01.fgedu.net.cn 500m 12% 2048Mi 25%
master02.fgedu.net.cn 450m 11% 1980Mi 24%
master03.fgedu.net.cn 480m 12% 2010Mi 25%
worker01.fgedu.net.cn 1200m 15% 4096Mi 25%
worker02.fgedu.net.cn 1100m 14% 3840Mi 23%
worker03.fgedu.net.cn 1150m 14% 3968Mi 24%

# 查看集群资源概览
# kubectl api-resources | head -20
NAME SHORTNAMES APIGROUP NAMESPACED KIND
bindings true Binding
componentstatuses cs false ComponentStatus
configmaps cm true ConfigMap
endpoints ep true Endpoints
events ev true Event
limitranges limits true LimitRange
namespaces ns false Namespace
nodes no false Node
persistentvolumeclaims pvc true PersistentVolumeClaim
persistentvolumes pv false PersistentVolume
pods po true Pod
podtemplates true PodTemplate
replicationcontrollers rc true ReplicationController
resourcequotas quota true ResourceQuota
secrets true Secret
serviceaccounts sa true ServiceAccount
services svc true Service

3. 节点管理

节点管理包括节点的添加、删除、维护和故障处理等操作。

# 查看节点标签
# kubectl get nodes –show-labels
NAME STATUS ROLES AGE VERSION LABELS
master01.fgedu.net.cn Ready control-plane 90d v1.28.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master01,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=

# 给节点添加标签
# kubectl label node worker01.fgedu.net.cn env=production
node/worker01.fgedu.net.cn labeled

# kubectl label node worker01.fgedu.net.cn zone=beijing
node/worker01.fgedu.net.cn labeled

# 验证标签
# kubectl get node worker01.fgedu.net.cn –show-labels | grep env
node/worker01.fgedu.net.cn Ready 90d v1.28.0 …,env=production,zone=beijing,…

# 删除节点标签
# kubectl label node worker01.fgedu.net.cn zone-
node/worker01.fgedu.net.cn labeled

# 驱逐节点上的Pod（维护模式）
# kubectl cordon worker01.fgedu.net.cn
node/worker01.fgedu.net.cn cordoned

# 查看节点状态
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01.fgedu.net.cn Ready control-plane 90d v1.28.0
worker01.fgedu.net.cn Ready,SchedulingDisabled 90d v1.28.0
worker02.fgedu.net.cn Ready 90d v1.28.0
worker03.fgedu.net.cn Ready 90d v1.28.0

# 驱逐节点上的所有Pod
# kubectl drain worker01.fgedu.net.cn –ignore-daemonsets –delete-emptydir-data
node/worker01.fgedu.net.cn already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-abcde, kube-system/weave-net-def45
evicting pod kube-system/coredns-5d78c9869d-abc12
evicting pod default/nginx-deployment-5678-xyz12
pod/nginx-deployment-5678-xyz12 evicted
pod/coredns-5d78c9869d-abc12 evicted
node/worker01.fgedu.net.cn drained

# 恢复节点调度
# kubectl uncordon worker01.fgedu.net.cn
node/worker01.fgedu.net.cn uncordoned

# 添加污点（Taint）
# kubectl taint nodes worker01.fgedu.net.cn dedicated=gpu:NoSchedule
node/worker01.fgedu.net.cn tainted

# 查看节点污点
# kubectl describe node worker01.fgedu.net.cn | grep Taints
Taints: dedicated=gpu:NoSchedule

# 删除污点
# kubectl taint nodes worker01.fgedu.net.cn dedicated=gpu:NoSchedule-
node/worker01.fgedu.net.cn untainted

# 查看节点资源分配
# kubectl describe node worker01.fgedu.net.cn | grep -A 10 “Allocated resources”
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
——– ——– ——
cpu 1800m (22%) 3600m (45%)
memory 4096Mi (25%) 8192Mi (50%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)

风哥风哥提示：在对节点进行维护前，务必使用kubectl drain命令安全地驱逐Pod，确保服务不中断。对于有状态应用，需要确保数据已持久化。

4. 命名空间管理

命名空间用于在集群中实现资源隔离，是Kubernetes多租户管理的基础。学习交流加群风哥QQ113257174

# 查看所有命名空间
# kubectl get namespaces
NAME STATUS AGE
default Active 90d
kube-node-lease Active 90d
kube-public Active 90d
kube-system Active 90d
production Active 60d
staging Active 60d
development Active 60d

# 创建命名空间
# kubectl create namespace fgedu-prod
namespace/fgedu-prod created

# 使用YAML文件创建命名空间
# cat <
Status: Active

No resource quota.

No LimitRange resource.

# 在命名空间中部署应用
# kubectl create deployment nginx –image=nginx:1.24 -n fgedu-prod
deployment.apps/nginx created

# 查看命名空间中的资源
# kubectl get all -n fgedu-prod
NAME READY STATUS RESTARTS AGE
pod/nginx-6799fc88d8-abc12 1/1 Running 0 30s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 30s

NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-6799fc88d8 1 1 1 30s

# 删除命名空间
# kubectl delete namespace fgedu-test
namespace “fgedu-test” deleted

# 设置默认命名空间
# kubectl config set-context –current –namespace=fgedu-prod
Context “kubernetes-admin@kubernetes” modified.

# 验证当前命名空间
# kubectl config view | grep namespace
namespace: fgedu-prod

5. 资源配额管理

资源配额用于限制命名空间的资源使用，防止单个租户占用过多资源。更多学习教程公众号风哥教程itpux_com

# 创建资源配额
# cat <

# 创建LimitRange限制Pod资源
# cat <

生产环境风哥建议：为每个命名空间设置合理的资源配额，结合LimitRange设置默认资源限制，防止资源浪费和过度使用。

6. 集群升级

Kubernetes集群升级需要谨慎操作，确保业务连续性。

# 检查当前版本
# kubectl version –short
Client Version: v1.28.0
Server Version: v1.28.0

# 查看可升级版本
# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster…
[upgrade/config] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
[upgrade] Running cluster health checks
[upgrade] Getting health check updates from the Kubernetes API server
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.29.0
[upgrade/versions] Target version: v1.29.0
[upgrade/versions] Latest version in the v1.28 series: v1.28.5

Components that must be upgraded manually after you have upgraded the control plane:
COMPONENT CURRENT TARGET
kubelet 3 x v1.28.0 v1.29.0
3 x v1.28.0 v1.29.0

Upgrade to the latest version in the v1.28 series:

COMPONENT CURRENT TARGET
kube-apiserver v1.28.0 v1.28.5
kube-controller-manager v1.28.0 v1.28.5
kube-scheduler v1.28.0 v1.28.5
kube-proxy v1.28.0 v1.28.5
CoreDNS v1.10.1 v1.11.1
etcd 3.5.9 3.5.10

# 升级kubeadm
# yum install -y kubeadm-1.29.0 –disableexcludes=kubernetes
Installed:
kubeadm-1.29.0-0.x86_64

# 验证kubeadm版本
# kubeadm version
kubeadm version: &version.Info{Major:”1″, Minor:”29″, GitVersion:”v1.29.0″, GitCommit:”abc123″, GitTreeState:”clean”, BuildDate:”2026-03-01T00:00:00Z”, GoVersion:”go1.21.0″, Compiler:”gc”, Platform:”linux/amd64″}

# 驱逐Master节点
# kubectl drain master01.fgedu.net.cn –ignore-daemonsets –delete-emptydir-data
node/master01.fgedu.net.cn cordoned
node/master01.fgedu.net.cn drained

# 升级Master节点
# kubeadm upgrade apply v1.29.0
[upgrade/config] Reading configuration from the cluster…
[upgrade/config] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -o yaml’
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Getting health check updates from the Kubernetes API server
[upgrade] Upgrading your Static Pod-hosted control plane to version “v1.29.0″…
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for “kube-apiserver” upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Moved new manifest to “/etc/kubernetes/manifests/kube-apiserver.yaml” and backed up old manifest to “/etc/kubernetes/tmp/kube-apiserver.yaml”
[upgrade/staticpods] Preparing for “kube-controller-manager” upgrade
[upgrade/staticpods] Moved new manifest to “/etc/kubernetes/manifests/kube-controller-manager.yaml” and backed up old manifest to “/etc/kubernetes/tmp/kube-controller-manager.yaml”
[upgrade/staticpods] Preparing for “kube-scheduler” upgrade
[upgrade/staticpods] Moved new manifest to “/etc/kubernetes/manifests/kube-scheduler.yaml” and backed up old manifest to “/etc/kubernetes/tmp/kube-scheduler.yaml”
[upgrade] The control plane has been upgraded successfully!

# 升级kubelet和kubectl
# yum install -y kubelet-1.29.0 kubectl-1.29.0 –disableexcludes=kubernetes
Installed:
kubelet-1.29.0-0.x86_64
kubectl-1.29.0-0.x86_64

# 重启kubelet
# systemctl daemon-reload
# systemctl restart kubelet

# 恢复Master节点调度
# kubectl uncordon master01.fgedu.net.cn
node/master01.fgedu.net.cn uncordoned

# 验证升级结果
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01.fgedu.net.cn Ready control-plane 90d v1.29.0
master02.fgedu.net.cn Ready control-plane 90d v1.28.0
master03.fgedu.net.cn Ready control-plane 90d v1.28.0
worker01.fgedu.net.cn Ready 90d v1.28.0
worker02.fgedu.net.cn Ready 90d v1.28.0
worker03.fgedu.net.cn Ready 90d v1.28.0

风哥风哥提示：升级集群时，建议逐个节点升级，先升级Master节点，再升级Worker节点。升级前务必做好etcd数据备份。

7. 集群备份恢复

定期备份集群数据是保障数据安全的重要措施。

# 备份etcd数据
# ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db \
–cacert=/etc/kubernetes/pki/etcd/ca.crt \
–cert=/etc/kubernetes/pki/etcd/server.crt \
–key=/etc/kubernetes/pki/etcd/server.key \
–endpoints=https://127.0.0.1:2379
Snapshot saved at /backup/etcd-snapshot-20260403.db

# 查看备份文件
# ls -lh /backup/etcd-snapshot-*.db
-rw——- 1 root root 25M Apr 3 10:00 /backup/etcd-snapshot-20260403.db

# 验证备份文件
# ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-20260403.db
Snapshot available at: /backup/etcd-snapshot-20260403.db
Revision: 12345678
Total keys: 5678
Total size: 26 MB

# 恢复etcd数据
# systemctl stop kubelet
# mv /var/lib/etcd /var/lib/etcd.bak

# 恢复快照
# ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-20260403.db \
–data-dir=/var/lib/etcd
2026-04-03 10:30:00.123456 I | mvcc: restore compact to 12345678
2026-04-03 10:30:00.234567 I | etcdserver/membership: added member 8e9e05c52164694d in the cluster
2026-04-03 10:30:00.345678 I | etcdserver/membership: added member ac450243d89e4e6b in the cluster
2026-04-03 10:30:00.456789 I | etcdserver/membership: added member bc561349d9af5e7c in the cluster

# 设置权限
# chown -R etcd:etcd /var/lib/etcd

# 启动kubelet
# systemctl start kubelet

# 验证集群状态
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01.fgedu.net.cn Ready control-plane 90d v1.29.0
master02.fgedu.net.cn Ready control-plane 90d v1.29.0
master03.fgedu.net.cn Ready control-plane 90d v1.29.0

# 使用Velero备份应用
# velero install –provider aws –plugins velero/velero-plugin-for-aws:v1.7.0 \
–bucket k8s-backup –secret-file ./credentials-velero \
–backup-location-config region=cn-north-1 \
–use-volume-snapshots=false

# 创建备份
# velero backup create fgedu-backup-$(date +%Y%m%d) –include-namespaces fgedu-prod
Backup request “fgedu-backup-20260403” submitted successfully.
Run `velero backup describe fgedu-backup-20260403` or `velero backup logs fgedu-backup-20260403` for more details.

# 查看备份状态
# velero backup describe fgedu-backup-20260403
Name: fgedu-backup-20260403
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.29.0
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=29

Phase: Completed
Total items to be backed up: 156
Items backed up: 156

Backup Volumes: 0

8. 集群监控

完善的监控体系是保障集群稳定运行的基础。

# 查看集群事件
# kubectl get events –sort-by=’.lastTimestamp’ -A | tail -20
NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
default 5m Normal Scheduled pod/nginx-abc12 Successfully assigned default/nginx-abc12 to worker01
default 5m Normal Pulling pod/nginx-abc12 Pulling image “nginx:1.24”
default 4m Normal Pulled pod/nginx-abc12 Successfully pulled image “nginx:1.24”
default 4m Normal Created pod/nginx-abc12 Created container nginx
default 4m Normal Started pod/nginx-abc12 Started container nginx

# 查看Pod资源使用
# kubectl top pods -n fgedu-prod
NAME CPU(cores) MEMORY(bytes)
nginx-6799fc88d8-abc12 1m 8Mi

# 查看所有命名空间的资源使用
# kubectl top pods -A | head -20
NAMESPACE NAME CPU(cores) MEMORY(bytes)
fgedu-prod nginx-6799fc88d8-abc12 1m 8Mi
kube-system coredns-5d78c9869d-abc12 3m 16Mi
kube-system coredns-5d78c9869d-def34 3m 15Mi
kube-system etcd-master01.fgedu.net.cn 25m 85Mi
kube-system kube-apiserver-master01.fgedu.net.cn 50m 256Mi
kube-system kube-controller-manager-master01.fgedu.net.cn 15m 64Mi
kube-system kube-proxy-abcde 1m 16Mi
kube-system kube-scheduler-master01.fgedu.net.cn 5m 32Mi

# 查看节点健康状况
# kubectl describe node worker01.fgedu.net.cn | grep -A 5 “Conditions:”
Conditions:
Type Status LastHeartbeatTime Reason Message
—- —— —————– —— ——-
MemoryPressure False Thu, 03 Apr 2026 10:00:00 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 03 Apr 2026 10:00:00 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 03 Apr 2026 10:00:00 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 03 Apr 2026 10:00:00 +0800 KubeletReady kubelet is posting ready status

# 查看组件日志
# kubectl logs -n kube-system kube-apiserver-master01.fgedu.net.cn –tail=20
I0403 10:00:00.123456 1 controller.go:123] Starting OpenAPI controller
I0403 10:00:00.234567 1 controller.go:234] Starting API Group and Version controller
I0403 10:00:00.345678 1 request.go:567] Request: “POST /api/v1/namespaces”
I0403 10:00:00.456789 1 controller.go:345] Starting namespace controller

9. 集群故障排查

掌握故障排查方法对于集群运维至关重要。

# 查看Pod状态
# kubectl get pods -n fgedu-prod
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-abc12 0/1 CrashLoopBackOff 5 10m

# 查看Pod详情
# kubectl describe pod nginx-6799fc88d8-abc12 -n fgedu-prod
Name: nginx-6799fc88d8-abc12
Namespace: fgedu-prod
Priority: 0
Node: worker01.fgedu.net.cn/192.168.1.20
Start Time: Thu, 03 Apr 2026 10:00:00 +0800
Labels: app=nginx
pod-template-hash=6799fc88d8
Status: Running
IP: 10.244.1.100
Containers:
nginx:
Container ID: containerd://abc123def456
Image: nginx:1.24
Image ID: docker.io/library/nginx@sha256:abc123
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1

# 查看Pod日志
# kubectl logs nginx-6799fc88d8-abc12 -n fgedu-prod –tail=50
2026/04/03 10:00:00 [emerg] 1#1: host not found in upstream “backend” in /etc/nginx/nginx.conf:15
nginx: [emerg] host not found in upstream “backend” in /etc/nginx/nginx.conf:15

# 查看之前容器的日志
# kubectl logs nginx-6799fc88d8-abc12 -n fgedu-prod –previous
2026/04/03 09:55:00 [emerg] 1#1: host not found in upstream “backend” in /etc/nginx/nginx.conf:15

# 进入容器调试
# kubectl exec -it nginx-6799fc88d8-abc12 -n fgedu-prod — /bin/bash
root@nginx-6799fc88d8-abc12:/# cat /etc/nginx/nginx.conf
…
upstream backend {
server backend-service:8080;
}
…

# 查看Service
# kubectl get svc -n fgedu-prod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
backend-service ClusterIP 10.96.100.100 8080/TCP 5m

# 查看节点问题
# kubectl describe node worker01.fgedu.net.cn | grep -A 10 “Events:”
Events:
Type Reason Age From Message
—- —— —- —- ——-
Warning SystemOOM 5m kubelet System OOM encountered
Normal NodeHasInsufficientMemory 5m kubelet Node is under memory pressure

# 查看kubelet日志
# journalctl -u kubelet –since “10 minutes ago” | tail -20
Apr 03 10:00:00 worker01 kubelet[12345]: I0403 10:00:00.123456 12345 kubelet.go:1234] SyncLoop (PLEG): “nginx-abc12_fgedu-prod”, event: &pleg.PodLifecycleEvent{ID:”abc123″}
Apr 03 10:00:00 worker01 kubelet[12345]: W0403 10:00:00.234567 12345 oom_watcher_linux.go:67] System OOM encountered
Apr 03 10:00:00 worker01 kubelet[12345]: E0403 10:00:00.345678 12345 kubelet.go:2345] “Failed to admit pod” err=”node has insufficient memory”

# 检查容器运行时
# crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT
abc123def456 10 minutes ago Ready nginx-6799fc88d8-abc12 fgedu-prod 0

# 检查容器状态
# crictl ps -a
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
def456ghi789 nginx:1.24 10 minutes ago Exited nginx 5

生产环境风哥建议：建立完善的监控告警体系，定期检查集群健康状态，制定故障应急预案，确保快速响应和处理问题。

10. 集群安全管理

安全管理是Kubernetes运维的核心任务之一。author:www.itpux.com

# 查看RBAC角色
# kubectl get roles -A
NAMESPACE NAME CREATED AT
kube-public kubeadm:bootstrap-signer-clusterinfo 2026-01-01T00:00:00Z
kube-public system:controller:bootstrap-signer 2026-01-01T00:00:00Z
kube-system extension-apiserver-authentication-reader 2026-01-01T00:00:00Z
kube-system kube-proxy 2026-01-01T00:00:00Z

# 查看集群角色
# kubectl get clusterroles | grep admin
admin 2026-01-01T00:00:00Z
cluster-admin 2026-01-01T00:00:00Z
system:admin 2026-01-01T00:00:00Z

# 创建ServiceAccount
# kubectl create serviceaccount fgedu-admin -n fgedu-prod
serviceaccount/fgedu-admin created

# 创建角色绑定
# kubectl create rolebinding fgedu-admin-binding \
–clusterrole=admin \
–serviceaccount=fgedu-prod:fgedu-admin \
-n fgedu-prod
rolebinding.rbac.authorization.k8s.io/fgedu-admin-binding created

# 查看网络策略
# kubectl get networkpolicy -A
NAMESPACE NAME POD-SELECTOR AGE
fgedu-prod deny-all-ingress 10d
fgedu-prod allow-nginx-ingress app=nginx 10d

# 创建网络策略
# cat <

风哥风哥提示：生产环境应启用RBAC权限控制，配置网络策略限制Pod间通信，使用Secret管理敏感信息，定期审计集群安全配置。

本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html