# 灾难恢复策略
– RTO (Recovery Time Objective)：恢复时间目标，即从灾难发生到系统恢复的时间
– RPO (Recovery Point Objective)：恢复点目标，即灾难发生后可以恢复到的数据点

# 灾难恢复级别
– 本地恢复：在本地数据中心恢复系统
– 异地恢复：在异地数据中心恢复系统
– 混合恢复：结合本地和异地恢复

# 灾难恢复方案
– 冷备份：定期备份数据，灾难发生后手动恢复
– 热备份：实时同步数据，灾难发生后自动切换
– 温备份：定期同步数据，灾难发生后半自动恢复

# 高可用架构
– 控制平面高可用：部署3-5个控制平面节点，分布在不同的可用区
– etcd高可用：部署3-5个etcd节点，分布在不同的可用区
– 计算节点高可用：部署多个计算节点，分布在不同的可用区
– 网络高可用：使用多网卡绑定，确保网络连接的可靠性
– 存储高可用：使用分布式存储或云存储，确保数据存储的可靠性

# 负载均衡设计
– 控制平面负载均衡：使用HAProxy或云负载均衡器
– 服务负载均衡：使用Kubernetes Service或Ingress
– 外部访问负载均衡：使用云负载均衡器或硬件负载均衡器

# 网络设计
– 多可用区网络：跨可用区部署，提高可靠性
– 网络冗余：多网络路径，避免单点故障
– 网络隔离：使用网络策略，提高安全性

# 灾难恢复计划
– 风险评估：识别可能的灾难类型和影响范围
– 恢复目标：设定RTO和RPO目标
– 恢复流程：详细的灾难恢复步骤和流程
– 责任分配：明确各角色的责任和职责
– 演练计划：定期进行灾难恢复演练

# 灾难类型
– 硬件故障：服务器、存储、网络设备故障
– 软件故障：操作系统、应用程序故障
– 人为错误：误操作、配置错误
– 自然灾害：地震、洪水、火灾等
– 网络攻击：DDoS攻击、数据泄露等

# 恢复优先级
– 核心业务应用：优先恢复核心业务应用
– 数据存储：确保数据的完整性和可用性
– 辅助服务：逐步恢复辅助服务
– 监控系统：确保监控系统的正常运行

# 备份策略
– 全量备份：完整备份所有数据，恢复速度快，但备份时间长
– 增量备份：只备份变化的数据，备份时间短，但恢复速度慢
– 差异备份：备份自上次全量备份以来变化的数据，平衡备份和恢复速度

# 备份频率
– etcd数据：每小时备份一次
– 集群配置：每天备份一次
– 应用数据：根据应用重要性，每天或每小时备份一次
– 系统配置：每周备份一次

# 备份存储
– 本地存储：用于快速恢复
– 异地存储：用于灾难恢复
– 云存储：用于长期备份
– 加密存储：确保备份数据的安全性

# 备份etcd数据
$ ETCDCTL_API=3 etcdctl –endpoints=https://192.168.1.101:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcd-backup-$(date +%Y%m%d%H%M%S).db

# 查看备份状态
$ ETCDCTL_API=3 etcdctl –endpoints=https://192.168.1.101:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot status /tmp/etcd-backup-20260403120000.db

# 恢复etcd数据
$ ETCDCTL_API=3 etcdctl –endpoints=https://192.168.1.101:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot restore /tmp/etcd-backup-20260403120000.db –data-dir=/var/lib/etcd-restore

# 重启etcd服务
$ sudo systemctl stop etcd
$ sudo mv /var/lib/etcd /var/lib/etcd.bak
$ sudo mv /var/lib/etcd-restore /var/lib/etcd
$ sudo systemctl start etcd

# 备份集群配置
$ mkdir -p /backup/kubernetes
$ kubectl get all –all-namespaces -o yaml > /backup/kubernetes/all-resources.yaml
$ kubectl get configmaps –all-namespaces -o yaml > /backup/kubernetes/configmaps.yaml
$ kubectl get secrets –all-namespaces -o yaml > /backup/kubernetes/secrets.yaml
$ kubectl get serviceaccounts –all-namespaces -o yaml > /backup/kubernetes/serviceaccounts.yaml
$ kubectl get roles –all-namespaces -o yaml > /backup/kubernetes/roles.yaml
$ kubectl get rolebindings –all-namespaces -o yaml > /backup/kubernetes/rolebindings.更多视频教程www.fgedu.net.cnyaml
$ kubectl get clusterroles –all-namespaces -o yaml > /backup/kubernetes/clusterroles.yaml
$ kubectl get clusterrolebindings –all-namespaces -o yaml > /backup/kubernetes/clusterrolebindings.yaml

# 备份kubeconfig文件
$ cp ~/.kube/config /backup/kubernetes/kubeconfig

# 备份证书
$ cp -r /etc/kubernetes/pki /backup/kubernetes/

# 备份应用数据
$ kubectl get pv –all-namespaces -o yaml > /backup/kubernetes/pv.yaml
$ kubectl get pvc –all-namespaces -o yaml > /backup/kubernetes/pvc.yaml

# 备份数据库应用数据
$ kubectl exec -it mysql-7d75c484b5-8x8x8 — mysqldump -u root -p fgedudb > /backup/kubernetes/fgedudb.sql

# 备份持久卷数据
$ kubectl cp default/nginx-6799fc88d8-2q4x2:/usr/share/nginx/html /backup/kubernetes/nginx-html

# 备份配置文件
$ kubectl get configmap nginx-config -o yaml > /backup/kubernetes/nginx-config.yaml

# 案例：企业级高可用K8s集群
# 环境：3个控制平面节点，5个计算节点，跨3个可用区
# 目标：部署高可用K8s集群，确保集群的可靠性

# 1. 配置负载均衡器
$ sudo dnf install -y haproxy
$ sudo vim /etc/haproxy/haproxy.cfg

frontend kubernetes
bind *:6443
mode tcp
option tcplog
default_backend kubernetes-master-nodes

backend kubernetes-master-nodes
mode tcp
option tcplog
option tcp-check
balance roundrobin
server master1 192.168.1.101:6443 check fall 3 rise 2
ser学习交流加群风哥微信: itpux-comver master2 192.168.1.102:6443 check fall 3 rise 2
server master3 192.168.1.103:6443 check fall 3 rise 2

# 2. 初始化第一个控制平面节点
$ sudo kubeadm init –control-plane-endpoint “192.168.1.100:6443” –pod-network-cidr=10.244.0.0/16

# 3. 加入其他控制平面节点
$ kubeadm join 192.168.1.100:6443 –token abcdef.0123456789abcdef –discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef –control-plane

# 4. 加入计算节点
$ kubeadm join 192.168.1.100:6443 –token abcdef.0123456789abcdef –discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

# 5. 安装网络插件
$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# 6. 查看集群状态
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane 20m v1.27.0
master2 Ready control-plane 15m v1.27.0
master3 Ready control-plane 10m v1.27.0
worker1 Ready 8m v1.27.0
worker2 Ready 7m v1.27.0
worker3 Ready 6m v1.27.0
worker4 Ready 5m v1.27.0
worker5 Ready 4m v1.27.0

# 7. 测试高可用性
# 模拟控制平面节点故障
$ sudo systemctl stop kube-apiserver kube-controller-manager kube-scheduler etcd

# 查看集群状态
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 NotReady control-plane 25m v1.27.0
master2 Ready control-plane 20m v1.27.0
master3 Ready control-plane 15m v1.27.0
worker1 Ready 13m v1.27.0
worker2 Ready 12m v1.27.0
worker3 Ready 11m v1.27.0
worker4 Ready 10m v1.27.0
worker5 Ready 9m v1.27.0

# 测试应用访问
$ kubectl run nginx –image=nginx –replicas=3
$ kubectl expose deployment nginx –port=80 –type=NodePort
$ curl http://192.168.1.104:30080

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.

For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.

Thank you for using nginx.

# 案例：企业级灾难恢复演练
# 环境：2个K8s集群（主集群和备用集群）
# 目标：演练灾难恢复流程，确保在主集群故障时能够快速切换到备用集群

# 1. 定期备份主集群数据
$ ETCDCTL_API=3 etcdctl –endpoints=https://192.168.1.101:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot save /backup/etcd-backup-$(date +%Y%m%d%H%M%S).db

# 2. 将备份数据复制到备用集群
$ scp /backup/etcd-backup-20260403120000.db root@192.168.2.101:/backup/

# 3. 模拟主集群故障
$ sudo systemctl stop kube-apiserver kube-controller-manager kube-scheduler etcd

# 4. 在备用集群恢复数据
$ ETCDCTL_API=3 etcdctl –endpoints=https://192.168.2.101:2379 –cacert=/etc/kubernetes/pki/etcd/ca.crt –cert=/etc/kubernetes/pki/etcd/server.crt –key=/etc/kubernetes/pki/etcd/server.key snapshot restore /backup/etcd-backup-20260403120000.db –data-dir=/var/lib/etcd-restore

# 5. 重启备用集群的etcd服务
$ sudo systemctl stop etcd
$ sudo mv /var/lib/etcd /var/lib/etcd.bak
$ sudo mv /var/lib/etcd-restore /var/lib/etcd
$ sudo systemctl start etcd

# 6. 重启备用集群的控制平面组件
$ sudo systemctl start kube-apiserver kube-controller-manager kube-scheduler

# 7. 验证备用集群状态
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane 20m v1.27.0
worker1 Ready 15m v1.27.0
worker2 Ready 10m v1.27.0

# 8. 测试应用访问
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-2q4x2 1/1 Running 0 5m
nginx-6799fc88d8-5b678 1/1 Running 0 5m
nginx-6799fc88d8-7c89d 1/1 Running 0 5m

$ curl http://192.168.2.102:30080

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.

For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.

Thank you for using nginx.

# 案例：企业级跨区域高可用K8s集群
# 环境：2个区域，每个区域3个控制平面节点，5个计算节点
# 目标：部署跨区域高可用K8s集群，确保在区域故障时能够快速切换

# 1. 在区域A部署主集群
$ sudo kubeadm init –control-plane-endpoint “192.168.1.100:6443” –pod-network-cidr=10.244.0.0/16
$ kubeadm join 192.168.1.100:6443 –token abcdef.0123456789abcdef –discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef –control-plane
$ kubeadm join 192.168.1.100:6443 –token abcdef.0123456789abcdef –discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

# 2. 在区域B部署备用集群
$ sudo kubeadm init –control-plane-endpoint “192.168.2.100:6443” –pod-network-cidr=10.245.0.0/16
$ kubeadm join 192.168.2.100:6443 –token abcdef.0123456789abcdef –discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef –control-plane
$ kubeadm join 192.168.2.100:6443 –token abcdef.0123456789abcdef –discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

# 3. 配置跨区域网络
$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# 4. 配置跨区域数据同步
$ helm repo add velero https://vmware-tanzu.github.io/helm-charts
$ helm install velero velero/velero –namespace velero –create-namespace –set configuration.provider=aws –set configuration.backupLocation.name=default –set configuration.backupLocation.bucket=velero-backup –set credentials.secretContents.cloud={}

# 5. 测试跨区域故障转移
# 模拟区域A故障
$ sudo systemctl stop kube-apiserver kube-controller-manager kube-scheduler etcd

# 切换到区域B集群
$ export KUBECONFIG=~/.kube/config-region-b
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane 20m v1.27.0
master2 Ready control-plane 15m v1.27.0
master3 Ready control-plane 10m v1.27.0
worker1 Ready 8m v1.27.0
worker2 Ready 7m v1.27.0
worker3 Ready 6m v1.27.0
worker4 Ready 5m v1.27.0
worker5 Ready 4m v1.27.0

# 测试应用访问
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-2q4x2 1/1 Running 0 5m
nginx-6799fc88d8-5b678 1/1 Running 0 5m
nginx-6799fc88d8-7c89d 1/1 Running 0 5m

$ curl http://192.168.2.104:30080

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.

For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.

Thank you for using nginx.

# 常见问题与解决方案

## 1. 控制平面节点故障
– 原因：硬件故障、网络故障、软件故障
– 解决方案：部署多个控制平面节点，确保控制平面的高可用

## 2. etcd数据损坏
– 原因：磁盘故障、网络分区、人为错误
– 解决方案：定期备份etcd数据，使用etcd集群，确保数据的一致性

## 3. 集群配置丢失
– 原因：人为错误、配置管理不当
– 解决方案：定期备份集群配置，使用版本控制管理配置

## 4. 应用数据丢失
– 原因：存储故障、备份不当、人为错误
– 解决方案：使用持久卷，定期备份应用数据，使用高可用存储

## 5. 灾难恢复失败
– 原因：恢复流程不当、备份数据损坏、环境差异
– 解决方案：定期演练灾难恢复流程，验证备份数据的完整性，确保环境一致性