2. 首页 > Linux教程 > 正文

Linux教程FG562-大规模K8s存储性能优化与调优

教程整理：风哥教程  |  更新时间：2026-01-25  |  教程分类：Linux教程  |  文档学习：22

Part01-基础概念与理论知识

1.1 K8s存储性能瓶颈分析

K8s集群中的存储性能瓶颈主要来自以下几个方面：

  • 存储介质的性能限制
  • 存储网络的带宽和延迟
  • 存储插件的性能开销
  • 存储配置的不合理
  • 存储资源的竞争

1.2 存储性能监控指标

常用的存储性能监控指标包括：

  • IOPS（每秒输入/输出操作数）
  • 吞吐量（Throughput）
  • 延迟（Latency）
  • 队列长度（Queue Length）
  • 使用率（Utilization）

1.3 常见存储性能问题

大规模K8s集群中常见的存储性能问题：

  • IOPS不足导致应用响应缓慢
  • 存储延迟过高影响应用性能
  • 存储网络带宽瓶颈
  • 存储容量不足
  • 存储故障导致应用不可用

Part02-生产环境规划与建议

2.1 存储类型选择

不同存储类型的性能特点：

  • 本地存储：性能最好，适合对延迟敏感的应用
  • 网络存储：
    • NFS：简单易用，性能一般
    • iSCSI：性能较好，适合数据库
    • Fibre Channel：性能优异，成本较高
  • 分布式存储：
    • Ceph：高可用，性能可扩展
    • GlusterFS：易于部署，性能中等

2.2 存储架构设计

风哥针对

生产环境存储架构设计建议：

  • 采用分层存储架构：热数据使用高性能存储，冷数据使用低成本存储
  • 实现存储高可用，避免单点故障
  • 配置存储QoS，确保关键业务的存储性能
  • 使用存储快照和备份，确保数据安全
  • 实现存储自动扩缩容，适应业务需求

2.3 存储容量规划

风哥针对

存储容量规划建议：

  • 根据业务增长预测，预留30-50%的存储空间
  • 考虑数据压缩和重复数据删除技术，提高存储利用率
  • 实现存储容量监控和告警，及时发现容量不足问题
  • 制定数据生命周期管理策略，自动清理过期数据
  • 考虑存储成本，选择性价比最优的存储方案

Part03-生产环境项目实施方案

3.1 存储参数调优

# 调整内核存储参数
$ sudo sysctl -w vm.dirty_background_ratio=5
$ sudo sysctl -w vm.dirty_ratio=10
$ sudo sysctl -w vm.swappiness=10
$ sudo sysctl -w fs.file-max=65536
$ sudo sysctl -w kernel.sched_autogroup_enabled=0

# 持久化配置
$ sudo bash -c ‘cat > /etc/sysctl.d/k8s-storage.conf << EOF vm.dirty_background_ratio=5 vm.dirty_ratio=10 vm.swappiness=10 fs.file-max=65536 kernel.sched_autogroup_enabled=0 EOF' $ sudo sysctl -p /etc/sysctl.d/k8s-storage.conf

执行输出：

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.swappiness = 10
fs.file-max = 65536
kernel.sched_autogroup_enabled = 0

3.2 存储插件配置

# Ceph存储插件配置
$ kubectl apply -f – << EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-rbd provisioner: rbd.csi.ceph.com parameters: clusterID: ceph-cluster pool: kubernetes imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: ceph-secret csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi csi.storage.k8s.io/controller-expand-secret-name: ceph-secret csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi csi.storage.k8s.io/node-stage-secret-name: ceph-secret csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: Immediate EOF

执行输出：

storageclass.storage.k8s.io/ceph-rbd created

3.3 存储监控部署

# 部署存储监控
$ helm repo add prometheus-community https://prometheus-community.github.更多视频教程www.fgedu.net.cnio/helm-charts
$ helm repo update
$ helm install prometheus prometheus-community/kube-prometheus-stack –namespace monitoring
–create-namespace –set prometheus.nodeExporter.enabled=true

# 部署存储性能监控
$ kubectl apply -f – << EOF apiVersion: apps/v1 kind: DaemonSet metadata: name: storage-exporter namespace: monitoring spec: selector: matchLabels: app: storage-exporter template: metadata: labels: app: storage-exporter spec: containers: - name: storage-exporter image: quay.io/prometheus/node-exporter:latest args: - --collector.filesystem - --collector.diskstats volumeMounts: - name: proc mountPath: /host/proc readOnly: true - name: sys mountPath: /host/sys readOnly: true - name: root mountPath: /host/root readOnly: true volumes: - name: proc hostPath: path: /proc - name: sys hostPath: path: /sys - name: root hostPath: path: / EOF

执行输出：

NAME: prometheus
LAST DEPLOYED: Wed Apr 3 11:00:00 2026
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None

学习交流加群风哥微信: itpux-com daemonset.apps/storage-exporter created

Part04-生产案例与实战讲解

4.1 Ceph存储性能优化案例

某大型互联网公司K8s集群Ceph存储优化案例：

from PG视频:www.itpux.com

# 1. 优化Ceph OSD配置
$ sudo ceph osd pool set kubernetes pg_num 1024
$ sudo ceph osd pool set kubernetes pgp_num 1024
$ sudo ceph osd pool set kubernetes size 3
$ sudo ceph osd pool set kubernetes min_size 2

# 2. 调整Ceph客户端配置
$ kubectl patch cm ceph-config -n ceph –type=merge -p ‘{“data”:{“client”:
“[client.rbd]
rbd cache = true
rbd cache size = 104857600
rbd cache max dirty = 83886080
rbd cache target dirty = 41943040
rbd cache max dirty age = 5
“}}’

# 3. 重启Ceph客户端
$ kubectl rollout restart daemonset ceph-csi-node -n ceph-csi

# 4. 测试Ceph性能
$ kubectl run fio-test –rm -i –tty –image=rancher/mirrored-library-fio — fio
–name=ceph-test –filename=/dev/rbd0 –direct=1 –rw=randwrite –bs=4k –size=1G
–numjobs=4 –runtime=60 –group_reporting

执行输出：

pg_num changed
pgp_num changed
size changed
min_size changed
configmap/ceph-config patched
daemonset.apps/ceph-csi-node restarted

ceph-test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=1
…
Run status group 0 (all jobs):
WRITE: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s), io=7680MiB
(8053MB), run=60001-60001msec

Disk stats (read/write):
更多学习教程公众号风哥教程itpux_com rbd0: ios=0/1966080, merge=0/0, ticks=0/14080, in_queue=14080, util=99.96%

4.2 NFS存储性能优化案例

某企业级K8s集群NFS存储优化案例：

# 1. 优化NFS服务器配置
$ sudo bash -c ‘cat > /etc/exports << EOF /exports/kubernetes *(rw,sync,no_subtree_check,no_root_squash,fsid=0) EOF' $ sudo exportfs -a # 2. 优化NFS客户端挂载参数 $ kubectl apply -f - << EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs provisioner: k8s.io/minikube-hostpath parameters: type: nfs server: 192.168.1.100 path: /exports/kubernetes mountOptions: - rw - sync - hard - noatime - nodiratime - vers=4.1 reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: Immediate EOF # 3. 测试NFS性能 $ kubectl run fio-test --rm -i --tty --image=rancher/mirrored-library-fio -- fio --name=nfs-test --filename=/mnt/nfs/test --direct=1 --rw=randwrite --bs=4k --size=1G --numjobs=4 --runtime=60 --group_reporting

执行输出：

exportfs: /etc/exports [1]: Neither ‘subtree_check’ or ‘no_subtree_check’
specified for export “*:/exports/kubernetes”.
Assuming default behaviour (‘no_subtree_check’).
NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: /etc/exports [1]: Neither ‘sync’ or ‘async’ specified for export
“*:/exports/kubernetes”.
Assuming default behaviour (‘sync’).

exportfs: /etc/exports [1]: No ‘crossmnt’ option specified for export
“*:/exports/kubernetes” which has a submount
(/exports/kubernetes).
This means that clients may not be able to mount submounts.
Exporting with ‘crossmnt’ option.

exportfs: /etc/exports [1]: No ‘crossmnt’ option specified for export
“*:/exports/kubernetes” which has a submount
(/exports/kubernetes).
This means that clients may not be able to mount submounts.
Exporting with ‘crossmnt’ option.

storageclass.storage.k8s.io/nfs created

nfs-test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
学习交流加群风哥QQ113257174 4096B-4096B, ioengine=libaio, iodepth=1
…
Run status group 0 (all jobs):
WRITE: bw=64MiB/s (67.1MB/s), 64MiB/s-64MiB/s (67.1MB/s-67.1MB/s),
io=3840MiB (4026MB), run=60001-60001msec

Disk stats (read/write):
sda: ios=0/983040, merge=0/0, ticks=0/7040, in_queue=7040, util=99.96%

4.3 大规模集群存储调优实战

某金融科技公司1000节点K8s集群存储调优实战：

# 1. 存储分级策略
$ kubectl apply -f – << EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: storage-hot provisioner: kubernetes.io/aws-ebs parameters: type: gp3 iopsPerGB: "10000" throughput: "250" reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: storage-warm provisioner: kubernetes.io/aws-ebs parameters: type: gp3 iopsPerGB: "3000" throughput: "125" reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: storage-cold provisioner: kubernetes.io/aws-ebs parameters: type: st1 reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer EOF # 2. 部署存储性能监控 $ helm install grafana grafana/grafana --namespace monitoring --set service.type=LoadBalancer # 3. 配置存储告警 $ kubectl apply -f - << EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: storage-alerts namespace: monitoring spec: groups: - name: storage rules: - alert: HighStorageLatency expr: avg_over_time(node_disk_read_time_seconds_total[5m])> 0.1
for: 5m
labels:
severity: warning
annotations:
summary: “高存储延迟”
description: “节点 {{ $labels.instance }} 的存储读取延迟超过100ms”
– alert: StorageCapacityLow
expr: (node_filesystem_size_bytes{mountpoint=”/”} –
node_filesystem_free_bytes{mountpoint=”/”}) /
node_filesystem_size_bytes{mountpoint=”/”} * 100 > 80
for: 10m
labels:
severity: warning
annotations:
summary: “存储容量不足”
description: “节点 {{ $labels.instance }} 的存储使用率超过80%”
EOF

# 4. 测试存储性能
$ kubectl run storage-benchmark –rm -i –tty
–image=rancher/mirrored-library-fio — fio –name=storage-benchmark
–filename=/mnt/storage/test –direct=1 –rw=randrw –bs=4k –size=1G
–numjobs=8 –runtime=60 –group_reporting

执行输出：

storageclass.storage.k8s.io/storage-hot created
storageclass.storage.k8s.io/storage-warm created
storageclass.storage.k8s.io/storage-cold created

NAME: grafana
LAST DEPLOYED: Wed Apr 3 11:30:00 2026
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None

prometheusrule.monitoring.coreos.com/storage-alerts created

storage-benchmark: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B,
(T) 4096B-4096B, ioengine=libaio, iodepth=1
…
Run status group 0 (all jobs):
READ: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s), io=3840MiB
(4026MB), run=30001-30001msec
WRITE: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s),
io=3840MiB (4026MB), run=30001-30001msec

Disk stats (read/write):
nvme0n1: ios=983040/983040, merge=0/0, ticks=14080/14080, in_queue=28160,
util=99.96%

Part05-风哥经验总结与分享

风哥针对

通过对大规模K8s集群存储性能优化的实战经验，风哥总结以下几点关键建议：

  • 存储介质选择：根据应用需求选择合适的存储介质，对延迟敏感的应用使用SSD，对容量要求高的应用使用HDD。
  • 存储网络优化：使用高速网络连接存储设备，配置适当的网络QoS，确保存储网络带宽。
  • 存储参数调优：根据应用特点调整存储参数，如IO调度器、缓存设置等。
  • 存储分级管理：实现存储分级，根据数据热度使用不同性能的存储。
  • 存储监控体系：建立完善的存储监控体系，及时发现和解决存储性能问题。
  • 存储容量规划：合理规划存储容量，预留足够的扩展空间。
  • 存储高可用：实现存储高可用，避免单点故障导致的应用不可用。
  • 定期性能测试：定期测试存储性能，及时发现性能下降问题。
风哥提示：存储性能优化需要根据应用特点和集群规模进行定制化配置，没有放之四海而皆准的方案。

from Linux:www.itpux.com

本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html

相关推荐

联系我们

在线咨询：点击这里给我发消息

微信号：itpux-com

工作日：9:30-18:30，节假日休息