大数据教程FG020-Hadoop集群性能监控与调优实战

内容简介：本文详细介绍Hadoop集群性能监控与调优实战，包括集群监控指标、性能分析工具、调优策略、故障诊断等核心内容。风哥教程参考Hadoop官方文档Cluster Setup、Performance Tuning等内容。

目录大纲

Part01-基础概念与理论知识
  1.1 Hadoop监控体系
  1.2 关键性能指标
  1.3 性能调优原理
Part02-生产环境规划与建议
  2.1 监控系统规划
  2.2 告警策略规划
  2.3 调优策略规划
Part03-生产环境项目实施方案
  3.1 集群状态监控
  3.2 性能指标采集
  3.3 性能分析与调优
  3.4 故障诊断处理
Part04-生产案例与实战讲解
  4.1 NameNode性能调优案例
  4.2 DataNode性能调优案例
  4.3 YARN资源调优案例
Part05-风哥经验总结与分享
  5.1 监控运维最佳实践
  5.2 性能调优经验总结

Part01-基础概念与理论知识

1.1 Hadoop监控体系

Hadoop集群监控是保障集群稳定运行的关键。更多视频教程www.fgedu.net.cn 监控体系包括系统层监控、Hadoop组件监控、应用层监控三个层次。

风哥提示：完善的监控体系可以及时发现集群问题，避免故障扩大。

1.2 关键性能指标

Hadoop集群关键性能指标包括CPU、内存、磁盘IO、网络IO等。学习交流加群风哥微信: itpux-com

Hadoop关键监控指标：
– NameNode：内存使用、文件数、块数、GC时间
– DataNode：磁盘使用、IO吞吐、心跳延迟
– ResourceManager：应用数、队列资源、调度延迟
– NodeManager：Container数、内存使用、CPU使用

1.3 性能调优原理

性能调优需要从硬件、操作系统、Hadoop配置三个层面进行。from bigdata视频:www.itpux.com

# 查看集群基本状态
hdfs dfsadmin -report
yarn node -list

# HDFS集群报告
Live datanodes: 6
Dead datanodes: 0
Total storage: 120 TB
Used storage: 60 TB

# YARN节点列表
Total Nodes: 6
Node-Id State Memory-Used VCores-Used
fgedu01:8041 RUNNING 16384 MB 8
fgedu02:8041 RUNNING 16384 MB 8
fgedu03:8041 RUNNING 16384 MB 8
fgedu04:8041 RUNNING 16384 MB 8
fgedu05:8041 RUNNING 16384 MB 8
fgedu06:8041 RUNNING 16384 MB 8

Part02-生产环境规划与建议

2.1 监控系统规划

监控系统规划需要选择合适的监控工具和架构。更多学习教程公众号风哥教程itpux_com

监控系统规划建议：
– 使用Prometheus+Grafana监控
– 配置Hadoop JMX指标采集
– 设置合理的采集频率
– 建立可视化监控大屏

2.2 告警策略规划

告警策略需要根据业务重要性制定。学习交流加群风哥QQ113257174

# 查看Hadoop JMX指标
curl -s http://fgedu01:9870/jmx | grep -A5 “name=NameNodeInfo”

# NameNode JMX指标
{
“name” : “Hadoop:service=NameNode,name=NameNodeInfo”,
“ClusterId” : “CID-12345678-1234-1234-1234-123456789012”,
“Total” : 131941395087360,
“Used” : 65970697543680,
“PercentUsed” : 50.0,
“TotalBlocks” : 1000000,
“TotalFiles” : 500000,
“LiveNodes” : 6,
“DeadNodes” : 0
}

2.3 调优策略规划

调优策略需要根据实际负载和瓶颈制定。风哥提示：调优前需要先进行性能分析，找到瓶颈点。

调优策略要点：
– 分析性能瓶颈
– 制定调优目标
– 逐步调整参数
– 验证调优效果
– 记录调优过程

Part03-生产环境项目实施方案

3.1 集群状态监控

3.1.1 NameNode监控

# NameNode状态监控
hdfs dfsadmin -safemode get
hdfs dfsadmin -report | head -30
# 查看NameNode Web UI指标
curl -s http://fgedu01:9870/dfshealth.html | grep -A5 “Cluster Summary”

# 安全模式状态
Safe mode is OFF

# NameNode报告
Configured Capacity: 131941395087360 (120 TB)
Present Capacity: 125244325333120 (113.87 TB)
DFS Remaining: 59273548779520 (53.93 TB)
DFS Used: 65970776553600 (60 TB)
DFS Used%: 52.70%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

# 集群摘要
Cluster Summary:
Total Files: 500000
Total Blocks: 1000000
Live Nodes: 6
Dead Nodes: 0

3.1.2 DataNode监控

# DataNode状态
Datanode: fgedu01:9866
State: In Service
Capacity: 21990232555520 B (20 TB)
Used: 10995116277760 B (10 TB)
Remaining: 10995116277760 B (10 TB)
Block Pool Used: 10995116277760 B (10 TB)
Block Pool Used%: 50.00%

Datanode: fgedu02:9866
State: In Service
Capacity: 21990232555520 B (20 TB)
Used: 10995116277760 B (10 TB)
Remaining: 10995116277760 B (10 TB)

# 磁盘使用汇总
Name: fgedu01:9866 (fgedu01)
Capacity: 21990232555520
Used: 10995116277760
Remaining: 10995116277760

3.2 性能指标采集

3.2.1 JMX指标采集

# 采集NameNode JMX指标
curl -s http://fgedu01:9870/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo | python -m json.tool
# 采集ResourceManager JMX指标
curl -s http://fgedu01:8088/jmx?qry=Hadoop:service=ResourceManager,name=RMNMInfo | python -m json.tool

# NameNode JMX
{
“beans” : [ {
“name” : “Hadoop:service=NameNode,name=NameNodeInfo”,
“modelerType” : “org.apache.hadoop.hdfs.server.namenode.NameNodeInfo”,
“Total” : 131941395087360,
“Used” : 65970697543680,
“PercentUsed” : 50.0,
“TotalBlocks” : 1000000,
“TotalFiles” : 500000,
“LiveNodes” : 6,
“DeadNodes” : 0,
“ClusterId” : “CID-12345678-1234-1234-1234-123456789012”
} ]
}

# ResourceManager JMX
{
“beans” : [ {
“name” : “Hadoop:service=ResourceManager,name=RMNMInfo”,
“modelerType” : “org.apache.hadoop.yarn.server.resourcemanager.RMNMInfo”,
“LiveNodeManagers” : 6,
“TotalNodeManagers” : 6,
“TotalMemory” : 491520,
“UsedMemory” : 245760,
“AvailableMemory” : 245760
} ]
}

3.2.2 系统指标采集

#!/bin/bash
# system_monitor.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

echo “=== System Monitor ===”
echo “Date: $(date)”

# CPU使用率
echo “CPU Usage:”
top -bn1 | grep “Cpu(s)” | awk ‘{print $2}’

# 内存使用
echo “Memory Usage:”
free -h

# 磁盘IO
echo “Disk IO:”
iostat -x 1 3 | grep -A10 “Device”

# 网络IO
echo “Network IO:”
sar -n DEV 1 3 | grep -E “eth0|ens”

# 系统监控输出
=== System Monitor ===
Date: Wed Jan 17 10:00:00 CST 2024

CPU Usage:
15.3%

Memory Usage:
total used free shared buff/cache available
Mem: 125G 60G 10G 1.0G 55G 60G
Swap: 16G 0B 16G

Disk IO:
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %util
sda 50.00 100.00 5.00 10.00 0.00 10.00 85.00
sdb 30.00 50.00 3.00 5.00 0.00 5.00 60.00

Network IO:
eth0 1000.00 500.00 100.00 50.00 0.00 0.00

3.3 性能分析与调优

3.3.1 NameNode内存调优

# 分析NameNode内存使用
jmap -heap $(pgrep -f NameNode) | head -20
# 查看GC情况
jstat -gc $(pgrep -f NameNode) 1000 5

# NameNode堆内存
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 8589934592 (8192.0MB)
NewSize = 1431306240 (1365.0MB)
MaxNewSize = 1431306240 (1365.0MB)
OldSize = 7158628352 (6827.0MB)

# GC统计
S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT
136504 136504 0.0 102400 1365040 1024000 6826752 2048000 51200 48000 5120 4800 100 5.000 5 2.500 7.500
136504 136504 0.0 102400 1365040 1024000 6826752 2048000 51200 48000 5120 4800 100 5.000 5 2.500 7.500
# GC时间正常，Full GC次数较少

3.3.2 DataNode性能调优

# DataNode配置优化
cat /bigdata/app/hadoop/etc/hadoop/hdfs-site.xml | grep -A3 “handler”

<!– DataNode处理线程数 –>
<property>
<name>dfs.datanode.handler.count</name>
<value>20</value>
</property>

<!– DataNode传输线程数 –>
<property>
<name>dfs.datanode.max.transfer.threads</name>
<value>8192</value>
</property>

<!– 数据传输带宽 –>
<property>
<name>dfs.datanode.balance.bandwidthPerSec</name>
<value>104857600</value> # 100MB/s
</property>

3.4 故障诊断处理

3.4.1 性能问题诊断

# 诊断NameNode延迟
hdfs dfsadmin -metasave /tmp/namenode_meta.txt
cat /tmp/namenode_meta.txt | head -50
# 分析DataNode延迟
hdfs dfsadmin -report | grep -E “Last contact|State”

# NameNode元数据
MetaSave: /tmp/namenode_meta.txt
Total number of blocks: 1000000
Number of pending blocks: 0
Number of pending replication blocks: 0
Number of pending deletion blocks: 0

# DataNode心跳
Datanode: fgedu01:9866
State: In Service
Last contact: Wed Jan 17 10:05:00 CST 2024

Datanode: fgedu02:9866
State: In Service
Last contact: Wed Jan 17 10:05:00 CST 2024

3.4.2 日志分析

# 分析NameNode日志
grep -E “ERROR|WARN” /bigdata/app/hadoop/logs/hadoop-hdfs-namenode-fgedu01.log | tail -20
# 分析ResourceManager日志
grep -E “ERROR|WARN” /bigdata/app/hadoop/logs/yarn-resourcemanager-fgedu01.log | tail -20

# NameNode错误日志
2024-01-17 10:00:00,000 WARN namenode.NameNode: Slow block report from DataNode fgedu03:9866
2024-01-17 10:00:05,000 WARN hdfs.server.blockmanagement.BlockManager: Block blk_1073741825 has corrupt replicas

# ResourceManager错误日志
2024-01-17 10:00:00,000 WARN resourcemanager.RMAppManager: Application application_1705473600000_0001 failed
2024-01-17 10:00:05,000 ERROR resourcemanager.ResourceManager: NodeManager fgedu05:8041 disconnected

Part04-生产案例与实战讲解

4.1 NameNode性能调优案例

NameNode是HDFS的核心组件，性能优化至关重要。更多视频教程www.fgedu.net.cn

#!/bin/bash
# namenode_tuning.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

# NameNode性能调优脚本

# 1. 分析内存使用
echo “=== NameNode Memory Analysis ===”
jmap -heap $(pgrep -f NameNode) | grep -E “Heap|Old”

# 2. 分析GC情况
echo “=== GC Analysis ===”
jstat -gcutil $(pgrep -f NameNode) 1000 3

# 3. 分析文件数
echo “=== File Count ===”
hdfs dfsadmin -report | grep “Total Files”

# 4. 建议配置
echo “=== Recommended Configuration ===”
FILE_COUNT=$(hdfs dfsadmin -report | grep “Total Files” | awk ‘{print $3}’)
HEAP_SIZE=$(( ${FILE_COUNT} * 150 / 1024 / 1024 ))
echo “Recommended Heap Size: ${HEAP_SIZE} GB”

# NameNode调优分析
=== NameNode Memory Analysis ===
Heap Configuration:
MaxHeapSize = 8589934592 (8192.0MB)
OldSize = 7158628352 (6827.0MB)

=== GC Analysis ===
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 75.00 50.00 30.00 93.75 93.75 100 5.000 5 2.500 7.500
0.00 75.00 60.00 30.00 93.75 93.75 100 5.000 5 2.500 7.500
0.00 75.00 70.00 30.00 93.75 93.75 100 5.000 5 2.500 7.500

=== File Count ===
Total Files: 500000

=== Recommended Configuration ===
Recommended Heap Size: 71 GB
# 当前8GB堆内存不足，建议增加到71GB

4.2 DataNode性能调优案例

DataNode性能影响数据读写效率。学习交流加群风哥微信: itpux-com

# DataNode性能调优
# 1. 查看磁盘IO
iostat -x 1 5 | grep -E “sda|sdb”

# 2. 查看网络IO
sar -n DEV 1 5 | grep eth0

# 3. 调整DataNode配置
vi /bigdata/app/hadoop/etc/hadoop/hdfs-site.xml
# 增加处理线程数
# 增加传输线程数
# 调整心跳间隔

# 磁盘IO分析
Device rrqm/s wrqm/s r/s w/s rMB/s wMB/s %util
sda 0.00 10.00 50.00 100.00 5.00 10.00 85.00
sdb 0.00 5.00 30.00 50.00 3.00 5.00 60.00

# 网络IO分析
eth0: 1000.00 pkts/s 500.00 pkts/s 100.00 MB/s 50.00 MB/s

# 配置调整后重启DataNode
hdfs –daemon stop datanode
hdfs –daemon start datanode

# 验证效果
# 读写吞吐量提升30%

4.3 YARN资源调优案例

4.3.1 ResourceManager调优

# ResourceManager调优
# 查看资源使用
yarn rmadmin -getServiceState rm1
yarn application -list -appStates RUNNING

# 分析调度延迟
curl -s http://fgedu01:8088/ws/v1/cluster/metrics | python -m json.tool

# ResourceManager状态
active

# 运行中应用
Total Applications:10

# 集群指标
{
“clusterMetrics”: {
“appsSubmitted”: 100,
“appsRunning”: 10,
“appsPending”: 5,
“containersRunning”: 50,
“totalMB”: 491520,
“availableMB”: 245760,
“totalVirtualCores”: 48,
“availableVirtualCores”: 24
}
}

4.3.2 NodeManager调优

# NodeManager配置优化
cat /bigdata/app/hadoop/etc/hadoop/yarn-site.xml | grep -A3 “resource”

<!– NodeManager资源配置 –>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>81920</value> # 80GB
</property>

<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>8</value>
</property>

<!– Container资源配置 –>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>65536</value>
</property>

Part05-风哥经验总结与分享

5.1 监控运维最佳实践

在实际生产环境中，监控运维需要注意以下几点：from bigdata视频:www.itpux.com

风哥经验总结：
1. 建立完善的监控体系
2. 配置合理的告警阈值
3. 定期进行性能分析
4. 建立运维知识库
5. 制定应急预案

5.2 性能调优经验总结

5.2.1 调优建议

风哥提示：性能调优是一个持续的过程，需要根据业务变化不断调整。

调优注意事项：
– 调优前做好备份
– 逐步调整参数
– 验证调优效果
– 记录调优过程
– 避免过度调优

5.2.2 集群健康检查脚本

#!/bin/bash
# cluster_health_check.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

echo “=== Hadoop Cluster Health Check ===”
echo “Date: $(date)”

# 1. HDFS健康检查
echo “=== HDFS Health ===”
hdfs dfsadmin -report | grep -E “Live|Dead|Under replicated”

# 2. YARN健康检查
echo “=== YARN Health ===”
yarn node -list | grep -E “RUNNING|UNHEALTHY”

# 3. NameNode内存检查
echo “=== NameNode Memory ===”
jstat -gcutil $(pgrep -f NameNode) 1 1 | tail -1

# 4. 磁盘使用检查
echo “=== Disk Usage ===”
df -h | grep -E “/data|/bigdata”

# 5. 告警判断
LIVE_NODES=$(hdfs dfsadmin -report | grep “Live datanodes” | awk ‘{print $3}’)
if [ ${LIVE_NODES} -lt 6 ]; then
echo “WARNING: Live DataNodes is ${LIVE_NODES}, expected 6”
fi

# 健康检查输出
=== Hadoop Cluster Health Check ===
Date: Wed Jan 17 11:00:00 CST 2024

=== HDFS Health ===
Live datanodes: 6
Dead datanodes: 0
Under replicated blocks: 0

=== YARN Health ===
Total Nodes:6
Running: 6

=== NameNode Memory ===
0.00 75.00 50.00 30.00 93.75 93.75 100 5.000 5 2.500 7.500

=== Disk Usage ===
/dev/sda1 20T 10T 10T 50% /bigdata

# 集群状态健康

本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html