大数据教程FG139-大数据集群故障处理实战

本教程主要介绍大数据集群故障处理的方法和实战技巧，包括HDFS故障处理、YARN故障处理、MapReduce故障处理等内容。风哥教程参考bigdata官方文档故障处理指南、日志分析说明等相关内容。

通过本教程的学习，您将掌握大数据集群的故障处理方法，确保集群的稳定运行和快速恢复。

目录大纲

Part01-基础概念与理论知识
Part02-生产环境规划与建议
Part03-生产环境项目实施方案
Part04-生产案例与实战讲解
Part05-风哥经验总结与分享

Part01-基础概念与理论知识

1.1 故障处理概述

大数据集群故障处理是指在集群遇到故障时，通过各种技术手段，快速定位和解决故障，确保集群的稳定运行和数据的安全性，主要包括：

故障检测：及时发现集群中的故障
故障定位：准确定位故障的原因和位置
故障处理：采取有效的措施解决故障
故障恢复：恢复集群的正常运行
故障预防：采取措施预防故障的发生

故障处理是大数据集群管理的重要组成部分，需要建立完善的故障处理机制，确保在遇到故障时能够快速响应和处理，学习交流加群风哥微信: itpux-com

1.2 故障类型

常见的故障类型：

硬件故障：服务器、磁盘、网络等硬件设备故障
软件故障：操作系统、应用程序等软件故障
网络故障：网络连接、网络设备等故障
数据故障：数据丢失、数据损坏等故障
配置故障：配置错误、配置冲突等故障
性能故障：性能下降、资源不足等故障

1.3 故障处理流程

故障处理流程：

故障检测：通过监控系统或用户反馈发现故障
故障定位：分析日志、检查状态，定位故障原因
故障处理：采取措施解决故障，如重启服务、修复硬件等
故障恢复：恢复集群的正常运行，验证服务是否正常
故障分析：分析故障原因，总结经验教训
故障预防：采取措施预防类似故障的发生

Part02-生产环境规划与建议

2.1 故障预防规划

风哥提示：故障预防规划应根据集群规模和业务需求，制定合理的故障预防策略，减少故障的发生。

故障预防规划建议：

硬件冗余：使用冗余的硬件设备，如多台服务器、多块磁盘等
软件高可用：部署高可用的软件架构，如HDFS HA、YARN HA等
网络冗余：使用冗余的网络设备和链路，确保网络的可靠性
数据备份：定期备份数据，防止数据丢失
监控告警：建立完善的监控告警机制，及时发现潜在问题
定期维护：定期进行系统维护，如清理日志、更新补丁等

2.2 故障处理策略

故障处理策略建议：

快速响应：建立故障响应机制，确保快速响应故障
分级处理：根据故障的严重程度，分级处理故障
团队协作：组织专业团队，共同处理故障
文档化：记录故障处理过程和结果，便于后续参考
持续改进：根据故障处理经验，持续改进故障处理策略

2.3 故障演练计划

故障演练计划建议：

定期演练：定期进行故障演练，提高故障处理能力
模拟故障：模拟各种故障场景，测试故障处理流程
演练评估：评估演练效果，发现问题并改进
培训学习：通过演练，培训团队的故障处理技能

Part03-生产环境项目实施方案

3.1 HDFS故障处理

配置HDFS故障处理：

# 1. HDFS故障处理
## 1.1 NameNode故障
### 1.1.1 NameNode无法启动
检查NameNode日志：
tail -f /bigdata/fgdata/logs/hadoop-fgedu-namenode-fgedu01.log

检查NameNode数据目录：
ls -la /bigdata/fgdata/hdfs/namenode

检查NameNode配置：
vi /bigdata/app/hadoop/etc/hadoop/hdfs-site.xml

### 1.1.2 NameNode故障转移
检查JournalNode状态：
jps | grep JournalNode

手动故障转移：
hdfs haadmin -failover nn1 nn2

### 1.2 DataNode故障
### 1.2.1 DataNode无法启动
检查DataNode日志：
tail -f /bigdata/fgdata/logs/hadoop-fgedu-datanode-fgedu01.log

检查DataNode数据目录：
ls -la /bigdata/fgdata/hdfs/datanode

重启DataNode：
hdfs –daemon start datanode

### 1.2.2 DataNode磁盘故障
检查DataNode状态：
hdfs dfsadmin -report

修复DataNode：
hdfs dfsadmin -refreshNodes

### 1.3 数据块丢失
检查数据块状态：
hdfs fsck /

修复数据块：
hdfs fsck / -delete
hdfs fsck / -move

## 1.4 网络故障
检查网络连接：
ping fgedu01
telnet fgedu01 9000

检查防火墙：
firewall-cmd –list-all

## 1.5 配置故障
检查配置文件：
vi /bigdata/app/hadoop/etc/hadoop/core-site.xml
vi /bigdata/app/hadoop/etc/hadoop/hdfs-site.xml

重启HDFS服务：
stop-dfs.sh
start-dfs.sh

3.2 YARN故障处理

配置YARN故障处理：

# 1. YARN故障处理
## 1.1 ResourceManager故障
### 1.1.1 ResourceManager无法启动
检查ResourceManager日志：
tail -f /bigdata/fgdata/logs/yarn-fgedu-resourcemanager-fgedu01.log

检查ResourceManager配置：
vi /bigdata/app/hadoop/etc/hadoop/yarn-site.xml

重启ResourceManager：
yarn –daemon start resourcemanager

### 1.1.2 ResourceManager故障转移
检查ZooKeeper状态：
zkServer.sh status

手动故障转移：
yarn rmadmin -failover rm1 rm2

## 1.2 NodeManager故障
### 1.2.1 NodeManager无法启动
检查NodeManager日志：
tail -f /bigdata/fgdata/logs/yarn-fgedu-nodemanager-fgedu01.log

检查NodeManager配置：
vi /bigdata/app/hadoop/etc/hadoop/yarn-site.xml

重启NodeManager：
yarn –daemon start nodemanager

## 1.3 作业失败
检查作业日志：
yarn logs -applicationId

检查作业状态：
yarn application -status

重新提交作业：
yarn jar /bigdata/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar pi 10 1000000

## 1.4 资源不足
检查资源使用情况：
yarn node -list -all
yarn top

调整资源分配：
vi /bigdata/app/hadoop/etc/hadoop/yarn-site.xml

3.3 MapReduce故障处理

配置MapReduce故障处理：

# 1. MapReduce故障处理
## 1.1 作业执行失败
检查作业日志：
yarn logs -applicationId

检查作业配置：
vi /bigdata/app/hadoop/etc/hadoop/mapred-site.xml

调整作业参数：
hadoop jar /bigdata/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar pi -Dmapreduce.map.memory.mb=4096 10 1000000

## 1.2 内存不足
检查内存配置：
vi /bigdata/app/hadoop/etc/hadoop/mapred-site.xml

调整内存参数： mapreduce.map.memory.mb
4096 mapreduce.reduce.memory.mb
8192

## 1.3 数据倾斜
检查数据分布：
hadoop jar /bigdata/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar wordcount -Dmapreduce.job.reduces=10 /user/fgedu/input /user/fgedu/output

优化数据分区：
hadoop jar /bigdata/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar wordcount -Dmapreduce.job.partitioner.class=org.apache.hadoop.mapreduce.lib.partition.HashPartitioner /user/fgedu/input /user/fgedu/output

## 1.4 网络超时
检查网络连接：
ping fgedu01

调整网络超时参数：
vi /bigdata/app/hadoop/etc/hadoop/core-site.xml ipc.client.connect.timeout
60000 ipc.client.connect.max.retries
10

Part04-生产案例与实战讲解

4.1 HDFS故障处理实战

案例：HDFS DataNode故障处理

# 检查DataNode状态

$ hdfs dfsadmin -report
Configured Capacity: 30963660800 (28.85 GB)
Present Capacity: 27867295744 (25.96 GB)
DFS Remaining: 27867293696 (25.96 GB)
DFS Used: 2048 (2 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

Live datanodes (2):

Name: 192.168.1.10:9866 (fgedu01)
Hostname: fgedu01
Decommission Status : Normal
Configured Capacity: 10321220266 (9.62 GB)
DFS Used: 683 (683 B)
Non DFS Used: 1032122026 (984 MB)
DFS Remaining: 9289100557 (8.65 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.00%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Apr 08 10:00:00 CST 2026

Name: 192.168.1.11:9866 (fgedu02)
Hostname: fgedu02
Decommission Status : Normal
Configured Capacity: 10321220266 (9.62 GB)
DFS Used: 682 (682 B)
Non DFS Used: 1032122026 (984 MB)
DFS Remaining: 9289100558 (8.65 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.00%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Apr 08 10:00:00 CST 2026

Dead datanodes (1):
Name: 192.168.1.12:9866 (fgedu03)
Hostname: fgedu03
Decommission Status : Normal
Configured Capacity: 10321220266 (9.62 GB)
DFS Used: 683 (683 B)
Non DFS Used: 1032122026 (984 MB)
DFS Remaining: 9289100557 (8.65 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.00%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Apr 08 09:00:00 CST 2026

# 检查DataNode日志

$ tail -f /bigdata/fgdata/logs/hadoop-fgedu-datanode-fgedu03.log
2026-04-08 09:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788)
at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1542)
at org.apache.hadoop.ipc.Client.call(Client.java:1486)
at org.apache.hadoop.ipc.Client.call(Client.java:1442)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at org.apache.hadoop.hdfs.server.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:106)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:802)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:283)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:870)
at java.lang.Thread.run(Thread.java:748)

# 检查网络连接

$ ping fgedu01
PING fgedu01 (192.168.1.10) 56(84) bytes of data.
64 bytes from fgedu01 (192.168.1.10): icmp_seq=1 ttl=64 time=0.123 ms
64 bytes from fgedu01 (192.168.1.10): icmp_seq=2 ttl=64 time=0.134 ms
64 bytes from fgedu01 (192.168.1.10): icmp_seq=3 ttl=64 time=0.145 ms

# 检查NameNode状态

$ jps | grep NameNode
23456 NameNode

# 重启DataNode

$ hdfs –daemon start datanode
2026-04-08 10:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = fgedu03/192.168.1.12
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.3.5
STARTUP_MSG: classpath = /bigdata/app/hadoop/etc/hadoop:/bigdata/app/hadoop/share/hadoop/common/lib/*:/bigdata/app/hadoop/share/hadoop/common/*:/bigdata/app/hadoop/share/hadoop/hdfs:/bigdata/app/hadoop/share/hadoop/hdfs/lib/*:/bigdata/app/hadoop/share/hadoop/hdfs/*:/bigdata/app/hadoop/share/hadoop/mapreduce/lib/*:/bigdata/app/hadoop/share/hadoop/mapreduce/*:/bigdata/app/hadoop/share/hadoop/yarn:/bigdata/app/hadoop/share/hadoop/yarn/lib/*:/bigdata/app/hadoop/share/hadoop/yarn/*
STARTUP_MSG: build = https://github.com/apache/hadoop.git -r 1a2b3c4d5e6f7g8h9i0j
STARTUP_MSG: java = 1.8.0_381
************************************************************/
2026-04-08 10:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2026-04-08 10:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: createAndStartDataNode
2026-04-08 10:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting data node 192.168.1.12:9866
2026-04-08 10:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Connecting to namenode fgedu01/192.168.1.10:9000
2026-04-08 10:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Successfully registered with namenode fgedu01/192.168.1.10:9000
2026-04-08 10:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode RPC server is running on fgedu03:9866
2026-04-08 10:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
2026-04-08 10:00:00,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Shutdown complete.

# 检查DataNode状态

Live datanodes (3):

Name: 192.168.1.12:9866 (fgedu03)
Hostname: fgedu03
Decommission Status : Normal
Configured Capacity: 10321220266 (9.62 GB)
DFS Used: 683 (683 B)
Non DFS Used: 1032122026 (984 MB)
DFS Remaining: 9289100557 (8.65 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.00%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Apr 08 10:00:00 CST 2026

4.2 YARN故障处理实战

案例：YARN ResourceManager故障处理

# 检查ResourceManager状态

$ jps | grep ResourceManager

# 检查ResourceManager日志

$ tail -f /bigdata/fgdata/logs/yarn-fgedu-resourcemanager-fgedu01.log
2026-04-08 10:00:00,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting ResourceManager
STARTUP_MSG: host = fgedu01/192.168.1.10
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.3.5
STARTUP_MSG: classpath = /bigdata/app/hadoop/etc/hadoop:/bigdata/app/hadoop/share/hadoop/common/lib/*:/bigdata/app/hadoop/share/hadoop/common/*:/bigdata/app/hadoop/share/hadoop/hdfs:/bigdata/app/hadoop/share/hadoop/hdfs/lib/*:/bigdata/app/hadoop/share/hadoop/hdfs/*:/bigdata/app/hadoop/share/hadoop/mapreduce/lib/*:/bigdata/app/hadoop/share/hadoop/mapreduce/*:/bigdata/app/hadoop/share/hadoop/yarn:/bigdata/app/hadoop/share/hadoop/yarn/lib/*:/bigdata/app/hadoop/share/hadoop/yarn/*
STARTUP_MSG: build = https://github.com/apache/hadoop.git -r 1a2b3c4d5e6f7g8h9i0j
STARTUP_MSG: java = 1.8.0_381
************************************************************/
2026-04-08 10:00:00,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: registered UNIX signal handlers for [TERM, HUP, INT]
2026-04-08 10:00:00,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: createAndStartResourceManager
2026-04-08 10:00:00,000 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
java.io.IOException: Failed to bind to fgedu01/192.168.1.10:8030
at org.apache.hadoop.ipc.Server.bind(Server.java:423)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:590)
at org.apache.hadoop.ipc.Server.(Server.java:2436)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:989)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:530)
at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:741)
at org.apache.hadoop.yarn.ipc.RPCUtil.getServer(RPCUtil.java:53)
at org.apache.hadoop.yarn.ipc.HAProtocolServer$Server.(HAProtocolServer.java:60)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1234)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1412)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1561)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:419)
… 12 more

# 检查端口占用

$ netstat -tuln | grep 8030
tcp 0 0 0.0.0.0:8030 0.0.0.0:* LISTEN

# 查找占用端口的进程

$ lsof -i :8030
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 12345 fgedu 42u IPv6 12345 0t0 TCP *:8030 (LISTEN)

# 终止占用端口的进程

$ kill -9 12345

# 重启ResourceManager

$ yarn –daemon start resourcemanager
2026-04-08 10:00:00,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting ResourceManager
STARTUP_MSG: host = fgedu01/192.168.1.10
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.3.5
STARTUP_MSG: classpath = /bigdata/app/hadoop/etc/hadoop:/bigdata/app/hadoop/share/hadoop/common/lib/*:/bigdata/app/hadoop/share/hadoop/common/*:/bigdata/app/hadoop/share/hadoop/hdfs:/bigdata/app/hadoop/share/hadoop/hdfs/lib/*:/bigdata/app/hadoop/share/hadoop/hdfs/*:/bigdata/app/hadoop/share/hadoop/mapreduce/lib/*:/bigdata/app/hadoop/share/hadoop/mapreduce/*:/bigdata/app/hadoop/share/hadoop/yarn:/bigdata/app/hadoop/share/hadoop/yarn/lib/*:/bigdata/app/hadoop/share/hadoop/yarn/*
STARTUP_MSG: build = https://github.com/apache/hadoop.git -r 1a2b3c4d5e6f7g8h9i0j
STARTUP_MSG: java = 1.8.0_381
************************************************************/
2026-04-08 10:00:00,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: registered UNIX signal handlers for [TERM, HUP, INT]
2026-04-08 10:00:00,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: createAndStartResourceManager
2026-04-08 10:00:00,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: ResourceManager WebApp is available at http://fgedu01:8088/
2026-04-08 10:00:00,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Scheduler initialized with 3 nodes and 64 vcores, 192 GB RAM
2026-04-08 10:00:00,000 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: ResourceManager started on fgedu01/192.168.1.10

# 检查ResourceManager状态

$ jps | grep ResourceManager
24123 ResourceManager

4.3 MapReduce故障处理实战

案例：MapReduce作业执行失败处理

# 提交MapReduce作业

# 查看作业日志

$ yarn logs -applicationId application_1234567890_0001
10:00:00 INFO mapreduce.Job: Running job: job_1234567890_0001
10:00:00 INFO mapreduce.Job: Job job_1234567890_0001 running in uber mode : false
10:00:00 INFO mapreduce.Job: map 0% reduce 0%
10:00:00 INFO mapreduce.Job: map 10% reduce 0%
10:00:00 INFO mapreduce.Job: map 20% reduce 0%
10:00:00 INFO mapreduce.Job: map 30% reduce 0%
10:00:00 INFO mapreduce.Job: map 40% reduce 0%
10:00:00 INFO mapreduce.Job: map 50% reduce 0%
10:00:00 INFO mapreduce.Job: map 60% reduce 0%
10:00:00 INFO mapreduce.Job: map 70% reduce 0%
10:00:00 INFO mapreduce.Job: map 80% reduce 0%
10:00:00 INFO mapreduce.Job: map 90% reduce 0%
10:00:00 INFO mapreduce.Job: map 100% reduce 0%
10:00:00 INFO mapreduce.Job: map 100% reduce 10%
10:00:00 INFO mapreduce.Job: map 100% reduce 20%
10:00:00 INFO mapreduce.Job: map 100% reduce 30%
10:00:00 INFO mapreduce.Job: map 100% reduce 40%
10:00:00 INFO mapreduce.Job: map 100% reduce 50%
10:00:00 INFO mapreduce.Job: map 100% reduce 60%
10:00:00 INFO mapreduce.Job: map 100% reduce 70%
10:00:00 INFO mapreduce.Job: map 100% reduce 80%
10:00:00 INFO mapreduce.Job: map 100% reduce 90%
10:00:00 INFO mapreduce.Job: map 100% reduce 100%
10:00:00 INFO mapreduce.Job: Job job_1234567890_0001 failed with state FAILED due to: Task failed task_1234567890_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

Container killed by YARN for exceeding memory limits. 1024 MB of 1024 MB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

# 调整MapReduce内存配置

$ vi /bigdata/app/hadoop/etc/hadoop/mapred-site.xml mapreduce.map.memory.mb
4096 mapreduce.map.java.opts
-Xmx3072m

# 重新提交作业

$ hadoop jar /bigdata/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar pi 10 1000000
10:00:00 INFO client.RMProxy: Connecting to ResourceManager at fgedu01/192.168.1.10:8032
10:00:00 INFO client.RMProxy: Connecting to ResourceManager at fgedu01/192.168.1.10:8032
10:00:00 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/fgedu/.staging/job_1234567890_0002
10:00:00 INFO input.FileInputFormat: Total input files to process : 1
10:00:00 INFO mapreduce.JobSubmitter: number of splits:10
10:00:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1234567890_0002
10:00:00 INFO mapreduce.JobSubmitter: Executing with tokens: []
10:00:00 INFO conf.Configuration: resource-types.xml not found
10:00:00 INFO resource.ResourceUtils: Unable to find ‘resource-types.xml’.
10:00:00 INFO resource.ResourceUtils: Adding resource type – name = memory-mb, units = MB, type = COUNTABLE
10:00:00 INFO resource.ResourceUtils: Adding resource type – name = vcores, units = , type = COUNTABLE
10:00:00 INFO mapreduce.JobSubmitter: Submitting job to ResourceManager
10:00:00 INFO mapreduce.Job: The url to track the job: http://fgedu01:8088/proxy/application_1234567890_0002/
10:00:00 INFO mapreduce.Job: Running job: job_1234567890_0002
10:00:00 INFO mapreduce.Job: Job job_1234567890_0002 running in uber mode : false
10:00:00 INFO mapreduce.Job: map 0% reduce 0%
10:00:00 INFO mapreduce.Job: map 10% reduce 0%
10:00:00 INFO mapreduce.Job: map 20% reduce 0%
10:00:00 INFO mapreduce.Job: map 30% reduce 0%
10:00:00 INFO mapreduce.Job: map 40% reduce 0%
10:00:00 INFO mapreduce.Job: map 50% reduce 0%
10:00:00 INFO mapreduce.Job: map 60% reduce 0%
10:00:00 INFO mapreduce.Job: map 70% reduce 0%
10:00:00 INFO mapreduce.Job: map 80% reduce 0%
10:00:00 INFO mapreduce.Job: map 90% reduce 0%
10:00:00 INFO mapreduce.Job: map 100% reduce 0%
10:00:00 INFO mapreduce.Job: map 100% reduce 10%
10:00:00 INFO mapreduce.Job: map 100% reduce 20%
10:00:00 INFO mapreduce.Job: map 100% reduce 30%
10:00:00 INFO mapreduce.Job: map 100% reduce 40%
10:00:00 INFO mapreduce.Job: map 100% reduce 50%
10:00:00 INFO mapreduce.Job: map 100% reduce 60%
10:00:00 INFO mapreduce.Job: map 100% reduce 70%
10:00:00 INFO mapreduce.Job: map 100% reduce 80%
10:00:00 INFO mapreduce.Job: map 100% reduce 90%
10:00:00 INFO mapreduce.Job: map 100% reduce 100%
10:00:00 INFO mapreduce.Job: Job job_1234567890_0002 completed successfully
10:00:00 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=1458
FILE: Number of bytes written=10485760
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=213
HDFS: Number of bytes written=123
HDFS: Number of read operations=40
HDFS: Number of large read operations=0
HDFS: Number of write operations=20
Job Counters
Launched map tasks=10
Launched reduce tasks=1
Data-local map tasks=10
Rack-local map tasks=0
Total time spent by all maps in occupied slots (ms)=10000
Total time spent by all reduces in occupied slots (ms)=5000
Total time spent by all map tasks (ms)=10000
Total time spent by all reduce tasks (ms)=5000
Total vcore-milliseconds taken by all map tasks=10000
Total vcore-milliseconds taken by all reduce tasks=5000
Total megabyte-milliseconds taken by all map tasks=40960000
Total megabyte-milliseconds taken by all reduce tasks=20480000
Map-Reduce Framework
Map input records=10
Map output records=20
Map output bytes=1458
Map output materialized bytes=1458
Input split bytes=213
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=1458
Reduce input records=20
Reduce output records=0
Spilled Records=20
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=500
CPU time spent (ms)=5000
Physical memory (bytes) snapshot=4096000000
Virtual memory (bytes) snapshot=8192000000
Total committed heap usage (bytes)=4096000000
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=213
File Output Format Counters
Bytes Written=123
Job Finished in 30.0 seconds
Estimated value of Pi is 3.141592653589793

Part05-风哥经验总结与分享

5.1 常见问题解决方案

常见问题解决方案：

服务无法启动：检查日志，查看错误信息，检查配置文件，检查端口占用
服务运行异常：检查服务状态，查看日志，检查网络连接，检查资源使用情况
作业执行失败：查看作业日志，分析失败原因，调整作业参数，优化代码
数据丢失：使用备份恢复数据，检查数据块状态，调整副本数
性能下降：分析性能瓶颈，优化配置，调整资源分配
网络故障：检查网络连接，检查防火墙，检查网络设备

5.2 最佳实践分享

风哥提示：在故障处理过程中，应注重日志分析和问题定位，根据实际情况采取有效的解决措施，确保集群的稳定运行。

最佳实践分享：

建立监控告警机制：及时发现和处理故障
定期备份数据：防止数据丢失
建立故障处理流程：规范故障处理步骤
培训团队技能：提高团队的故障处理能力
文档化故障处理过程：便于后续参考
持续改进：根据故障处理经验，持续改进集群配置和管理

5.3 故障处理建议

故障处理建议：

快速响应：及时发现和处理故障，减少故障影响
准确定位：通过日志分析和状态检查，准确定位故障原因
有效处理：采取有效的措施解决故障，恢复服务
预防为主：采取措施预防故障的发生，如定期维护、监控告警等
团队协作：组织专业团队，共同处理故障
持续学习：不断学习故障处理技术，提高故障处理能力
更多视频教程www.fgedu.net.cn

通过本教程的学习，您已经掌握了大数据集群故障处理的方法和实战技巧。在实际生产环境中，应根据集群的实际情况和业务需求，建立完善的故障处理机制，及时发现和处理故障，确保集群的稳定运行和数据的安全性。学习交流加群风哥QQ113257174

更多学习教程公众号风哥教程itpux_com

from bigdata视频:www.itpux.com

本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html