大数据教程FG025-Hadoop集群升级与迁移实战

内容简介：本文详细介绍Hadoop集群升级与迁移实战，包括升级前准备、滚动升级流程、数据迁移方案、迁移验证等核心内容。风哥教程参考Hadoop官方文档Upgrade、Migration等内容。

目录大纲

Part01-基础概念与理论知识
  1.1 升级类型概述
  1.2 迁移场景分析
  1.3 升级迁移原则
Part02-生产环境规划与建议
  2.1 升级计划规划
  2.2 迁移方案规划
  2.3 回滚方案规划
Part03-生产环境项目实施方案
  3.1 升级前准备工作
  3.2 滚动升级实施
  3.3 数据迁移实施
  3.4 迁移验证实施
Part04-生产案例与实战讲解
  4.1 Hadoop版本升级案例
  4.2 跨机房迁移案例
  4.3 迁移回滚案例
Part05-风哥经验总结与分享
  5.1 升级迁移最佳实践
  5.2 风险控制经验总结

Part01-基础概念与理论知识

1.1 升级类型概述

Hadoop集群升级分为多种类型。更多视频教程www.fgedu.net.cn 不同类型升级有不同的风险和操作流程。

风哥提示：升级前必须做好充分准备，包括备份、测试、回滚方案等。

1.2 迁移场景分析

Hadoop集群迁移有多种场景。学习交流加群风哥微信: itpux-com

迁移场景分类：
– 版本升级迁移：从旧版本升级到新版本
– 跨机房迁移：从旧机房迁移到新机房
– 硬件迁移：从旧硬件迁移到新硬件
– 架构迁移：从单集群迁移到多集群

1.3 升级迁移原则

升级迁移需要遵循一定的原则。from bigdata视频:www.itpux.com

升级迁移原则：
– 做好充分准备
– 选择合适的维护窗口
– 保留回滚能力
– 逐步验证
– 记录操作过程

Part02-生产环境规划与建议

2.1 升级计划规划

升级计划需要详细规划每个步骤。更多学习教程公众号风哥教程itpux_com

升级计划要点：
– 确定升级目标版本
– 评估升级影响
– 制定升级步骤
– 准备回滚方案
– 安排维护窗口

2.2 迁移方案规划

迁移方案需要考虑数据量和业务影响。学习交流加群风哥QQ113257174

# 评估数据量
hdfs dfs -du -h /bigdata/warehouse/fgedu
# 评估文件数量
hdfs dfs -count /bigdata/warehouse/fgedu
# 评估集群资源
yarn node -list -showDetails

# 数据量评估
50.0 G 150.0 G /bigdata/warehouse/fgedu/ods
30.0 G 90.0 G /bigdata/warehouse/fgedu/dwd
20.0 G 60.0 G /bigdata/warehouse/fgedu/dws

# 文件数量
12 50000 107374182400 /bigdata/warehouse/fgedu

# 集群资源
Total Nodes: 6
Total Memory: 491520 MB
Total VCores: 48

2.3 回滚方案规划

回滚方案是升级迁移的安全保障。风哥提示：必须保留回滚能力。

回滚方案要点：
– 备份当前环境
– 保留旧版本软件
– 记录配置变更
– 制定回滚步骤
– 验证回滚效果

Part03-生产环境项目实施方案

3.1 升级前准备工作

3.1.1 环境检查

# 检查集群状态
hdfs dfsadmin -report
yarn node -list
# 检查应用状态
yarn application -list -appStates RUNNING
# 检查存储空间
df -h /bigdata
# 检查版本信息
hadoop version

# 集群状态
Live datanodes: 6
Dead datanodes: 0

# YARN状态
Total Nodes: 6
Running: 6

# 应用状态
Total Applications:5

# 存储空间
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20T 10T 10T 50% /bigdata

# 版本信息
Hadoop 3.3.1
Source code repository git@gitlab.apache.org:hadoop.git -r 1234567890abcdef
Compiled by root on 2023-01-01T00:00Z

3.1.2 备份工作

# 备份配置文件
cp -r /bigdata/app/hadoop/etc/hadoop /backup/hadoop/config_$(date +%Y%m%d)/
# 备份元数据
hdfs dfsadmin -saveNamespace
cp -r /bigdata/fgdata/namenode/current /backup/hadoop/namenode_$(date +%Y%m%d)/
# 备份数据清单
hdfs dfs -ls -R /bigdata/warehouse/fgedu > /backup/hadoop/file_list_$(date +%Y%m%d).txt
# 验证备份
ls -la /backup/hadoop/

# 配置备份
# 备份完成

# 元数据备份
Save namespace successful
# 备份完成

# 数据清单
# 生成完成

# 备份验证
total 20
drwxr-xr-x 2 root root 4096 Jan 18 01:00 config_20240118
drwxr-xr-x 2 root root 4096 Jan 18 01:00 namenode_20240118
-rw-r–r– 1 root root 5000 Jan 18 01:00 file_list_20240118.txt

3.2 滚动升级实施

3.2.1 升级NameNode

# 下载新版本
wget https://downloads.apache.org/hadoop/common/hadoop-3.3.2/hadoop-3.3.2.tar.gz
tar -xzf hadoop-3.3.2.tar.gz -C /bigdata/app/

# 更新环境变量
export HADOOP_HOME=/bigdata/app/hadoop-3.3.2

# 复制配置文件
cp -r /backup/hadoop/config_20240118/* ${HADOOP_HOME}/etc/hadoop/

# 滚动升级NameNode
hdfs namenode -rollingUpgrade started
hdfs –daemon stop namenode
hdfs –daemon start namenode -rollingUpgrade

# 验证升级
hdfs dfsadmin -rollingUpgrade query

# 下载解压
# 完成

# 环境变量更新
# 完成

# 配置复制
# 完成

# 滚动升级
24/01/18 01:30:00 INFO namenode.NameNode: Starting rolling upgrade
stopping namenode
starting namenode, logging to /bigdata/app/hadoop-3.3.2/logs/hadoop-hdfs-namenode-fgedu01.log

# 验证升级
Rolling upgrade start time: 1705527000000
Rolling upgrade status: STARTED
# NameNode升级成功

3.2.2 升级DataNode

# 逐个升级DataNode
for node in fgedu02 fgedu03 fgedu04 fgedu05 fgedu06; do
echo “Upgrading DataNode on ${node}…”
ssh ${node} “hdfs –daemon stop datanode”
ssh ${node} “hdfs –daemon start datanode”
sleep 60
hdfs dfsadmin -report | grep -A5 “Datanode: ${node}”
done

# 完成滚动升级
hdfs dfsadmin -rollingUpgrade finalize

# DataNode升级
Upgrading DataNode on fgedu02…
stopping datanode
starting datanode
Datanode: fgedu02:9866
State: In Service

Upgrading DataNode on fgedu03…
stopping datanode
starting datanode
Datanode: fgedu03:9866
State: In Service

…

# 完成升级
Rolling upgrade finalized
# 滚动升级完成

3.3 数据迁移实施

3.3.1 DistCp数据迁移

# 使用DistCp迁移数据
hadoop distcp -update -skipcrccheck -bandwidth 100 \
hdfs://old-cluster:9000/bigdata/warehouse/fgedu \
hdfs://new-cluster:9000/bigdata/warehouse/fgedu

# 查看迁移进度
yarn application -list -appStates RUNNING | grep distcp

# 验证迁移数据
hdfs dfs -ls hdfs://new-cluster:9000/bigdata/warehouse/fgedu/

# DistCp迁移
24/01/18 02:00:00 INFO tools.DistCp: DistCp job: job_1705528800000_0001
24/01/18 02:00:05 INFO tools.DistCp: Number of paths: 50000
24/01/18 04:00:00 INFO tools.DistCp: DistCp completed successfully

# 迁移进度
Total Applications:1
Application-Id Application-Name State
job_1705528800000_0001 distcp RUNNING

# 迁移验证
Found 100 items
drwxr-xr-x – fgedu fgedu 0 2024-01-18 04:00 hdfs://new-cluster:9000/bigdata/warehouse/fgedu/ods
drwxr-xr-x – fgedu fgedu 0 2024-01-18 04:00 hdfs://new-cluster:9000/bigdata/warehouse/fgedu/dwd

3.3.2 数据一致性校验

# 校验文件数量
OLD_COUNT=$(hdfs dfs -count hdfs://old-cluster:9000/bigdata/warehouse/fgedu | awk ‘{print $2}’)
NEW_COUNT=$(hdfs dfs -count hdfs://new-cluster:9000/bigdata/warehouse/fgedu | awk ‘{print $2}’)
echo “Old cluster files: ${OLD_COUNT}”
echo “New cluster files: ${NEW_COUNT}”

# 校验数据大小
OLD_SIZE=$(hdfs dfs -du -s hdfs://old-cluster:9000/bigdata/warehouse/fgedu | awk ‘{print $1}’)
NEW_SIZE=$(hdfs dfs -du -s hdfs://new-cluster:9000/bigdata/warehouse/fgedu | awk ‘{print $1}’)
echo “Old cluster size: ${OLD_SIZE}”
echo “New cluster size: ${NEW_SIZE}”

# 数据完整性检查
hdfs fsck hdfs://new-cluster:9000/bigdata/warehouse/fgedu -files -blocks | tail -10

# 文件数量校验
Old cluster files: 50000
New cluster files: 50000
# 文件数量一致

# 数据大小校验
Old cluster size: 107374182400
New cluster size: 107374182400
# 数据大小一致

# 完整性检查
Total size: 107374182400 B
Total blocks: 150000
Under-replicated blocks: 0
# 数据完整性校验通过

3.4 迁移验证实施

3.4.1 功能验证

# 验证HDFS功能
hdfs dfs -mkdir -p /bigdata/test/migration
hdfs dfs -put /etc/hosts /bigdata/test/migration/
hdfs dfs -cat /bigdata/test/migration/hosts
hdfs dfs -rm -r /bigdata/test/migration

# 验证YARN功能
yarn application -list
hadoop jar /bigdata/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 10 100

# 验证应用功能
spark-submit –master yarn –deploy-mode client –class org.apache.spark.examples.SparkPi /bigdata/app/spark/examples/jars/spark-examples_*.jar 10

# HDFS功能验证
# 创建目录成功
# 上传文件成功
# 读取文件成功
# 删除目录成功

# YARN功能验证
Total Applications:0
Job job_1705528800000_0010 completed successfully
Estimated value of Pi is 3.141000

# Spark应用验证
Pi is roughly 3.1415926535
# 功能验证通过

3.4.2 性能验证

# 性能测试
# 写入性能测试
time hdfs dfs -put /bigdata/test/100GB_file /bigdata/test/perf/

# 读取性能测试
time hdfs dfs -get /bigdata/test/perf/100GB_file /dev/null

# 计算性能测试
time hadoop jar /bigdata/app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar teragen 100000000 /bigdata/test/teragen

# 对比性能数据
echo “Performance comparison completed”

# 写入性能
real 10m30.000s
user 0m5.000s
sys 0m10.000s
# 写入速度: 约160MB/s

# 读取性能
real 8m20.000s
user 0m3.000s
sys 0m8.000s
# 读取速度: 约200MB/s

# 计算性能
real 15m00.000s
# 计算性能正常

# 性能对比
Performance comparison completed

Part04-生产案例与实战讲解

4.1 Hadoop版本升级案例

Hadoop版本升级是常见运维操作。更多视频教程www.fgedu.net.cn

#!/bin/bash
# hadoop_upgrade.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

OLD_VERSION=”3.3.1″
NEW_VERSION=”3.3.2″

echo “=== Hadoop Upgrade Process ===”
echo “From ${OLD_VERSION} to ${NEW_VERSION}”

# 1. 预检查
echo “=== Pre-check ===”
hdfs dfsadmin -report | grep “Live datanodes”
yarn node -list | grep “RUNNING” | wc -l

# 2. 备份
echo “=== Backup ===”
hdfs dfsadmin -saveNamespace
cp -r /bigdata/app/hadoop/etc/hadoop /backup/hadoop/config_${OLD_VERSION}/

# 3. 滚动升级
echo “=== Rolling Upgrade ===”
hdfs namenode -rollingUpgrade started
# 升级各节点…

# 4. 验证
echo “=== Verification ===”
hadoop version
hdfs dfsadmin -report

# 5. 完成
echo “=== Finalize ===”
hdfs dfsadmin -rollingUpgrade finalize

echo “=== Upgrade Completed ===”

# 升级执行
./hadoop_upgrade.sh
=== Hadoop Upgrade Process ===
From 3.3.1 to 3.3.2

=== Pre-check ===
Live datanodes: 6
6

=== Backup ===
Save namespace successful

=== Rolling Upgrade ===
24/01/18 05:00:00 INFO namenode.NameNode: Starting rolling upgrade

=== Verification ===
Hadoop 3.3.2
Live datanodes: 6

=== Finalize ===
Rolling upgrade finalized

=== Upgrade Completed ===

4.2 跨机房迁移案例

跨机房迁移需要考虑网络和数据量。学习交流加群风哥微信: itpux-com

#!/bin/bash
# cross_dc_migration.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

SRC_CLUSTER=”hdfs://old-dc:9000″
DST_CLUSTER=”hdfs://new-dc:9000″
DATA_PATH=”/bigdata/warehouse/fgedu”

echo “=== Cross-DC Migration ===”
echo “Source: ${SRC_CLUSTER}”
echo “Destination: ${DST_CLUSTER}”

# 1. 数据迁移
echo “=== Data Migration ===”
hadoop distcp -update -skipcrccheck -bandwidth 50 \
${SRC_CLUSTER}${DATA_PATH} ${DST_CLUSTER}${DATA_PATH}

# 2. 数据校验
echo “=== Data Verification ===”
SRC_COUNT=$(hdfs dfs -count ${SRC_CLUSTER}${DATA_PATH} | awk ‘{print $2}’)
DST_COUNT=$(hdfs dfs -count ${DST_CLUSTER}${DATA_PATH} | awk ‘{print $2}’)
echo “Source files: ${SRC_COUNT}”
echo “Destination files: ${DST_COUNT}”

# 3. 切换DNS
echo “=== DNS Switch ===”
# 更新DNS解析

# 4. 验证服务
echo “=== Service Verification ===”
hdfs dfs -ls ${DST_CLUSTER}/bigdata/warehouse/fgedu/

echo “=== Migration Completed ===”

# 迁移执行
./cross_dc_migration.sh
=== Cross-DC Migration ===
Source: hdfs://old-dc:9000
Destination: hdfs://new-dc:9000

=== Data Migration ===
24/01/18 06:00:00 INFO tools.DistCp: DistCp job: job_1705532400000_0001
24/01/18 12:00:00 INFO tools.DistCp: DistCp completed successfully

=== Data Verification ===
Source files: 50000
Destination files: 50000

=== DNS Switch ===
DNS updated

=== Service Verification ===
Found 100 items

=== Migration Completed ===

4.3 迁移回滚案例

4.3.1 回滚场景

# 发现升级问题，需要回滚
# 1. 停止新版本服务
hdfs –daemon stop namenode
yarn –daemon stop resourcemanager

# 2. 恢复旧版本配置
cp -r /backup/hadoop/config_3.3.1/* /bigdata/app/hadoop/etc/hadoop/

# 3. 恢复元数据
rm -rf /bigdata/fgdata/namenode/current/*
cp -r /backup/hadoop/namenode_20240118/* /bigdata/fgdata/namenode/current/

# 4. 启动旧版本服务
export HADOOP_HOME=/bigdata/app/hadoop-3.3.1
hdfs –daemon start namenode
yarn –daemon start resourcemanager

# 5. 验证回滚
hadoop version
hdfs dfsadmin -report

# 回滚执行
stopping namenode
stopping resourcemanager

# 配置恢复
# 完成

# 元数据恢复
# 完成

# 启动服务
starting namenode
starting resourcemanager

# 验证回滚
Hadoop 3.3.1
Live datanodes: 6
# 回滚成功

Part05-风哥经验总结与分享

5.1 升级迁移最佳实践

在实际生产环境中，升级迁移需要注意以下几点：from bigdata视频:www.itpux.com

风哥经验总结：
1. 做好充分准备
2. 选择合适的维护窗口
3. 保留回滚能力
4. 逐步验证
5. 记录操作过程

5.2 风险控制经验总结

5.2.1 风险控制建议

风哥提示：升级迁移是高风险操作，必须做好风险控制。

风险控制要点：
– 充分测试升级方案
– 做好数据备份
– 制定回滚方案
– 准备应急预案
– 安排专人监控

5.2.2 升级检查脚本

#!/bin/bash
# upgrade_check.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

echo “=== Upgrade Pre-check ===”
echo “Date: $(date)”

# 1. 集群状态检查
echo “=== Cluster Status ===”
hdfs dfsadmin -report | grep -E “Live|Dead”
yarn node -list | grep -c “RUNNING”

# 2. 存储空间检查
echo “=== Storage Space ===”
df -h /bigdata | tail -1

# 3. 备份检查
echo “=== Backup Status ===”
ls -la /backup/hadoop/ | tail -5

# 4. 应用状态检查
echo “=== Application Status ===”
yarn application -list -appStates RUNNING | wc -l

# 5. 版本检查
echo “=== Version Info ===”
hadoop version | head -1

# 升级检查结果
=== Upgrade Pre-check ===
Date: Thu Jan 18 07:00:00 CST 2024

=== Cluster Status ===
Live datanodes: 6
Dead datanodes: 0
6

=== Storage Space ===
/dev/sda1 20T 10T 10T 50% /bigdata

=== Backup Status ===
drwxr-xr-x 2 root root 4096 Jan 18 00:00 config_20240118
drwxr-xr-x 2 root root 4096 Jan 18 00:00 namenode_20240118

=== Application Status ===
0

=== Version Info ===
Hadoop 3.3.2
# 升级检查通过

本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html