1. 硬件环境检查
在安装CDH之前,必须对服务器的硬件环境进行全面检查,确保满足CDH 6.3.2的最低要求。更多学习教程www.fgedu.net.cn
# free -h
total used free shared buff/cache available
Mem: 64G 2.2G 60G 9.1M 1.8G 61G
Swap: 32G 0B 32G
# 检查磁盘空间
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 9.1M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda1 50G 18G 33G 35% /
/dev/sdb1 500G 25G 475G 5% /data
# 检查CPU核心数
# nproc
32
# 检查系统架构
# uname -m
x86_64
2. 操作系统检查
CDH 6.3.2支持RHEL 7.3+、RHEL 8.0+等操作系统。本文以RHEL 9为例。学习交流加群风哥微信: itpux-com
# cat /etc/redhat-release
Red Hat Enterprise Linux release 9.0 (Plow)
# 检查内核版本
# uname -r
5.14.0-70.22.1.el9_0.x86_64
# 检查SELinux状态
# getenforce
Disabled
# 关闭SELinux(如未关闭)
# vi /etc/selinux/config
SELINUX=disabled
# 检查防火墙状态
# systemctl status firewalld
# 关闭防火墙(生产环境建议开放特定端口而非完全关闭)
# systemctl stop firewalld
# systemctl disable firewalld
3. 安装准备
在安装CDH之前,需要进行一系列准备工作,包括安装依赖包、配置网络和创建用户等。
# dnf install -y java-1.8.0-openjdk-devel wget curl ntp telnet net-tools python3 python3-pip
# 配置NTP
# systemctl start ntpd
# systemctl enable ntpd
# 创建用户
# useradd -m cloudera-scm
# passwd cloudera-scm
Changing password for user cloudera-scm.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
# 创建目录
# mkdir -p /opt/cloudera
# mkdir -p /data/cdh
# chown -R cloudera-scm:cloudera-scm /opt/cloudera
# chown -R cloudera-scm:cloudera-scm /data/cdh
4. CDH软件安装
现在开始安装CDH 6.3.2软件,按照以下步骤进行。
# wget https://archive.cloudera.com/cm6/6.3.1/cloudera-manager-installer.bin
# 给安装包添加执行权限
# chmod +x cloudera-manager-installer.bin
# 执行安装
# ./cloudera-manager-installer.bin
# 安装过程交互
1. 接受许可协议
2. 选择安装类型(默认完整安装)
3. 输入Cloudera Manager管理员密码
4. 等待安装完成
5. 数据库配置
配置CDH所需的数据库,包括MySQL或PostgreSQL。
# dnf install -y mysql-server
# systemctl start mysqld
# systemctl enable mysqld
# 配置MySQL
# mysql_secure_installation
# 创建CDH所需数据库
# mysql -u root -p
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO ‘scm’@’%’ IDENTIFIED BY ‘scm_password’;
GRANT ALL ON amon.* TO ‘amon’@’%’ IDENTIFIED BY ‘amon_password’;
GRANT ALL ON rman.* TO ‘rman’@’%’ IDENTIFIED BY ‘rman_password’;
GRANT ALL ON hue.* TO ‘hue’@’%’ IDENTIFIED BY ‘hue_password’;
GRANT ALL ON metastore.* TO ‘hive’@’%’ IDENTIFIED BY ‘hive_password’;
GRANT ALL ON sentry.* TO ‘sentry’@’%’ IDENTIFIED BY ‘sentry_password’;
GRANT ALL ON nav.* TO ‘nav’@’%’ IDENTIFIED BY ‘nav_password’;
GRANT ALL ON navms.* TO ‘navms’@’%’ IDENTIFIED BY ‘navms_password’;
FLUSH PRIVILEGES;
EXIT;
# 初始化SCM数据库
# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm_password
6. 测试验证
启动Cloudera Manager并验证CDH集群功能正常。学习交流加群风哥QQ113257174
# systemctl start cloudera-scm-server
# systemctl enable cloudera-scm-server
# 检查服务状态
# systemctl status cloudera-scm-server
# 访问Cloudera Manager控制台
# 打开浏览器,访问 http://localhost:7180
# 用户名: admin
# 密码: 安装时设置的密码
# 按照向导配置CDH集群
1. 选择要管理的主机
2. 选择安装的服务
3. 配置服务参数
4. 启动服务
7. 备份配置
配置CDH集群备份策略,确保数据安全。
# mkdir -p /backup/cdh
# cp -r /etc/cloudera-scm-server /backup/cdh/
# cp -r /var/lib/cloudera-scm-server /backup/cdh/
# 备份HDFS数据
# su – hdfs -c “hdfs dfs -mkdir -p /user/backup”
# su – hdfs -c “hdfs dfs -put /etc/hadoop/conf /user/backup/hadoop-conf”
# 配置自动备份
# 创建备份脚本
# vi /root/backup_cdh.sh
#!/bin/bash
DATE=$(date +%Y%m%d)
BACKUP_DIR=”/backup/cdh/$DATE”
# 创建备份目录
mkdir -p $BACKUP_DIR
# 备份Cloudera Manager配置
cp -r /etc/cloudera-scm-server $BACKUP_DIR/
cp -r /var/lib/cloudera-scm-server $BACKUP_DIR/
# 备份HDFS配置
su – hdfs -c “hdfs dfs -mkdir -p /user/backup/$DATE”
su – hdfs -c “hdfs dfs -put /etc/hadoop/conf /user/backup/$DATE/hadoop-conf”
# 给脚本添加执行权限
# chmod +x /root/backup_cdh.sh
# 添加到crontab
# crontab -e
# 添加以下内容(每天凌晨2点执行备份)
0 2 * * * /root/backup_cdh.sh
8. 升级迁移
CDH集群的升级和迁移过程。
# 1. 备份集群配置
# /root/backup_cdh.sh
# 2. 停止Cloudera Manager服务
# systemctl stop cloudera-scm-server
# 3. 下载新版本安装包
# wget https://archive.cloudera.com/cm6/6.3.2/cloudera-manager-installer.bin
# 4. 执行升级
# chmod +x cloudera-manager-installer.bin
# ./cloudera-manager-installer.bin
# 5. 启动Cloudera Manager服务
# systemctl start cloudera-scm-server
# 6. 通过Cloudera Manager升级集群
# 打开浏览器,访问 http://localhost:7180
# 按照向导升级集群
# 从旧CDH集群迁移到新CDH集群
# 1. 备份旧集群数据
# su – hdfs -c “hdfs dfs -copyToLocal /user/hive/warehouse /backup/hive-warehouse”
# 2. 将备份数据复制到新集群
# scp -r /backup/hive-warehouse root@new-cluster:/backup/
# 3. 在新集群上恢复数据
# su – hdfs -c “hdfs dfs -copyFromLocal /backup/hive-warehouse /user/hive/warehouse”
# 4. 验证迁移结果
# su – hive -c “hive -e ‘SHOW TABLES;'”
9. 总结
本文详细介绍了CDH 6.3.2 for RHEL 9的安装、配置、升级和迁移过程。通过按照本文的步骤操作,可以成功部署CDH集群并确保其稳定运行。from:www.itpux.com
– 定期备份集群配置和数据,建议每天执行一次全备份
– 监控集群性能,定期检查集群日志和错误信息
– 定期更新CDH版本,确保系统安全性和性能
– 合理规划集群存储,避免空间不足
– 配置合适的参数,优化集群性能
– 对于生产环境,建议部署高可用集群,提高可用性和可靠性
本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html
