本文档风哥主要介绍HBase集群安装部署,包括安装环境要求、安装规划、环境准备、HBase安装配置等内容,风哥教程参考HBase官方文档Installation、Configuration等内容,适合大数据开发运维人员在学习和测试中使用,如果要应用于生产环境则需要自行确认。更多视频教程www.fgedu.net.cn
Part01-基础概念与理论知识
1.1 安装环境要求
HBase安装环境要求包括硬件要求、操作系统要求、软件依赖等。学习交流加群风哥微信: itpux-com
- 操作系统:Linux(RHEL/CentOS 7.x/8.x、Ubuntu等)
- JDK:JDK 8或JDK 11
- Hadoop:Hadoop 3.x
- ZooKeeper:ZooKeeper 3.6+
- 硬件:内存64GB以上、磁盘多块、网络千兆以上
1.2 安装规划
HBase安装规划详解:
1. 节点规划(5节点集群)
节点 IP地址 角色
fgedu-node1 192.168.1.60 NameNode + HMaster + ZooKeeper
fgedu-node2 192.168.1.61 NameNode(备) + HMaster(备) + ZooKeeper
fgedu-node3 192.168.1.62 DataNode + RegionServer + ZooKeeper
fgedu-node4 192.168.1.63 DataNode + RegionServer
fgedu-node5 192.168.1.64 DataNode + RegionServer
2. 目录规划
安装目录 /bigdata/app/hbase
数据目录 /bigdata/hbase/data
日志目录 /var/log/hbase
PID目录 /var/run/hbase
临时目录 /tmp/hbase
3. 用户规划
用户名 hbase
用户组 hadoop
4. 端口规划
服务 端口
HMaster 16000
HMaster Web 16010
RegionServer 16020
RegionServer Web 16030
ZooKeeper 2181
1.3 安装模式选择
HBase安装模式详解:
1. 单机模式
– 单个JVM进程
– 不依赖HDFS
– 仅用于测试
2. 伪分布式模式
– 单节点多进程
– 依赖HDFS
– 用于开发测试
3. 完全分布式模式
– 多节点集群
– 依赖HDFS和ZooKeeper
– 生产环境推荐
# 模式对比
模式 进程数 HDFS ZooKeeper 适用场景
单机模式 1 否 否 快速测试
伪分布式 多个 是 否 开发测试
完全分布式 多个 是 是 生产环境
# 本文档采用完全分布式模式
Part02-生产环境规划与建议
2.1 环境规划
环境规划建议:
1. 关闭防火墙
$ systemctl stop firewalld
$ systemctl disable firewalld
2. 关闭SELinux
$ setenforce 0
$ sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/’ /etc/selinux/config
3. 配置主机名
$ hostnamectl set-hostname fgedu-node1
4. 配置hosts
$ cat >> /etc/hosts << 'EOF'
192.168.1.60 fgedu-node1
192.168.1.61 fgedu-node2
192.168.1.62 fgedu-node3
192.168.1.63 fgedu-node4
192.168.1.64 fgedu-node5
EOF
5. 配置时间同步
$ yum install -y chrony
$ systemctl start chronyd
$ systemctl enable chronyd
6. 配置SSH免密登录
$ ssh-keygen -t rsa
$ ssh-copy-id fgedu-node1
$ ssh-copy-id fgedu-node2
$ ssh-copy-id fgedu-node3
$ ssh-copy-id fgedu-node4
$ ssh-copy-id fgedu-node5
2.2 依赖组件规划
依赖组件规划建议:
$ tar -xzf jdk-17_linux-x64_bin.tar.gz -C /usr/lib/jvm/
$ ln -s /usr/lib/jvm/jdk-17 /usr/lib/jvm/java-17-openjdk
$ cat >> /etc/profile << 'EOF' export JAVA_HOME=/usr/lib/jvm/java-17-openjdk export PATH=$JAVA_HOME/bin:$PATH EOF $ source /etc/profile $ java -version java version "17.0.8" 2023-07-18 LTS Java(TM) SE Runtime Environment (build 17.0.8+9-LTS-211) Java HotSpot(TM) 64-Bit Server VM (build 17.0.8+9-LTS-211, mixed mode, sharing) # Hadoop安装(已安装) $ hdfs version Hadoop 3.3.6 Source code repository https://github.com/apache/hadoop.git -r 1be78238728da9266a4f88195058f08fd012bf9c Compiled by ubuntu on 2023-06-18T08:22Z Compiled with protoc 3.7.1 From source with checksum 565094e450d96c1c1935736c7b285004 This command was run using /bigdata/app/hadoop/share/hadoop/common/hadoop-common-3.3.6.jar # ZooKeeper安装(已安装) $ zkServer.sh status ZooKeeper JMX enabled by default Using config: /bigdata/app/zookeeper/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower # 创建HBase用户 $ useradd -g hadoop hbase $ echo "hbase:fgedu123" | chpasswd
2.3 配置规划
配置规划建议:
# Master JVM参数
-Xmx4g -Xms4g -XX:+UseG1GC -XX:MaxGCPauseMillis=200
# RegionServer JVM参数
-Xmx32g -Xms32g -XX:+UseG1GC -XX:MaxGCPauseMillis=200
-XX:InitiatingHeapOccupancyPercent=45
-XX:G1HeapRegionSize=16m
# 内存规划
总内存:128GB
– 操作系统:4GB
– HDFS DataNode:4GB
– HBase RegionServer Heap:32GB
– BlockCache:12GB(40%)
– MemStore:12GB(40%)
– 其他:8GB(20%)
– 系统缓存:88GB
# 文件系统规划
# XFS文件系统(推荐)
$ mkfs.xfs -f /dev/sdb1
$ mount -o noatime,nodiratime /dev/sdb1 /data1
# 挂载选项
noatime,nodiratime,noexec
Part03-生产环境项目实施方案
3.1 环境准备
3.1.1 系统参数配置
$ cat >> /etc/sysctl.conf << 'EOF' # 网络参数 net.core.somaxconn = 32768 net.core.netdev_max_backlog = 32768 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 30 # 文件描述符 fs.file-max = 6815744 # 内存参数 vm.swappiness = 10 vm.dirty_ratio = 80 vm.dirty_background_ratio = 5 vm.max_map_count = 262144 EOF $ sysctl -p # 配置文件描述符限制 $ cat >> /etc/security/limits.conf << 'EOF' hbase soft nofile 65536 hbase hard nofile 65536 hbase soft nproc 65536 hbase hard nproc 65536 EOF # 配置透明大页 $ echo never > /sys/kernel/mm/transparent_hugepage/enabled
$ echo never > /sys/kernel/mm/transparent_hugepage/defrag
$ cat >> /etc/rc.local << 'EOF'
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
EOF
# 创建目录
$ mkdir -p /bigdata/app/hbase
$ mkdir -p /bigdata/hbase/data
$ mkdir -p /var/log/hbase
$ mkdir -p /var/run/hbase
$ chown -R hbase:hadoop /bigdata/app/hbase
$ chown -R hbase:hadoop /bigdata/hbase
$ chown -R hbase:hadoop /var/log/hbase
$ chown -R hbase:hadoop /var/run/hbase
3.1.2 HDFS目录准备
$ hdfs dfs -mkdir -p /hbase
$ hdfs dfs -chown hbase:hadoop /hbase
$ hdfs dfs -chmod 755 /hbase
# 验证HDFS
$ hdfs dfs -ls /
Found 3 items
drwxr-xr-x – hbase hadoop 0 2026-04-08 13:00 /hbase
drwxr-xr-x – hdfs hadoop 0 2026-04-08 12:00 /tmp
drwxr-xr-x – hdfs hadoop 0 2026-04-08 12:00 /user
# 检查HDFS健康状态
$ hdfs fsck /
Connecting to namenode via http://fgedu-node1:9870/fsck?ugi=hbase&path=%2F
FSCK started by hbase (auth:SIMPLE) from /192.168.1.60 for path / at Mon Apr 08 13:00:00 CST 2026
……..Status: HEALTHY
Total size: 0 B
Total dirs: 3
Total files: 0
Total symlinks: 0
Total blocks (validated): 0
Minimally replicated blocks: 0
Over-replicated blocks: 0
Under-replicated blocks: 0
Mis-replicated blocks: 0
Default replication factor: 3
Average block replication: 0.0
Corrupt blocks: 0
Missing replicas: 0
Number of data-nodes: 5
Number of racks: 1
FSCK ended at Mon Apr 08 13:00:00 CST 2026 in 1 milliseconds
The filesystem under path ‘/’ is HEALTHY
3.2 HBase安装
$ cd /bigdata/app
$ wget https://downloads.apache.org/hbase/2.5.5/hbase-2.5.5-bin.tar.gz
# 解压安装
$ tar -xzf hbase-2.5.5-bin.tar.gz
$ ln -s hbase-2.5.5 hbase
$ chown -R hbase:hadoop hbase-2.5.5 hbase
# 配置环境变量
$ cat >> /etc/profile << 'EOF'
export HBASE_HOME=/bigdata/app/hbase
export PATH=$HBASE_HOME/bin:$PATH
EOF
$ source /etc/profile
# 验证安装
$ hbase version
HBase 2.5.5
Source code repository git://fgedu-node1/home/busbey/projects/hbase.git revision=1a5a961c8d1a8c9c7b4c8d1a8c9c7b4c8d1a8c9c
Compiled by busbey on Mon Apr 8 13:00:00 CST 2026
From source with checksum 1a5a961c8d1a8c9c7b4c8d1a8c9c7b4c
# 分发到其他节点
$ scp -r /bigdata/app/hbase-2.5.5 fgedu-node2:/bigdata/app/
$ scp -r /bigdata/app/hbase-2.5.5 fgedu-node3:/bigdata/app/
$ scp -r /bigdata/app/hbase-2.5.5 fgedu-node4:/bigdata/app/
$ scp -r /bigdata/app/hbase-2.5.5 fgedu-node5:/bigdata/app/
$ ssh fgedu-node2 "ln -s /bigdata/app/hbase-2.5.5 /bigdata/app/hbase"
$ ssh fgedu-node3 "ln -s /bigdata/app/hbase-2.5.5 /bigdata/app/hbase"
$ ssh fgedu-node4 "ln -s /bigdata/app/hbase-2.5.5 /bigdata/app/hbase"
$ ssh fgedu-node5 "ln -s /bigdata/app/hbase-2.5.5 /bigdata/app/hbase"
3.3 HBase配置
3.3.1 配置hbase-env.sh
$ cat > /bigdata/app/hbase/conf/hbase-env.sh << 'EOF' #!/bin/bash # hbase-env.sh # from:www.itpux.com.qq113257174.wx:itpux-com # web: http://www.fgedu.net.cn # Java环境 export JAVA_HOME=/usr/lib/jvm/java-17-openjdk export HBASE_HOME=/bigdata/app/hbase # Hadoop配置目录 export HBASE_CLASSPATH=/bigdata/app/hadoop/etc/hadoop # Master JVM参数 export HBASE_MASTER_OPTS="-Xmx4g -Xms4g -XX:+UseG1GC -XX:MaxGCPauseMillis=200" # RegionServer JVM参数 export HBASE_REGIONSERVER_OPTS="-Xmx32g -Xms32g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:G1HeapRegionSize=16m" # 日志目录 export HBASE_LOG_DIR=/var/log/hbase export HBASE_PID_DIR=/var/run/hbase # 不使用内置ZooKeeper export HBASE_MANAGES_ZK=false # 远程JMX export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101" export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102" EOF # 分发配置文件 $ scp /bigdata/app/hbase/conf/hbase-env.sh fgedu-node2:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/hbase-env.sh fgedu-node3:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/hbase-env.sh fgedu-node4:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/hbase-env.sh fgedu-node5:/bigdata/app/hbase/conf/
3.3.2 配置hbase-site.xml
$ cat > /bigdata/app/hbase/conf/hbase-site.xml << 'EOF'
EOF
# 分发配置文件
$ scp /bigdata/app/hbase/conf/hbase-site.xml fgedu-node2:/bigdata/app/hbase/conf/
$ scp /bigdata/app/hbase/conf/hbase-site.xml fgedu-node3:/bigdata/app/hbase/conf/
$ scp /bigdata/app/hbase/conf/hbase-site.xml fgedu-node4:/bigdata/app/hbase/conf/
$ scp /bigdata/app/hbase/conf/hbase-site.xml fgedu-node5:/bigdata/app/hbase/conf/
3.3.3 配置regionservers和backup-masters
$ cat > /bigdata/app/hbase/conf/regionservers << 'EOF' fgedu-node3 fgedu-node4 fgedu-node5 EOF # 配置backup-masters $ cat > /bigdata/app/hbase/conf/backup-masters << 'EOF' fgedu-node2 EOF # 分发配置文件 $ scp /bigdata/app/hbase/conf/regionservers fgedu-node2:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/regionservers fgedu-node3:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/regionservers fgedu-node4:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/regionservers fgedu-node5:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/backup-masters fgedu-node2:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/backup-masters fgedu-node3:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/backup-masters fgedu-node4:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/backup-masters fgedu-node5:/bigdata/app/hbase/conf/
Part04-生产案例与实战讲解
4.1 集群启动验证
$ /bigdata/app/hbase/bin/start-hbase.sh
fgedu-node3: starting regionserver, logging to /var/log/hbase/hbase-hbase-regionserver-fgedu-node3.out
fgedu-node4: starting regionserver, logging to /var/log/hbase/hbase-hbase-regionserver-fgedu-node4.out
fgedu-node5: starting regionserver, logging to /var/log/hbase/hbase-hbase-regionserver-fgedu-node5.out
fgedu-node2: starting master, logging to /var/log/hbase/hbase-hbase-master-fgedu-node2.out
fgedu-node1: starting master, logging to /var/log/hbase/hbase-hbase-master-fgedu-node1.out
# 查看进程
$ jps
12345 HMaster
12456 Jps
# 在RegionServer节点
$ ssh fgedu-node3 jps
12345 HRegionServer
12456 Jps
# 查看HBase状态
$ echo “status” | hbase shell
HBase Shell
Use “help” to get list of supported commands.
Use “exit” to quit this interactive shell.
Version 2.5.5, rUnknown, Mon Apr 8 13:00:00 CST 2026
active master: fgedu-node1:16000
1 backup masters
fgedu-node2:16000
3 live servers
fgedu-node3:16020
requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=512, maxHeapMB=32768
fgedu-node4:16020
requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=512, maxHeapMB=32768
fgedu-node5:16020
requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=512, maxHeapMB=32768
0 dead servers
Master is fgedu-node1:16000
Region load balancing is enabled
4.2 集群测试
$ echo ”
create ‘fgedu_test’, ‘cf1’, ‘cf2’
put ‘fgedu_test’, ‘row1’, ‘cf1:col1’, ‘value1’
put ‘fgedu_test’, ‘row1’, ‘cf1:col2’, ‘value2’
put ‘fgedu_test’, ‘row2’, ‘cf1:col1’, ‘value3’
put ‘fgedu_test’, ‘row2’, ‘cf2:col1’, ‘value4’
scan ‘fgedu_test’
” | hbase shell
Created table fgedu_test
0 row(s) in 2.3450 seconds
0 row(s) in 0.1230 seconds
0 row(s) in 0.0120 seconds
0 row(s) in 0.0110 seconds
0 row(s) in 0.0100 seconds
ROW COLUMN+CELL
row1 column=cf1:col1, timestamp=1680940800000, value=value1
row1 column=cf1:col2, timestamp=1680940800001, value=value2
row2 column=cf1:col1, timestamp=1680940800002, value=value3
row2 column=cf2:col1, timestamp=1680940800003, value=value4
2 row(s) in 0.0450 seconds
# 查看表信息
$ echo “describe ‘fgedu_test'” | hbase shell
Table fgedu_test is ENABLED
fgedu_test
COLUMN FAMILIES DESCRIPTION
{NAME => ‘cf1’, BLOOMFILTER => ‘ROW’, VERSIONS => ‘1’, IN_MEMORY => ‘false’, KEEP_DELETED_CELLS => ‘FALSE’, DATA_BLOCK_ENCODING => ‘NONE’, COMPRESSION => ‘NONE’, TTL => ‘FOREVER’}
{NAME => ‘cf2’, BLOOMFILTER => ‘ROW’, VERSIONS => ‘1’, IN_MEMORY => ‘false’, KEEP_DELETED_CELLS => ‘FALSE’, DATA_BLOCK_ENCODING => ‘NONE’, COMPRESSION => ‘NONE’, TTL => ‘FOREVER’}
# 禁用和删除表
$ echo ”
disable ‘fgedu_test’
drop ‘fgedu_test’
” | hbase shell
0 row(s) in 1.2340 seconds
Table fgedu_test is DISABLED
0 row(s) in 0.5670 seconds
4.3 常见问题处理
4.3.1 HMaster启动失败
# 排查步骤
# 1. 查看日志
$ tail -100 /var/log/hbase/hbase-hbase-master-fgedu-node1.log
# 2. 检查ZooKeeper连接
$ zkCli.sh -server fgedu-node1:2181
ls /hbase
# 3. 检查HDFS目录
$ hdfs dfs -ls /hbase
# 解决方案
# 1. 检查ZooKeeper状态
$ zkServer.sh status
# 2. 检查HDFS状态
$ hdfs dfsadmin -report
# 3. 清理并重启
$ rm -rf /tmp/hbase/*
$ hdfs dfs -rm -r /hbase
$ hdfs dfs -mkdir -p /hbase
$ hdfs dfs -chown hbase:hadoop /hbase
$ /bigdata/app/hbase/bin/start-hbase.sh
4.3.2 RegionServer无法连接
# 排查步骤
# 1. 检查网络
$ ping fgedu-node1
# 2. 检查端口
$ telnet fgedu-node1 16000
# 3. 检查防火墙
$ firewall-cmd –list-ports
# 解决方案
# 1. 开放端口
$ firewall-cmd –add-port=16000/tcp –permanent
$ firewall-cmd –add-port=16010/tcp –permanent
$ firewall-cmd –add-port=16020/tcp –permanent
$ firewall-cmd –add-port=16030/tcp –permanent
$ firewall-cmd –reload
# 2. 检查hosts配置
$ cat /etc/hosts
# 3. 重启RegionServer
$ /bigdata/app/hbase/bin/hbase-daemon.sh stop regionserver
$ /bigdata/app/hbase/bin/hbase-daemon.sh start regionserver
Part05-风哥经验总结与分享
5.1 安装最佳实践
HBase安装最佳实践建议:
1. 使用独立的ZooKeeper集群
2. 配置Master高可用
3. 合理配置JVM参数
4. 确保时间同步
5. 配置系统参数优化
6. 使用SSD磁盘
5.2 安装建议
安装建议:
- 安装前确保HDFS和ZooKeeper正常运行
- 配置文件分发到所有节点
- 启动后验证集群状态
- 创建测试表验证功能
5.3 工具推荐
安装管理工具:
- HBase Shell:命令行管理
- HBase Web UI:Web监控
- Ambari/Cloudera Manager:集群管理
- Ansible:自动化部署
本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html
