1. 首页 > Hadoop教程 > 正文

大数据教程FG073-HBase集群安装部署

本文档风哥主要介绍HBase集群安装部署,包括安装环境要求、安装规划、环境准备、HBase安装配置等内容,风哥教程参考HBase官方文档Installation、Configuration等内容,适合大数据开发运维人员在学习和测试中使用,如果要应用于生产环境则需要自行确认。更多视频教程www.fgedu.net.cn

Part01-基础概念与理论知识

1.1 安装环境要求

HBase安装环境要求包括硬件要求、操作系统要求、软件依赖等。学习交流加群风哥微信: itpux-com

HBase安装环境要求:

  • 操作系统:Linux(RHEL/CentOS 7.x/8.x、Ubuntu等)
  • JDK:JDK 8或JDK 11
  • Hadoop:Hadoop 3.x
  • ZooKeeper:ZooKeeper 3.6+
  • 硬件:内存64GB以上、磁盘多块、网络千兆以上

1.2 安装规划

HBase安装规划详解:

# 安装规划

1. 节点规划(5节点集群)
节点 IP地址 角色
fgedu-node1 192.168.1.60 NameNode + HMaster + ZooKeeper
fgedu-node2 192.168.1.61 NameNode(备) + HMaster(备) + ZooKeeper
fgedu-node3 192.168.1.62 DataNode + RegionServer + ZooKeeper
fgedu-node4 192.168.1.63 DataNode + RegionServer
fgedu-node5 192.168.1.64 DataNode + RegionServer

2. 目录规划
安装目录 /bigdata/app/hbase
数据目录 /bigdata/hbase/data
日志目录 /var/log/hbase
PID目录 /var/run/hbase
临时目录 /tmp/hbase

3. 用户规划
用户名 hbase
用户组 hadoop

4. 端口规划
服务 端口
HMaster 16000
HMaster Web 16010
RegionServer 16020
RegionServer Web 16030
ZooKeeper 2181

1.3 安装模式选择

HBase安装模式详解:

# 安装模式

1. 单机模式
– 单个JVM进程
– 不依赖HDFS
– 仅用于测试

2. 伪分布式模式
– 单节点多进程
– 依赖HDFS
– 用于开发测试

3. 完全分布式模式
– 多节点集群
– 依赖HDFS和ZooKeeper
– 生产环境推荐

# 模式对比
模式 进程数 HDFS ZooKeeper 适用场景
单机模式 1 否 否 快速测试
伪分布式 多个 是 否 开发测试
完全分布式 多个 是 是 生产环境

# 本文档采用完全分布式模式

风哥提示:生产环境必须使用完全分布式模式,确保高可用和可扩展性。建议使用独立的ZooKeeper集群。

Part02-生产环境规划与建议

2.1 环境规划

环境规划建议:

# 操作系统配置

1. 关闭防火墙
$ systemctl stop firewalld
$ systemctl disable firewalld

2. 关闭SELinux
$ setenforce 0
$ sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/’ /etc/selinux/config

3. 配置主机名
$ hostnamectl set-hostname fgedu-node1

4. 配置hosts
$ cat >> /etc/hosts << 'EOF' 192.168.1.60 fgedu-node1 192.168.1.61 fgedu-node2 192.168.1.62 fgedu-node3 192.168.1.63 fgedu-node4 192.168.1.64 fgedu-node5 EOF 5. 配置时间同步 $ yum install -y chrony $ systemctl start chronyd $ systemctl enable chronyd 6. 配置SSH免密登录 $ ssh-keygen -t rsa $ ssh-copy-id fgedu-node1 $ ssh-copy-id fgedu-node2 $ ssh-copy-id fgedu-node3 $ ssh-copy-id fgedu-node4 $ ssh-copy-id fgedu-node5

2.2 依赖组件规划

依赖组件规划建议:

# JDK安装
$ tar -xzf jdk-17_linux-x64_bin.tar.gz -C /usr/lib/jvm/
$ ln -s /usr/lib/jvm/jdk-17 /usr/lib/jvm/java-17-openjdk

$ cat >> /etc/profile << 'EOF' export JAVA_HOME=/usr/lib/jvm/java-17-openjdk export PATH=$JAVA_HOME/bin:$PATH EOF $ source /etc/profile $ java -version java version "17.0.8" 2023-07-18 LTS Java(TM) SE Runtime Environment (build 17.0.8+9-LTS-211) Java HotSpot(TM) 64-Bit Server VM (build 17.0.8+9-LTS-211, mixed mode, sharing) # Hadoop安装(已安装) $ hdfs version Hadoop 3.3.6 Source code repository https://github.com/apache/hadoop.git -r 1be78238728da9266a4f88195058f08fd012bf9c Compiled by ubuntu on 2023-06-18T08:22Z Compiled with protoc 3.7.1 From source with checksum 565094e450d96c1c1935736c7b285004 This command was run using /bigdata/app/hadoop/share/hadoop/common/hadoop-common-3.3.6.jar # ZooKeeper安装(已安装) $ zkServer.sh status ZooKeeper JMX enabled by default Using config: /bigdata/app/zookeeper/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower # 创建HBase用户 $ useradd -g hadoop hbase $ echo "hbase:fgedu123" | chpasswd

2.3 配置规划

配置规划建议:

# JVM参数规划

# Master JVM参数
-Xmx4g -Xms4g -XX:+UseG1GC -XX:MaxGCPauseMillis=200

# RegionServer JVM参数
-Xmx32g -Xms32g -XX:+UseG1GC -XX:MaxGCPauseMillis=200
-XX:InitiatingHeapOccupancyPercent=45
-XX:G1HeapRegionSize=16m

# 内存规划
总内存:128GB
– 操作系统:4GB
– HDFS DataNode:4GB
– HBase RegionServer Heap:32GB
– BlockCache:12GB(40%)
– MemStore:12GB(40%)
– 其他:8GB(20%)
– 系统缓存:88GB

# 文件系统规划
# XFS文件系统(推荐)
$ mkfs.xfs -f /dev/sdb1
$ mount -o noatime,nodiratime /dev/sdb1 /data1

# 挂载选项
noatime,nodiratime,noexec

生产环境建议:生产环境建议使用JDK 11或JDK 17,配置G1垃圾回收器。RegionServer内存建议32GB以上。学习交流加群风哥QQ113257174

Part03-生产环境项目实施方案

3.1 环境准备

3.1.1 系统参数配置

# 配置系统参数
$ cat >> /etc/sysctl.conf << 'EOF' # 网络参数 net.core.somaxconn = 32768 net.core.netdev_max_backlog = 32768 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 30 # 文件描述符 fs.file-max = 6815744 # 内存参数 vm.swappiness = 10 vm.dirty_ratio = 80 vm.dirty_background_ratio = 5 vm.max_map_count = 262144 EOF $ sysctl -p # 配置文件描述符限制 $ cat >> /etc/security/limits.conf << 'EOF' hbase soft nofile 65536 hbase hard nofile 65536 hbase soft nproc 65536 hbase hard nproc 65536 EOF # 配置透明大页 $ echo never > /sys/kernel/mm/transparent_hugepage/enabled
$ echo never > /sys/kernel/mm/transparent_hugepage/defrag

$ cat >> /etc/rc.local << 'EOF' echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
EOF

# 创建目录
$ mkdir -p /bigdata/app/hbase
$ mkdir -p /bigdata/hbase/data
$ mkdir -p /var/log/hbase
$ mkdir -p /var/run/hbase

$ chown -R hbase:hadoop /bigdata/app/hbase
$ chown -R hbase:hadoop /bigdata/hbase
$ chown -R hbase:hadoop /var/log/hbase
$ chown -R hbase:hadoop /var/run/hbase

3.1.2 HDFS目录准备

# 创建HDFS目录
$ hdfs dfs -mkdir -p /hbase
$ hdfs dfs -chown hbase:hadoop /hbase
$ hdfs dfs -chmod 755 /hbase

# 验证HDFS
$ hdfs dfs -ls /

Found 3 items
drwxr-xr-x – hbase hadoop 0 2026-04-08 13:00 /hbase
drwxr-xr-x – hdfs hadoop 0 2026-04-08 12:00 /tmp
drwxr-xr-x – hdfs hadoop 0 2026-04-08 12:00 /user

# 检查HDFS健康状态
$ hdfs fsck /

Connecting to namenode via http://fgedu-node1:9870/fsck?ugi=hbase&path=%2F
FSCK started by hbase (auth:SIMPLE) from /192.168.1.60 for path / at Mon Apr 08 13:00:00 CST 2026
……..Status: HEALTHY
Total size: 0 B
Total dirs: 3
Total files: 0
Total symlinks: 0
Total blocks (validated): 0
Minimally replicated blocks: 0
Over-replicated blocks: 0
Under-replicated blocks: 0
Mis-replicated blocks: 0
Default replication factor: 3
Average block replication: 0.0
Corrupt blocks: 0
Missing replicas: 0
Number of data-nodes: 5
Number of racks: 1
FSCK ended at Mon Apr 08 13:00:00 CST 2026 in 1 milliseconds

The filesystem under path ‘/’ is HEALTHY

3.2 HBase安装

# 下载HBase
$ cd /bigdata/app
$ wget https://downloads.apache.org/hbase/2.5.5/hbase-2.5.5-bin.tar.gz

# 解压安装
$ tar -xzf hbase-2.5.5-bin.tar.gz
$ ln -s hbase-2.5.5 hbase
$ chown -R hbase:hadoop hbase-2.5.5 hbase

# 配置环境变量
$ cat >> /etc/profile << 'EOF' export HBASE_HOME=/bigdata/app/hbase export PATH=$HBASE_HOME/bin:$PATH EOF $ source /etc/profile # 验证安装 $ hbase version HBase 2.5.5 Source code repository git://fgedu-node1/home/busbey/projects/hbase.git revision=1a5a961c8d1a8c9c7b4c8d1a8c9c7b4c8d1a8c9c Compiled by busbey on Mon Apr 8 13:00:00 CST 2026 From source with checksum 1a5a961c8d1a8c9c7b4c8d1a8c9c7b4c # 分发到其他节点 $ scp -r /bigdata/app/hbase-2.5.5 fgedu-node2:/bigdata/app/ $ scp -r /bigdata/app/hbase-2.5.5 fgedu-node3:/bigdata/app/ $ scp -r /bigdata/app/hbase-2.5.5 fgedu-node4:/bigdata/app/ $ scp -r /bigdata/app/hbase-2.5.5 fgedu-node5:/bigdata/app/ $ ssh fgedu-node2 "ln -s /bigdata/app/hbase-2.5.5 /bigdata/app/hbase" $ ssh fgedu-node3 "ln -s /bigdata/app/hbase-2.5.5 /bigdata/app/hbase" $ ssh fgedu-node4 "ln -s /bigdata/app/hbase-2.5.5 /bigdata/app/hbase" $ ssh fgedu-node5 "ln -s /bigdata/app/hbase-2.5.5 /bigdata/app/hbase"

3.3 HBase配置

3.3.1 配置hbase-env.sh

# 配置hbase-env.sh
$ cat > /bigdata/app/hbase/conf/hbase-env.sh << 'EOF' #!/bin/bash # hbase-env.sh # from:www.itpux.com.qq113257174.wx:itpux-com # web: http://www.fgedu.net.cn # Java环境 export JAVA_HOME=/usr/lib/jvm/java-17-openjdk export HBASE_HOME=/bigdata/app/hbase # Hadoop配置目录 export HBASE_CLASSPATH=/bigdata/app/hadoop/etc/hadoop # Master JVM参数 export HBASE_MASTER_OPTS="-Xmx4g -Xms4g -XX:+UseG1GC -XX:MaxGCPauseMillis=200" # RegionServer JVM参数 export HBASE_REGIONSERVER_OPTS="-Xmx32g -Xms32g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:G1HeapRegionSize=16m" # 日志目录 export HBASE_LOG_DIR=/var/log/hbase export HBASE_PID_DIR=/var/run/hbase # 不使用内置ZooKeeper export HBASE_MANAGES_ZK=false # 远程JMX export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101" export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102" EOF # 分发配置文件 $ scp /bigdata/app/hbase/conf/hbase-env.sh fgedu-node2:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/hbase-env.sh fgedu-node3:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/hbase-env.sh fgedu-node4:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/hbase-env.sh fgedu-node5:/bigdata/app/hbase/conf/

3.3.2 配置hbase-site.xml

# 配置hbase-site.xml
$ cat > /bigdata/app/hbase/conf/hbase-site.xml << 'EOF'


hbase.rootdir
hdfs://192.168.1.60:9000/hbase
hbase.cluster.distributed
true
hbase.tmp.dir
/tmp/hbase

hbase.zookeeper.quorum
fgedu-node1,fgedu-node2,fgedu-node3
hbase.zookeeper.property.clientPort
2181
hbase.zookeeper.property.dataDir
/bigdata/zookeeper/data

hbase.master.port
16000
hbase.master.info.port
16010
hbase.master.wait.on.regionservers.mintostart
3

hbase.regionserver.port
16020
hbase.regionserver.info.port
16030
hbase.regionserver.handler.count
200
hbase.regionserver.global.memstore.size
0.4
hfile.block.cache.size
0.4

hbase.hregion.memstore.flush.size
134217728
hbase.hregion.max.filesize
10737418240

hbase.wal.provider
asyncfs

hbase.ipc.server.tcpnodelay
true
hbase.ipc.client.tcpnodelay
true

EOF

# 分发配置文件
$ scp /bigdata/app/hbase/conf/hbase-site.xml fgedu-node2:/bigdata/app/hbase/conf/
$ scp /bigdata/app/hbase/conf/hbase-site.xml fgedu-node3:/bigdata/app/hbase/conf/
$ scp /bigdata/app/hbase/conf/hbase-site.xml fgedu-node4:/bigdata/app/hbase/conf/
$ scp /bigdata/app/hbase/conf/hbase-site.xml fgedu-node5:/bigdata/app/hbase/conf/

3.3.3 配置regionservers和backup-masters

# 配置regionservers
$ cat > /bigdata/app/hbase/conf/regionservers << 'EOF' fgedu-node3 fgedu-node4 fgedu-node5 EOF # 配置backup-masters $ cat > /bigdata/app/hbase/conf/backup-masters << 'EOF' fgedu-node2 EOF # 分发配置文件 $ scp /bigdata/app/hbase/conf/regionservers fgedu-node2:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/regionservers fgedu-node3:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/regionservers fgedu-node4:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/regionservers fgedu-node5:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/backup-masters fgedu-node2:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/backup-masters fgedu-node3:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/backup-masters fgedu-node4:/bigdata/app/hbase/conf/ $ scp /bigdata/app/hbase/conf/backup-masters fgedu-node5:/bigdata/app/hbase/conf/
风哥提示:HBase配置需要根据实际环境进行调整,特别是内存配置和RegionServer配置。生产环境建议使用独立的ZooKeeper集群。更多学习教程公众号风哥教程itpux_com

Part04-生产案例与实战讲解

4.1 集群启动验证

# 启动HBase集群
$ /bigdata/app/hbase/bin/start-hbase.sh

fgedu-node3: starting regionserver, logging to /var/log/hbase/hbase-hbase-regionserver-fgedu-node3.out
fgedu-node4: starting regionserver, logging to /var/log/hbase/hbase-hbase-regionserver-fgedu-node4.out
fgedu-node5: starting regionserver, logging to /var/log/hbase/hbase-hbase-regionserver-fgedu-node5.out
fgedu-node2: starting master, logging to /var/log/hbase/hbase-hbase-master-fgedu-node2.out
fgedu-node1: starting master, logging to /var/log/hbase/hbase-hbase-master-fgedu-node1.out

# 查看进程
$ jps

12345 HMaster
12456 Jps

# 在RegionServer节点
$ ssh fgedu-node3 jps

12345 HRegionServer
12456 Jps

# 查看HBase状态
$ echo “status” | hbase shell

HBase Shell
Use “help” to get list of supported commands.
Use “exit” to quit this interactive shell.
Version 2.5.5, rUnknown, Mon Apr 8 13:00:00 CST 2026

active master: fgedu-node1:16000
1 backup masters
fgedu-node2:16000
3 live servers
fgedu-node3:16020
requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=512, maxHeapMB=32768
fgedu-node4:16020
requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=512, maxHeapMB=32768
fgedu-node5:16020
requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=512, maxHeapMB=32768
0 dead servers
Master is fgedu-node1:16000
Region load balancing is enabled

4.2 集群测试

# 创建测试表
$ echo ”
create ‘fgedu_test’, ‘cf1’, ‘cf2’
put ‘fgedu_test’, ‘row1’, ‘cf1:col1’, ‘value1’
put ‘fgedu_test’, ‘row1’, ‘cf1:col2’, ‘value2’
put ‘fgedu_test’, ‘row2’, ‘cf1:col1’, ‘value3’
put ‘fgedu_test’, ‘row2’, ‘cf2:col1’, ‘value4’
scan ‘fgedu_test’
” | hbase shell

Created table fgedu_test
0 row(s) in 2.3450 seconds

0 row(s) in 0.1230 seconds
0 row(s) in 0.0120 seconds
0 row(s) in 0.0110 seconds
0 row(s) in 0.0100 seconds

ROW COLUMN+CELL
row1 column=cf1:col1, timestamp=1680940800000, value=value1
row1 column=cf1:col2, timestamp=1680940800001, value=value2
row2 column=cf1:col1, timestamp=1680940800002, value=value3
row2 column=cf2:col1, timestamp=1680940800003, value=value4
2 row(s) in 0.0450 seconds

# 查看表信息
$ echo “describe ‘fgedu_test'” | hbase shell

Table fgedu_test is ENABLED
fgedu_test
COLUMN FAMILIES DESCRIPTION
{NAME => ‘cf1’, BLOOMFILTER => ‘ROW’, VERSIONS => ‘1’, IN_MEMORY => ‘false’, KEEP_DELETED_CELLS => ‘FALSE’, DATA_BLOCK_ENCODING => ‘NONE’, COMPRESSION => ‘NONE’, TTL => ‘FOREVER’}
{NAME => ‘cf2’, BLOOMFILTER => ‘ROW’, VERSIONS => ‘1’, IN_MEMORY => ‘false’, KEEP_DELETED_CELLS => ‘FALSE’, DATA_BLOCK_ENCODING => ‘NONE’, COMPRESSION => ‘NONE’, TTL => ‘FOREVER’}

# 禁用和删除表
$ echo ”
disable ‘fgedu_test’
drop ‘fgedu_test’
” | hbase shell

0 row(s) in 1.2340 seconds
Table fgedu_test is DISABLED
0 row(s) in 0.5670 seconds

4.3 常见问题处理

4.3.1 HMaster启动失败

# 问题现象:HMaster无法启动

# 排查步骤
# 1. 查看日志
$ tail -100 /var/log/hbase/hbase-hbase-master-fgedu-node1.log

# 2. 检查ZooKeeper连接
$ zkCli.sh -server fgedu-node1:2181
ls /hbase

# 3. 检查HDFS目录
$ hdfs dfs -ls /hbase

# 解决方案
# 1. 检查ZooKeeper状态
$ zkServer.sh status

# 2. 检查HDFS状态
$ hdfs dfsadmin -report

# 3. 清理并重启
$ rm -rf /tmp/hbase/*
$ hdfs dfs -rm -r /hbase
$ hdfs dfs -mkdir -p /hbase
$ hdfs dfs -chown hbase:hadoop /hbase
$ /bigdata/app/hbase/bin/start-hbase.sh

4.3.2 RegionServer无法连接

# 问题现象:RegionServer无法连接到Master

# 排查步骤
# 1. 检查网络
$ ping fgedu-node1

# 2. 检查端口
$ telnet fgedu-node1 16000

# 3. 检查防火墙
$ firewall-cmd –list-ports

# 解决方案
# 1. 开放端口
$ firewall-cmd –add-port=16000/tcp –permanent
$ firewall-cmd –add-port=16010/tcp –permanent
$ firewall-cmd –add-port=16020/tcp –permanent
$ firewall-cmd –add-port=16030/tcp –permanent
$ firewall-cmd –reload

# 2. 检查hosts配置
$ cat /etc/hosts

# 3. 重启RegionServer
$ /bigdata/app/hbase/bin/hbase-daemon.sh stop regionserver
$ /bigdata/app/hbase/bin/hbase-daemon.sh start regionserver

Part05-风哥经验总结与分享

5.1 安装最佳实践

HBase安装最佳实践建议:

# 安装最佳实践
1. 使用独立的ZooKeeper集群
2. 配置Master高可用
3. 合理配置JVM参数
4. 确保时间同步
5. 配置系统参数优化
6. 使用SSD磁盘

5.2 安装建议

安装建议:

HBase安装建议:

  • 安装前确保HDFS和ZooKeeper正常运行
  • 配置文件分发到所有节点
  • 启动后验证集群状态
  • 创建测试表验证功能

5.3 工具推荐

安装管理工具:

  • HBase Shell:命令行管理
  • HBase Web UI:Web监控
  • Ambari/Cloudera Manager:集群管理
  • Ansible:自动化部署
风哥提示:HBase安装部署需要仔细检查每个步骤,确保依赖组件正常运行。建议先在测试环境验证,再部署到生产环境。from bigdata视频:www.itpux.com

本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html

联系我们

在线咨询:点击这里给我发消息

微信号:itpux-com

工作日:9:30-18:30,节假日休息