1. 首页 > Hadoop教程 > 正文

大数据教程FG032-ZooKeeper分布式协调服务实战

内容简介:本文详细介绍ZooKeeper分布式协调服务实战,包括ZooKeeper架构原理、集群部署、节点操作、应用场景等核心内容。风哥教程参考ZooKeeper官方文档Getting Started、Programmer’s Guide等内容。

目录大纲

Part01-基础概念与理论知识
  1.1 ZooKeeper架构概述
  1.2 数据模型与节点
  1.3 一致性协议原理
Part02-生产环境规划与建议
  2.1 集群部署规划
  2.2 节点设计规划
  2.3 性能优化规划
Part03-生产环境项目实施方案
  3.1 ZooKeeper安装配置
  3.2 节点操作管理
  3.3 集群监控维护
  3.4 客户端开发
Part04-生产案例与实战讲解
  4.1 配置中心案例
  4.2 分布式锁案例
  4.3 服务注册发现案例
Part05-风哥经验总结与分享
  5.1 ZooKeeper最佳实践
  5.2 运维经验总结

Part01-基础概念与理论知识

1.1 ZooKeeper架构概述

ZooKeeper是分布式协调服务。更多视频教程www.fgedu.net.cn 提供配置维护、分布式同步、组服务等功能。

风哥提示:ZooKeeper是Hadoop生态系统的核心组件,为HBase、Kafka等提供协调服务。

1.2 数据模型与节点

ZooKeeper数据模型是树形结构,每个节点称为ZNode。学习交流加群风哥微信: itpux-com

ZNode类型:
– 持久节点:永久存在,直到被删除
– 临时节点:会话结束自动删除
– 持久顺序节点:带顺序号的持久节点
– 临时顺序节点:带顺序号的临时节点

1.3 一致性协议原理

ZooKeeper使用ZAB协议保证一致性。from bigdata视频:www.itpux.com

# 查看ZooKeeper版本
zkServer.sh version

# 查看ZooKeeper状态
zkServer.sh status

# ZooKeeper版本
Apache ZooKeeper, version 3.8.1

# ZooKeeper状态
Mode: leader
Zxid: 0x100000abc
# 当前节点为Leader

Part02-生产环境规划与建议

2.1 集群部署规划

ZooKeeper集群部署需要考虑节点数量和资源配置。更多学习教程公众号风哥教程itpux_com

集群部署建议:
– 节点数量:奇数个,至少3个
– 内存配置:建议4GB以上
– 磁盘配置:使用独立磁盘存储快照
– 网络配置:低延迟网络环境

2.2 节点设计规划

节点设计影响应用性能。学习交流加群风哥QQ113257174

# 连接ZooKeeper
zkCli.sh -server fgedu01:2181

# 查看根节点
ls /

# 查看节点详情
ls -s /fgedu

# 连接ZooKeeper
Connecting to fgedu01:2181
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::
WatchedEvent state:SyncConnected type:None path:null

# 根节点
[cluster, controller, controller_epoch, fgedu, zookeeper]

# 节点详情
[config, lock, registry]
czxid = 0x100000001
ctime = Thu Jan 18 00:00:00 CST 2024
mzxid = 0x100000001
mtime = Thu Jan 18 00:00:00 CST 2024
pZxid = 0x100000010
cversion = 3
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 3

2.3 性能优化规划

性能优化需要从多个角度考虑。风哥提示:合理的快照和事务日志配置是性能优化的关键。

性能优化要点:
– 分离快照和事务日志
– 调整tickTime
– 优化JVM参数
– 监控节点状态
– 定期清理旧快照

Part03-生产环境项目实施方案

3.1 ZooKeeper安装配置

3.1.1 安装ZooKeeper

# 下载ZooKeeper
wget https://downloads.apache.org/zookeeper/zookeeper-3.8.1/apache-zookeeper-3.8.1-bin.tar.gz
tar -xzf apache-zookeeper-3.8.1-bin.tar.gz -C /bigdata/app/
ln -s /bigdata/app/apache-zookeeper-3.8.1-bin /bigdata/app/zookeeper

# 配置环境变量
export ZOOKEEPER_HOME=/bigdata/app/zookeeper
export PATH=$PATH:$ZOOKEEPER_HOME/bin

# 配置zoo.cfg
cat > /bigdata/app/zookeeper/conf/zoo.cfg << 'EOF'
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/bigdata/zookeeper/data
dataLogDir=/bigdata/zookeeper/logs
clientPort=2181
server.1=fgedu01:2888:3888
server.2=fgedu02:2888:3888
server.3=fgedu03:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
EOF

# 配置myid
echo “1” > /bigdata/zookeeper/data/myid

# 验证安装
zkServer.sh version

# 下载解压
# 完成

# 环境变量
# 配置完成

# 配置文件
# 配置完成

# myid配置
# 配置完成

# 验证安装
Apache ZooKeeper, version 3.8.1
# ZooKeeper安装成功

3.1.2 启动ZooKeeper

# 启动ZooKeeper
zkServer.sh start

# 查看状态
zkServer.sh status

# 查看进程
jps | grep QuorumPeerMain

# 验证连接
zkCli.sh -server fgedu01:2181 <<< "quit"

# 启动ZooKeeper
ZooKeeper JMX enabled by default
Using config: /bigdata/app/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper … STARTED

# 查看状态
Mode: leader
Zxid: 0x100000000

# 进程查看
12345 QuorumPeerMain

# 验证连接
Connecting to fgedu01:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
# ZooKeeper启动成功

3.2 节点操作管理

3.2.1 创建节点

# 连接ZooKeeper
zkCli.sh -server fgedu01:2181

# 创建持久节点
create /fgedu/config “config_data”

# 创建临时节点
create -e /fgedu/temp “temp_data”

# 创建顺序节点
create -s /fgedu/seq “seq_data”

# 创建子节点
create /fgedu/config/db “mysql”
create /fgedu/config/cache “redis”

# 查看节点
ls /fgedu

# 连接ZooKeeper
# 连接成功

# 创建持久节点
Created /fgedu/config

# 创建临时节点
Created /fgedu/temp

# 创建顺序节点
Created /fgedu/seq0000000001

# 创建子节点
Created /fgedu/config/db
Created /fgedu/config/cache

# 查看节点
[config, temp, seq0000000001]

3.2.2 读取节点

# 获取节点数据
get /fgedu/config

# 获取节点详情
get -s /fgedu/config

# 获取子节点列表
ls /fgedu/config

# 获取子节点详情
ls -s /fgedu/config

# 递归获取节点
getAllChildrenNumber /fgedu

# 获取节点数据
config_data

# 获取节点详情
config_data
czxid = 0x100000001
ctime = Fri Jan 19 04:00:00 CST 2024
mzxid = 0x100000001
mtime = Fri Jan 19 04:00:00 CST 2024
pZxid = 0x100000005
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 11
numChildren = 2

# 子节点列表
[db, cache]

# 子节点详情
[db, cache]
total = 2

# 递归获取
5

3.2.3 更新和删除节点

# 更新节点数据
set /fgedu/config “new_config_data”

# 带版本更新
set /fgedu/config “version_config” 1

# 删除节点
delete /fgedu/temp

# 递归删除节点
deleteall /fgedu/test

# 验证删除
ls /fgedu

# 更新节点数据
cZxid = 0x100000001
ctime = Fri Jan 19 04:00:00 CST 2024
mZxid = 0x100000010
mtime = Fri Jan 19 04:10:00 CST 2024

# 带版本更新
cZxid = 0x100000001
mZxid = 0x100000011

# 删除节点
# 删除成功

# 递归删除
# 删除成功

# 验证删除
[config, seq0000000001]

3.3 集群监控维护

3.3.1 集群状态监控

# 查看集群状态
zkServer.sh status

# 查看所有节点状态
for host in fgedu01 fgedu02 fgedu03; do
echo “=== ${host} ===”
ssh ${host} “zkServer.sh status”
done

# 查看四字命令
echo “stat” | nc fgedu01 2181
echo “mntr” | nc fgedu01 2181
echo “conf” | nc fgedu01 2181

# 集群状态
Mode: leader
Zxid: 0x100000100

# 所有节点状态
=== fgedu01 ===
Mode: leader

=== fgedu02 ===
Mode: follower

=== fgedu03 ===
Mode: follower

# 四字命令
Zookeeper version: 3.8.1
Clients:
/192.168.1.10:12345[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/1/10
Received: 1000
Sent: 1000
Connections: 1
Outstanding: 0
Zxid: 0x100000100
Mode: leader
Node count: 100

3.3.2 集群维护操作

# 查看快照和日志
ls -la /bigdata/zookeeper/data/
ls -la /bigdata/zookeeper/logs/

# 清理旧快照
zkCleanup.sh /bigdata/zookeeper/data 3

# 备份数据
tar -czf /backup/zookeeper/zk_data_$(date +%Y%m%d).tar.gz /bigdata/zookeeper/data/

# 恢复数据
zkServer.sh stop
tar -xzf /backup/zookeeper/zk_data_20240119.tar.gz -C /
zkServer.sh start

# 快照和日志
total 100
-rw-r–r– 1 zookeeper zookeeper 1 Jan 19 04:00 myid
-rw-r–r– 1 zookeeper zookeeper 1024 Jan 19 04:00 snapshot.100000010

total 100
-rw-r–r– 1 zookeeper zookeeper 1024 Jan 19 04:00 log.100000001

# 清理旧快照
Removing old snapshots
# 清理完成

# 备份数据
# 备份完成

# 恢复数据
ZooKeeper JMX enabled by default
Using config: /bigdata/app/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper … STOPPED
# 恢复完成
Starting zookeeper … STARTED

3.4 客户端开发

3.4.1 Java客户端示例

// Java客户端代码
import org.apache.zookeeper.*;
import org.apache.zookeeper.data.Stat;

public class FgeduZkClient {
private ZooKeeper zk;

public void connect(String hosts) throws Exception {
zk = new ZooKeeper(hosts, 3000, event -> {
System.out.println(“Event: ” + event);
});
}

public void create(String path, String data) throws Exception {
zk.create(path, data.getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
}

public String getData(String path) throws Exception {
byte[] data = zk.getData(path, false, new Stat());
return new String(data);
}

public void close() throws Exception {
zk.close();
}
}

// 使用示例
FgeduZkClient client = new FgeduZkClient();
client.connect(“fgedu01:2181,fgedu02:2181,fgedu03:2181”);
client.create(“/fgedu/app”, “app_config”);
String data = client.getData(“/fgedu/app”);
System.out.println(“Data: ” + data);
client.close();

// Java客户端执行
Event: WatchedEvent state:SyncConnected type:None path:null
Data: app_config
# 客户端操作成功

3.4.2 Python客户端示例

# Python客户端代码
from kazoo.client import KazooClient

# 连接ZooKeeper
zk = KazooClient(hosts=’fgedu01:2181,fgedu02:2181,fgedu03:2181′)
zk.start()

# 创建节点
zk.ensure_path(‘/fgedu/python’)
zk.create(‘/fgedu/python/config’, b’python_config’)

# 读取节点
data, stat = zk.get(‘/fgedu/python/config’)
print(f”Data: {data.decode()}”)

# 更新节点
zk.set(‘/fgedu/python/config’, b’new_config’)

# 删除节点
zk.delete(‘/fgedu/python/config’)

# 关闭连接
zk.stop()

# Python客户端执行
Data: python_config
# 客户端操作成功

Part04-生产案例与实战讲解

4.1 配置中心案例

ZooKeeper可以作为分布式配置中心。更多视频教程www.fgedu.net.cn

#!/bin/bash
# zk_config_center.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

echo “=== ZooKeeper Config Center ===”
echo “Date: $(date)”

# 创建配置节点
zkCli.sh -server fgedu01:2181 << EOF
create /fgedu/config/database “mysql://fgedu01:3306/fgedudb”
create /fgedu/config/redis “redis://fgedu01:6379”
create /fgedu/config/kafka “fgedu01:9092,fgedu02:9092,fgedu03:9092”
quit
EOF

# 读取配置
echo “=== Read Config ===”
zkCli.sh -server fgedu01:2181 << EOF
get /fgedu/config/database
get /fgedu/config/redis
get /fgedu/config/kafka
quit
EOF

echo “=== Config Center Completed ===”

# 配置中心执行
./zk_config_center.sh
=== ZooKeeper Config Center ===
Date: Fri Jan 19 05:00:00 CST 2024

# 创建配置节点
Created /fgedu/config/database
Created /fgedu/config/redis
Created /fgedu/config/kafka

# 读取配置
=== Read Config ===
mysql://fgedu01:3306/fgedudb
redis://fgedu01:6379
fgedu01:9092,fgedu02:9092,fgedu03:9092

=== Config Center Completed ===

4.2 分布式锁案例

分布式锁是ZooKeeper的经典应用。学习交流加群风哥微信: itpux-com

# 分布式锁实现
zkCli.sh -server fgedu01:2181 << 'EOF' # 创建锁节点
create /fgedu/lock “”

# 获取锁(创建临时顺序节点)
create -e -s /fgedu/lock/request_ “”

# 查看锁节点
ls -s /fgedu/lock

# 判断是否获取锁
# 如果是最小序号节点,则获取锁成功

# 释放锁
delete /fgedu/lock/request_0000000001

quit
EOF

# 分布式锁执行

# 创建锁节点
Created /fgedu/lock

# 创建临时顺序节点
Created /fgedu/lock/request_0000000001

# 查看锁节点
[request_0000000001]
total = 1

# 释放锁
# 删除成功

4.3 服务注册发现案例

4.3.1 服务注册

# 服务注册
zkCli.sh -server fgedu01:2181 << 'EOF' # 创建服务根节点
create /fgedu/services “”

# 注册服务
create -e /fgedu/services/user-service “192.168.1.10:8080”
create -e /fgedu/services/order-service “192.168.1.11:8080”
create -e /fgedu/services/product-service “192.168.1.12:8080”

# 查看服务列表
ls /fgedu/services

quit
EOF

# 服务注册执行

# 创建服务根节点
Created /fgedu/services

# 注册服务
Created /fgedu/services/user-service
Created /fgedu/services/order-service
Created /fgedu/services/product-service

# 服务列表
[user-service, order-service, product-service]

4.3.2 服务发现

# 服务发现
zkCli.sh -server fgedu01:2181 << 'EOF' # 获取服务地址
get /fgedu/services/user-service

# 监听服务变化
ls -w /fgedu/services

# 获取所有服务
ls /fgedu/services

quit
EOF

# 服务发现执行

# 获取服务地址
192.168.1.10:8080

# 监听服务变化
WATCHER::
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/fgedu/services

# 所有服务
[user-service, order-service, product-service]

Part05-风哥经验总结与分享

5.1 ZooKeeper最佳实践

在实际生产环境中,ZooKeeper使用需要注意以下几点:from bigdata视频:www.itpux.com

风哥经验总结:
1. 集群节点数为奇数
2. 分离快照和事务日志
3. 监控集群状态
4. 定期备份数据
5. 合理设置会话超时

5.2 运维经验总结

5.2.1 运维建议

风哥提示:ZooKeeper运维需要关注集群状态、性能指标、数据备份等方面。

运维注意事项:
– 监控Leader选举
– 关注事务延迟
– 定期清理旧快照
– 做好容量规划
– 制定故障恢复方案

5.2.2 ZooKeeper运维脚本

#!/bin/bash
# zk_maintenance.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

echo “=== ZooKeeper Maintenance ===”
echo “Date: $(date)”

# 1. 检查集群状态
echo “=== Check Cluster Status ===”
for host in fgedu01 fgedu02 fgedu03; do
echo “— ${host} —“
ssh ${host} “zkServer.sh status”
done

# 2. 检查节点数量
echo “=== Check Node Count ===”
echo “stat” | nc fgedu01 2181 | grep “Node count”

# 3. 检查磁盘使用
echo “=== Check Disk Usage ===”
df -h /bigdata/zookeeper/

# 4. 清理旧快照
echo “=== Cleanup Old Snapshots ===”
zkCleanup.sh /bigdata/zookeeper/data 3

echo “=== Maintenance Completed ===”

# 维护执行
./zk_maintenance.sh
=== ZooKeeper Maintenance ===
Date: Fri Jan 19 06:00:00 CST 2024

=== Check Cluster Status ===
— fgedu01 —
Mode: leader
— fgedu02 —
Mode: follower
— fgedu03 —
Mode: follower

=== Check Node Count ===
Node count: 100

=== Check Disk Usage ===
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 10G 40G 20% /bigdata

=== Cleanup Old Snapshots ===
Removing old snapshots

=== Maintenance Completed ===

本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html

联系我们

在线咨询:点击这里给我发消息

微信号:itpux-com

工作日:9:30-18:30,节假日休息