1. 首页 > Memcached教程 > 正文

Memcached教程FG011-Memcached大Key问题识别与优化解决方案

内容简介:
本文详细介绍Memcached大Key问题的识别方法、影响分析和优化解决方案,包括大Key检测工具使用、大Key拆分策略、压缩优化方案、内存碎片处理以及生产环境大Key治理最佳实践。风哥教程参考Memcached官方文档Memory Management、Performance Optimization等章节内容。

目录大纲

Part01-基础概念与理论知识
    1.1 Memcached大Key问题概述
    1.2 Memcached大Key产生原因
    1.3 Memcached大Key影响分析
Part02-生产环境规划与建议
    2.1 Memcached大Key检测方案
    2.2 Memcached大Key阈值制定
    2.3 Memcached大Key预防策略
Part03-生产环境项目实施方案
    3.1 Memcached大Key检测工具部署
    3.2 Memcached大Key分析实战
    3.3 Memcached大Key拆分优化
Part04-生产案例与实战讲解
    4.1 Memcached大Key压缩方案
    4.2 Memcached大Key内存优化
    4.3 Memcached大Key治理案例
Part05-风哥经验总结与分享
    5.1 Memcached大Key优化经验
    5.2 Memcached大Key监控方案
    5.3 Memcached大Key最佳实践

Part01-基础概念与理论知识

1.1 Memcached大Key问题概述

Memcached大Key是指存储在缓存中的单个Key对应的Value值过大,通常超过1MB的数据被定义为大Key。更多视频教程www.fgedu.net.cn。大Key问题会导致内存分配不均、网络传输延迟增加、Slab内存碎片增多等问题,严重影响Memcached的整体性能和稳定性。

在生产环境中,大Key的常见形式包括:大型JSON对象、序列化的大对象、批量数据集合、大型HTML页面缓存、图片或文件二进制数据等。学习交流加群风哥微信: itpux-com。这些大Key会占用大量内存空间,并可能导致内存分配效率下降。

1.2 Memcached大Key产生原因

大Key产生的主要原因包括:业务设计不合理,将大量数据聚合存储在单个Key中;缓存策略不当,未对大数据进行分片处理;数据结构设计问题,使用不当的序列化方式;历史遗留问题,早期设计未考虑数据增长;缺乏监控和治理机制,大Key问题长期积累。

风哥提示:Memcached默认最大Value大小为1MB,但可以通过-I参数调整,不过不建议设置过大,会增加内存碎片和网络传输开销。

1.3 Memcached大Key影响分析

大Key对Memcached的影响主要体现在以下几个方面:内存分配效率下降,大Key需要分配较大的Slab,可能导致内存碎片;网络传输延迟增加,大Key的读写操作需要更长的网络传输时间;并发性能下降,大Key操作会占用更多CPU资源;内存利用率降低,大Key可能导致Slab分配不均衡;Eviction风险增加,大Key占用过多内存可能导致正常数据被驱逐。

# 查看当前Memcached配置
# echo “stats settings” | nc 192.168.1.101 11211 | grep -E ‘maxbytes|chunk_size|item_size_max’

[root@fgedu101 ~]# echo “stats settings” | nc 192.168.1.101 11211 | grep -E ‘maxbytes|chunk_size|item_size_max’
STAT maxbytes 64424509440
STAT chunk_size 48
STAT item_size_max 1048576

Part02-生产环境规划与建议

2.1 Memcached大Key检测方案

建立完善的大Key检测机制是治理大Key问题的基础。from Memcached视频:www.itpux.com。检测方案包括:实时检测,在数据写入时检查Value大小;定期扫描,定时扫描缓存中的大Key;统计分析,通过stats命令分析内存使用情况;日志记录,记录大Key操作日志便于追踪。

2.2 Memcached大Key阈值制定

根据业务特点制定合理的大Key阈值标准:一般建议Value大小不超过100KB;超过100KB需要评估是否需要拆分;超过500KB必须进行优化处理;超过1MB需要重新设计缓存策略。更多学习教程公众号风哥教程itpux_com。阈值制定需要综合考虑业务需求、网络带宽、内存资源等因素。

# 大Key阈值配置示例
# cat > /memcached/app/config/bigkey_threshold.conf << 'EOF'
# Memcached大Key阈值配置
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

# 警告阈值(字节)
WARNING_THRESHOLD=102400

# 严重阈值(字节)
CRITICAL_THRESHOLD=512000

# 最大允许阈值(字节)
MAX_ALLOWED=1048576

# 扫描间隔(秒)
SCAN_INTERVAL=3600

# 告警通知
ALERT_EMAIL=admin@fgedu.net.cn
EOF

[root@fgedu101 ~]# cat > /memcached/app/config/bigkey_threshold.conf << 'EOF'
> # Memcached大Key阈值配置
> # from:www.itpux.com.qq113257174.wx:itpux-com
> # web: http://www.fgedu.net.cn
>
> # 警告阈值(字节)
> WARNING_THRESHOLD=102400
>
> # 严重阈值(字节)
> CRITICAL_THRESHOLD=512000
>
> # 最大允许阈值(字节)
> MAX_ALLOWED=1048576
>
> # 扫描间隔(秒)
> SCAN_INTERVAL=3600
>
> # 告警通知
> ALERT_EMAIL=admin@fgedu.net.cn
> EOF

2.3 Memcached大Key预防策略

预防大Key问题需要从设计和开发阶段入手:数据分片存储,将大数据拆分为多个小Key;压缩存储,对大Value进行压缩处理;设置合理过期时间,避免大Key长期占用内存;使用合适的数据结构,减少序列化开销;建立代码审查机制,防止大Key代码上线。

Part03-生产环境项目实施方案

3.1 Memcached大Key检测工具部署

部署大Key检测工具,实现自动化检测和告警。更多视频教程www.fgedu.net.cn。

# 大Key检测脚本
# cat > /memcached/app/scripts/bigkey_scanner.sh << 'EOF'
#!/bin/bash
# bigkey_scanner.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

HOST=”192.168.1.101″
PORT=”11211″
WARNING_SIZE=102400
CRITICAL_SIZE=512000
LOG_FILE=”/memcached/app/logs/bigkey_scan.log”

echo “===== Memcached大Key扫描报告 =====” > $LOG_FILE
echo “扫描时间: $(date ‘+%Y-%m-%d %H:%M:%S’)” >> $LOG_FILE
echo “” >> $LOG_FILE

# 获取Slab统计
SLABS=$(echo “stats slabs” | nc $HOST $PORT)

# 分析每个Slab
echo “Slab分析结果:” >> $LOG_FILE
echo “$SLABS” | grep “total_pages” | while read line; do
SLAB_ID=$(echo $line | awk -F: ‘{print $1}’ | awk ‘{print $2}’)
CHUNK_SIZE=$(echo “$SLABS” | grep “STAT ${SLAB_ID}:chunk_size” | awk ‘{print $3}’)
USED_CHUNKS=$(echo “$SLABS” | grep “STAT ${SLAB_ID}:used_chunks” | awk ‘{print $3}’)

if [ -n “$CHUNK_SIZE” ] && [ -n “$USED_CHUNKS” ]; then
if [ $CHUNK_SIZE -gt $WARNING_SIZE ]; then
echo ” [警告] Slab $SLAB_ID: chunk_size=${CHUNK_SIZE}B, used_chunks=$USED_CHUNKS” >> $LOG_FILE
fi
fi
done

# 获取详细统计
echo “” >> $LOG_FILE
echo “内存使用统计:” >> $LOG_FILE
STATS=$(echo “stats” | nc $HOST $PORT)
BYTES=$(echo “$STATS” | grep “^STAT bytes ” | awk ‘{print $3}’)
LIMIT=$(echo “$STATS” | grep “limit_maxbytes” | awk ‘{print $3}’)
ITEMS=$(echo “$STATS” | grep “curr_items” | awk ‘{print $3}’)

echo ” 总内存使用: $((BYTES/1024/1024))MB” >> $LOG_FILE
echo ” 总Key数量: $ITEMS” >> $LOG_FILE
echo ” 平均Key大小: $((BYTES/ITEMS))B” >> $LOG_FILE

# 检测大Key数量
echo “” >> $LOG_FILE
echo “大Key检测:” >> $LOG_FILE
BIGKEY_COUNT=0
for slab_id in $(echo “$SLABS” | grep “chunk_size” | awk -F: ‘{print $1}’ | awk ‘{print $2}’ | sort -n); do
CHUNK_SIZE=$(echo “$SLABS” | grep “STAT ${slab_id}:chunk_size” | awk ‘{print $3}’)
USED_CHUNKS=$(echo “$SLABS” | grep “STAT ${slab_id}:used_chunks” | awk ‘{print $3}’)

if [ -n “$CHUNK_SIZE” ] && [ $CHUNK_SIZE -gt $WARNING_SIZE ] && [ $USED_CHUNKS -gt 0 ]; then
BIGKEY_COUNT=$((BIGKEY_COUNT + USED_CHUNKS))
fi
done

echo ” 大Key数量: $BIGKEY_COUNT” >> $LOG_FILE

# 告警判断
if [ $BIGKEY_COUNT -gt 100 ]; then
echo “” >> $LOG_FILE
echo “[严重告警] 发现大量大Key,请及时处理!” >> $LOG_FILE
# 发送告警邮件
# mail -s “Memcached大Key告警” admin@fgedu.net.cn < $LOG_FILE
fi

echo “扫描完成,报告已保存到: $LOG_FILE”
cat $LOG_FILE
EOF

# chmod +x /memcached/app/scripts/bigkey_scanner.sh

[root@fgedu101 ~]# cat > /memcached/app/scripts/bigkey_scanner.sh << 'EOF'
> #!/bin/bash
> # bigkey_scanner.sh
> # from:www.itpux.com.qq113257174.wx:itpux-com
> # web: http://www.fgedu.net.cn
>
> HOST=”192.168.1.101″
> PORT=”11211″
> WARNING_SIZE=102400
> CRITICAL_SIZE=512000
> LOG_FILE=”/memcached/app/logs/bigkey_scan.log”
>
> echo “===== Memcached大Key扫描报告 =====” > $LOG_FILE
> echo “扫描时间: $(date ‘+%Y-%m-%d %H:%M:%S’)” >> $LOG_FILE
> echo “” >> $LOG_FILE
>
> # 获取Slab统计
> SLABS=$(echo “stats slabs” | nc $HOST $PORT)
>
> # 分析每个Slab
> echo “Slab分析结果:” >> $LOG_FILE
> echo “$SLABS” | grep “total_pages” | while read line; do
> SLAB_ID=$(echo $line | awk -F: ‘{print $1}’ | awk ‘{print $2}’)
> CHUNK_SIZE=$(echo “$SLABS” | grep “STAT ${SLAB_ID}:chunk_size” | awk ‘{print $3}’)
> USED_CHUNKS=$(echo “$SLABS” | grep “STAT ${SLAB_ID}:used_chunks” | awk ‘{print $3}’)
>
> if [ -n “$CHUNK_SIZE” ] && [ -n “$USED_CHUNKS” ]; then
> if [ $CHUNK_SIZE -gt $WARNING_SIZE ]; then
> echo ” [警告] Slab $SLAB_ID: chunk_size=${CHUNK_SIZE}B, used_chunks=$USED_CHUNKS” >> $LOG_FILE
> fi
> fi
> done
>
> # 获取详细统计
> echo “” >> $LOG_FILE
> echo “内存使用统计:” >> $LOG_FILE
> STATS=$(echo “stats” | nc $HOST $PORT)
> BYTES=$(echo “$STATS” | grep “^STAT bytes ” | awk ‘{print $3}’)
> LIMIT=$(echo “$STATS” | grep “limit_maxbytes” | awk ‘{print $3}’)
> ITEMS=$(echo “$STATS” | grep “curr_items” | awk ‘{print $3}’)
>
> echo ” 总内存使用: $((BYTES/1024/1024))MB” >> $LOG_FILE
> echo ” 总Key数量: $ITEMS” >> $LOG_FILE
> echo ” 平均Key大小: $((BYTES/ITEMS))B” >> $LOG_FILE
>
> # 检测大Key数量
> echo “” >> $LOG_FILE
> echo “大Key检测:” >> $LOG_FILE
> BIGKEY_COUNT=0
> for slab_id in $(echo “$SLABS” | grep “chunk_size” | awk -F: ‘{print $1}’ | awk ‘{print $2}’ | sort -n); do
> CHUNK_SIZE=$(echo “$SLABS” | grep “STAT ${slab_id}:chunk_size” | awk ‘{print $3}’)
> USED_CHUNKS=$(echo “$SLABS” | grep “STAT ${slab_id}:used_chunks” | awk ‘{print $3}’)
>
> if [ -n “$CHUNK_SIZE” ] && [ $CHUNK_SIZE -gt $WARNING_SIZE ] && [ $USED_CHUNKS -gt 0 ]; then
> BIGKEY_COUNT=$((BIGKEY_COUNT + USED_CHUNKS))
> fi
> done
>
> echo ” 大Key数量: $BIGKEY_COUNT” >> $LOG_FILE
>
> # 告警判断
> if [ $BIGKEY_COUNT -gt 100 ]; then
> echo “” >> $LOG_FILE
> echo “[严重告警] 发现大量大Key,请及时处理!” >> $LOG_FILE
> # 发送告警邮件
> # mail -s “Memcached大Key告警” admin@fgedu.net.cn < $LOG_FILE
> fi
>
> echo “扫描完成,报告已保存到: $LOG_FILE”
> cat $LOG_FILE
> EOF

[root@fgedu101 ~]# chmod +x /memcached/app/scripts/bigkey_scanner.sh

3.2 Memcached大Key分析实战

执行大Key扫描,分析当前缓存中的大Key分布情况。学习交流加群风哥QQ113257174。

# 执行大Key扫描
# /memcached/app/scripts/bigkey_scanner.sh

[root@fgedu101 ~]# /memcached/app/scripts/bigkey_scanner.sh
===== Memcached大Key扫描报告 =====
扫描时间: 2026-04-08 10:45:00

Slab分析结果:
[警告] Slab 15: chunk_size=108344B, used_chunks=234
[警告] Slab 16: chunk_size=135424B, used_chunks=156
[警告] Slab 17: chunk_size=169280B, used_chunks=89
[警告] Slab 18: chunk_size=211600B, used_chunks=45
[警告] Slab 19: chunk_size=264496B, used_chunks=23
[警告] Slab 20: chunk_size=330616B, used_chunks=12
[警告] Slab 21: chunk_size=413272B, used_chunks=8
[警告] Slab 22: chunk_size=516592B, used_chunks=5

内存使用统计:
总内存使用: 15360MB
总Key数量: 1234567
平均Key大小: 12B

大Key检测:
大Key数量: 572

扫描完成,报告已保存到: /memcached/app/logs/bigkey_scan.log

# 查看Slab详细信息
# echo “stats slabs” | nc 192.168.1.101 11211 | head -50

[root@fgedu101 ~]# echo “stats slabs” | nc 192.168.1.101 11211 | head -50
STAT 1:chunk_size 96
STAT 1:chunks_per_page 10922
STAT 1:total_pages 5
STAT 1:total_chunks 54610
STAT 1:used_chunks 45678
STAT 1:free_chunks 8932
STAT 1:free_chunks_end 0
STAT 2:chunk_size 120
STAT 2:chunks_per_page 8738
STAT 2:total_pages 8
STAT 2:total_chunks 69904
STAT 2:used_chunks 62345
STAT 2:free_chunks 7559
STAT 2:free_chunks_end 0

STAT 15:chunk_size 108344
STAT 15:chunks_per_page 9
STAT 15:total_pages 26
STAT 15:total_chunks 234
STAT 15:used_chunks 234
STAT 15:free_chunks 0
STAT 15:free_chunks_end 0
STAT 16:chunk_size 135424
STAT 16:chunks_per_page 7
STAT 16:total_pages 23
STAT 16:total_chunks 156
STAT 16:used_chunks 156
STAT 16:free_chunks 0
STAT 16:free_chunks_end 0
STAT active_slabs 22
STAT total_malloced 16106127360
END

3.3 Memcached大Key拆分优化

大Key拆分是解决大Key问题的核心方案,将大Value拆分为多个小Key存储。from Memcached视频:www.itpux.com。

# 大Key拆分示例脚本
# cat > /memcached/app/scripts/bigkey_split.py << 'EOF'
#!/usr/bin/env python3
# bigkey_split.py
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

import memcache
import json
import hashlib

class BigKeySplitter:
def __init__(self, servers=[‘192.168.1.101:11211’]):
self.mc = memcache.Client(servers, debug=0)
self.chunk_size = 50000 # 50KB per chunk

def split_key(self, key, value):
“””拆分大Key为多个小Key”””
if isinstance(value, (dict, list)):
value_str = json.dumps(value)
else:
value_str = str(value)

value_bytes = value_str.encode(‘utf-8’)
total_size = len(value_bytes)

if total_size <= self.chunk_size:
return False # 不需要拆分

# 计算需要的分片数量
chunk_count = (total_size + self.chunk_size – 1) // self.chunk_size

# 存储元数据
meta = {
‘total_size’: total_size,
‘chunk_count’: chunk_count,
‘chunk_size’: self.chunk_size
}
self.mc.set(f”{key}:meta”, meta)

# 拆分存储
for i in range(chunk_count):
start = i * self.chunk_size
end = min((i + 1) * self.chunk_size, total_size)
chunk_data = value_bytes[start:end]
chunk_key = f”{key}:chunk:{i}”
self.mc.set(chunk_key, chunk_data)

print(f”拆分完成: {key} -> {chunk_count}个分片, 总大小: {total_size}B”)
return True

def get_split_key(self, key):
“””获取拆分后的完整数据”””
meta = self.mc.get(f”{key}:meta”)
if not meta:
return self.mc.get(key) # 未拆分,直接获取

# 合并分片数据
chunks = []
for i in range(meta[‘chunk_count’]):
chunk_key = f”{key}:chunk:{i}”
chunk_data = self.mc.get(chunk_key)
if chunk_data:
chunks.append(chunk_data)

if len(chunks) == meta[‘chunk_count’]:
full_data = b”.join(chunks)
return json.loads(full_data.decode(‘utf-8’))
return None

def delete_split_key(self, key):
“””删除拆分的Key”””
meta = self.mc.get(f”{key}:meta”)
if meta:
for i in range(meta[‘chunk_count’]):
self.mc.delete(f”{key}:chunk:{i}”)
self.mc.delete(f”{key}:meta”)
else:
self.mc.delete(key)

if __name__ == “__main__”:
splitter = BigKeySplitter()

# 测试大Key拆分
big_data = {“fgedu_users”: [f”user_{i}” for i in range(10000)]}
splitter.split_key(“fgedu:user_list”, big_data)

# 获取拆分后的数据
result = splitter.get_split_key(“fgedu:user_list”)
print(f”获取数据成功,用户数量: {len(result[‘fgedu_users’])}”)
EOF

# chmod +x /memcached/app/scripts/bigkey_split.py

[root@fgedu101 ~]# cat > /memcached/app/scripts/bigkey_split.py << 'EOF'
> #!/usr/bin/env python3
> # bigkey_split.py
> # from:www.itpux.com.qq113257174.wx:itpux-com
> # web: http://www.fgedu.net.cn
>
> import memcache
> import json
> import hashlib
>
> class BigKeySplitter:
> def __init__(self, servers=[‘192.168.1.101:11211’]):
> self.mc = memcache.Client(servers, debug=0)
> self.chunk_size = 50000 # 50KB per chunk
>
> def split_key(self, key, value):
> “””拆分大Key为多个小Key”””
> if isinstance(value, (dict, list)):
> value_str = json.dumps(value)
> else:
> value_str = str(value)
>
> value_bytes = value_str.encode(‘utf-8’)
> total_size = len(value_bytes)
>
> if total_size <= self.chunk_size:
> return False # 不需要拆分
>
> # 计算需要的分片数量
> chunk_count = (total_size + self.chunk_size – 1) // self.chunk_size
>
> # 存储元数据
> meta = {
> ‘total_size’: total_size,
> ‘chunk_count’: chunk_count,
> ‘chunk_size’: self.chunk_size
> }
> self.mc.set(f”{key}:meta”, meta)
>
> # 拆分存储
> for i in range(chunk_count):
> start = i * self.chunk_size
> end = min((i + 1) * self.chunk_size, total_size)
> chunk_data = value_bytes[start:end]
> chunk_key = f”{key}:chunk:{i}”
> self.mc.set(chunk_key, chunk_data)
>
> print(f”拆分完成: {key} -> {chunk_count}个分片, 总大小: {total_size}B”)
> return True
>
> def get_split_key(self, key):
> “””获取拆分后的完整数据”””
> meta = self.mc.get(f”{key}:meta”)
> if not meta:
> return self.mc.get(key) # 未拆分,直接获取
>
> # 合并分片数据
> chunks = []
> for i in range(meta[‘chunk_count’]):
> chunk_key = f”{key}:chunk:{i}”
> chunk_data = self.mc.get(chunk_key)
> if chunk_data:
chunks.append(chunk_data)
>
> if len(chunks) == meta[‘chunk_count’]:
> full_data = b”.join(chunks)
> return json.loads(full_data.decode(‘utf-8’))
> return None
>
> def delete_split_key(self, key):
> “””删除拆分的Key”””
> meta = self.mc.get(f”{key}:meta”)
> if meta:
> for i in range(meta[‘chunk_count’]):
> self.mc.delete(f”{key}:chunk:{i}”)
> self.mc.delete(f”{key}:meta”)
> else:
> self.mc.delete(key)
>
> if __name__ == “__main__”:
> splitter = BigKeySplitter()
>
> # 测试大Key拆分
> big_data = {“fgedu_users”: [f”user_{i}” for i in range(10000)]}
> splitter.split_key(“fgedu:user_list”, big_data)
>
> # 获取拆分后的数据
> result = splitter.get_split_key(“fgedu:user_list”)
> print(f”获取数据成功,用户数量: {len(result[‘fgedu_users’])}”)
> EOF

[root@fgedu101 ~]# chmod +x /memcached/app/scripts/bigkey_split.py

Part04-生产案例与实战讲解

4.1 Memcached大Key压缩方案

压缩是处理大Key的有效方法之一,可以显著减少Value大小。更多视频教程www.fgedu.net.cn。常用的压缩算法包括GZIP、Snappy、LZ4等,需要根据压缩率和压缩速度选择合适的算法。

# 大Key压缩存储示例
# cat > /memcached/app/scripts/bigkey_compress.py << 'EOF'
#!/usr/bin/env python3
# bigkey_compress.py
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

import memcache
import gzip
import json
import base64

class CompressedCache:
def __init__(self, servers=[‘192.168.1.101:11211’]):
self.mc = memcache.Client(servers, debug=0)
self.compress_threshold = 10240 # 10KB以上压缩

def set_compressed(self, key, value, compress_level=6):
“””压缩存储数据”””
value_str = json.dumps(value) if isinstance(value, (dict, list)) else str(value)
value_bytes = value_str.encode(‘utf-8’)

if len(value_bytes) > self.compress_threshold:
# 压缩数据
compressed = gzip.compress(value_bytes, compresslevel=compress_level)
# 标记为压缩数据
data = {
‘compressed’: True,
‘data’: base64.b64encode(compressed).decode(‘utf-8’)
}
print(f”压缩存储: {key}, 原始大小: {len(value_bytes)}B, 压缩后: {len(compressed)}B, 压缩率: {len(compressed)*100//len(value_bytes)}%”)
return self.mc.set(key, data)
else:
return self.mc.set(key, value)

def get_compressed(self, key):
“””获取压缩数据并解压”””
data = self.mc.get(key)
if not data:
return None

if isinstance(data, dict) and data.get(‘compressed’):
# 解压数据
compressed = base64.b64decode(data[‘data’])
value_bytes = gzip.decompress(compressed)
return json.loads(value_bytes.decode(‘utf-8’))
else:
return data

if __name__ == “__main__”:
cache = CompressedCache()

# 测试压缩存储
big_data = {
“fgedu_products”: [
{“id”: i, “name”: f”product_{i}”, “desc”: “x” * 1000}
for i in range(1000)
]
}

cache.set_compressed(“fgedu:product_list”, big_data)
result = cache.get_compressed(“fgedu:product_list”)
print(f”获取数据成功,商品数量: {len(result[‘fgedu_products’])}”)
EOF

# chmod +x /memcached/app/scripts/bigkey_compress.py

[root@fgedu101 ~]# cat > /memcached/app/scripts/bigkey_compress.py << 'EOF'
> #!/usr/bin/env python3
> # bigkey_compress.py
> # from:www.itpux.com.qq113257174.wx:itpux-com
> # web: http://www.fgedu.net.cn
>
> import memcache
> import gzip
> import json
> import base64
>
> class CompressedCache:
> def __init__(self, servers=[‘192.168.1.101:11211’]):
> self.mc = memcache.Client(servers, debug=0)
> self.compress_threshold = 10240 # 10KB以上压缩
>
> def set_compressed(self, key, value, compress_level=6):
> “””压缩存储数据”””
> value_str = json.dumps(value) if isinstance(value, (dict, list)) else str(value)
> value_bytes = value_str.encode(‘utf-8’)
>
> if len(value_bytes) > self.compress_threshold:
> # 压缩数据
> compressed = gzip.compress(value_bytes, compresslevel=compress_level)
> # 标记为压缩数据
> data = {
> ‘compressed’: True,
> ‘data’: base64.b64encode(compressed).decode(‘utf-8’)
> }
> print(f”压缩存储: {key}, 原始大小: {len(value_bytes)}B, 压缩后: {len(compressed)}B, 压缩率: {len(compressed)*100//len(value_bytes)}%”)
> return self.mc.set(key, data)
> else:
> return self.mc.set(key, value)
>
> def get_compressed(self, key):
> “””获取压缩数据并解压”””
> data = self.mc.get(key)
> if not data:
> return None
>
> if isinstance(data, dict) and data.get(‘compressed’):
> # 解压数据
> compressed = base64.b64decode(data[‘data’])
> value_bytes = gzip.decompress(compressed)
> return json.loads(value_bytes.decode(‘utf-8’))
> else:
> return data
>
> if __name__ == “__main__”:
> cache = CompressedCache()
>
> # 测试压缩存储
> big_data = {
> “fgedu_products”: [
> {“id”: i, “name”: f”product_{i}”, “desc”: “x” * 1000}
> for i in range(1000)
> ]
> }
>
> cache.set_compressed(“fgedu:product_list”, big_data)
> result = cache.get_compressed(“fgedu:product_list”)
> print(f”获取数据成功,商品数量: {len(result[‘fgedu_products’])}”)
> EOF

[root@fgedu101 ~]# chmod +x /memcached/app/scripts/bigkey_compress.py

4.2 Memcached大Key内存优化

通过调整Slab参数优化大Key的内存分配效率。学习交流加群风哥微信: itpux-com。

# 分析当前Slab分配情况
# echo “stats slabs” | nc 192.168.1.101 11211 | grep -E “chunk_size|used_chunks|total_pages”

[root@fgedu101 ~]# echo “stats slabs” | nc 192.168.1.101 11211 | grep -E “chunk_size|used_chunks|total_pages”
STAT 1:chunk_size 96
STAT 1:used_chunks 45678
STAT 1:total_pages 5
STAT 2:chunk_size 120
STAT 2:used_chunks 62345
STAT 2:total_pages 8

STAT 15:chunk_size 108344
STAT 15:used_chunks 234
STAT 15:total_pages 26
STAT 16:chunk_size 135424
STAT 16:used_chunks 156
STAT 16:total_pages 23
STAT 17:chunk_size 169280
STAT 17:used_chunks 89
STAT 17:total_pages 13
STAT 18:chunk_size 211600
STAT 18:used_chunks 45
STAT 18:total_pages 7
STAT 19:chunk_size 264496
STAT 19:used_chunks 23
STAT 19:total_pages 4
STAT 20:chunk_size 330616
STAT 20:used_chunks 12
STAT 20:total_pages 2
STAT 21:chunk_size 413272
STAT 21:used_chunks 8
STAT 21:total_pages 1
STAT 22:chunk_size 516592
STAT 22:used_chunks 5
STAT 22:total_pages 1

# 优化Memcached Slab配置
# 调整增长因子以优化大Key内存分配
# 停止服务
# systemctl stop memcached

# 修改启动参数,调整增长因子
# sed -i ‘s/-f 1.25/-f 1.15/’ /etc/systemd/system/memcached.service

# 调整最小chunk大小
# sed -i ‘s/-n 48/-n 64/’ /etc/systemd/system/memcached.service

# 重载并启动服务
# systemctl daemon-reload
# systemctl start memcached

# 验证配置
# echo “stats settings” | nc 192.168.1.101 11211 | grep -E ‘chunk_size|growth_factor’

[root@fgedu101 ~]# systemctl stop memcached

[root@fgedu101 ~]# sed -i ‘s/-f 1.25/-f 1.15/’ /etc/systemd/system/memcached.service

[root@fgedu101 ~]# sed -i ‘s/-n 48/-n 64/’ /etc/systemd/system/memcached.service

[root@fgedu101 ~]# systemctl daemon-reload
[root@fgedu101 ~]# systemctl start memcached

[root@fgedu101 ~]# echo “stats settings” | nc 192.168.1.101 11211 | grep -E ‘chunk_size|growth_factor’
STAT chunk_size 64
STAT growth_factor 1.15

风哥提示:调整增长因子后,Slab分配会更细粒度,减少大Key的内存浪费。但会增加Slab数量,需要根据实际数据大小分布进行调整。

4.3 Memcached大Key治理案例

生产环境大Key治理实战案例:某电商平台商品列表缓存优化。from Memcached视频:www.itpux.com。

# 大Key治理案例:商品列表缓存优化
# cat > /memcached/app/scripts/product_cache_optimize.py << 'EOF'
#!/usr/bin/env python3
# product_cache_optimize.py
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

import memcache
import json

class ProductCacheOptimizer:
def __init__(self, servers=[‘192.168.1.101:11211’]):
self.mc = memcache.Client(servers, debug=0)
self.page_size = 50 # 每页50条

def set_product_list(self, category_id, products):
“””分页存储商品列表,避免大Key”””
total = len(products)
page_count = (total + self.page_size – 1) // self.page_size

# 存储元数据
meta = {
‘category_id’: category_id,
‘total’: total,
‘page_count’: page_count,
‘page_size’: self.page_size
}
self.mc.set(f”fgedu:product:meta:{category_id}”, meta, time=3600)

# 分页存储
for page in range(page_count):
start = page * self.page_size
end = min((page + 1) * self.page_size, total)
page_products = products[start:end]
self.mc.set(f”fgedu:product:page:{category_id}:{page}”, page_products, time=3600)

print(f”商品列表分页存储完成: category_id={category_id}, 总数={total}, 页数={page_count}”)

def get_product_list(self, category_id, page=0):
“””获取指定页的商品列表”””
meta = self.mc.get(f”fgedu:product:meta:{category_id}”)
if not meta:
return None

page_data = self.mc.get(f”fgedu:product:page:{category_id}:{page}”)
return {
‘total’: meta[‘total’],
‘page’: page,
‘page_size’: meta[‘page_size’],
‘products’: page_data
}

def set_product_detail(self, product_id, detail):
“””存储商品详情,使用压缩”””
import gzip
import base64

detail_str = json.dumps(detail)
detail_bytes = detail_str.encode(‘utf-8’)

if len(detail_bytes) > 10240: # 大于10KB压缩
compressed = gzip.compress(detail_bytes)
data = {
‘compressed’: True,
‘data’: base64.b64encode(compressed).decode(‘utf-8’)
}
print(f”商品详情压缩存储: product_id={product_id}, 原始={len(detail_bytes)}B, 压缩后={len(compressed)}B”)
else:
data = detail

self.mc.set(f”fgedu:product:detail:{product_id}”, data, time=3600)

def get_product_detail(self, product_id):
“””获取商品详情”””
import gzip
import base64

data = self.mc.get(f”fgedu:product:detail:{product_id}”)
if not data:
return None

if isinstance(data, dict) and data.get(‘compressed’):
compressed = base64.b64decode(data[‘data’])
detail_bytes = gzip.decompress(compressed)
return json.loads(detail_bytes.decode(‘utf-8’))
return data

if __name__ == “__main__”:
optimizer = ProductCacheOptimizer()

# 测试商品列表分页存储
products = [{“id”: i, “name”: f”商品{i}”, “price”: i * 10} for i in range(500)]
optimizer.set_product_list(1001, products)

# 获取第一页数据
result = optimizer.get_product_list(1001, page=0)
print(f”获取第1页数据,商品数: {len(result[‘products’])}”)

# 测试商品详情压缩存储
detail = {
“id”: 10001,
“name”: “测试商品”,
“description”: “x” * 50000, # 大文本
“specs”: {“color”: “red”, “size”: “L”}
}
optimizer.set_product_detail(10001, detail)
result = optimizer.get_product_detail(10001)
print(f”获取商品详情成功: {result[‘name’]}”)
EOF

# chmod +x /memcached/app/scripts/product_cache_optimize.py

[root@fgedu101 ~]# cat > /memcached/app/scripts/product_cache_optimize.py << 'EOF'
> #!/usr/bin/env python3
> # product_cache_optimize.py
> # from:www.itpux.com.qq113257174.wx:itpux-com
> # web: http://www.fgedu.net.cn
>
> import memcache
> import json
>
> class ProductCacheOptimizer:
> def __init__(self, servers=[‘192.168.1.101:11211’]):
> self.mc = memcache.Client(servers, debug=0)
> self.page_size = 50 # 每页50条
>
> def set_product_list(self, category_id, products):
> “””分页存储商品列表,避免大Key”””
> total = len(products)
> page_count = (total + self.page_size – 1) // self.page_size
>
> # 存储元数据
> meta = {
> ‘category_id’: category_id,
> ‘total’: total,
> ‘page_count’: page_count,
> ‘page_size’: self.page_size
> }
> self.mc.set(f”fgedu:product:meta:{category_id}”, meta, time=3600)
>
> # 分页存储
> for page in range(page_count):
> start = page * self.page_size
> end = min((page + 1) * self.page_size, total)
> page_products = products[start:end]
> self.mc.set(f”fgedu:product:page:{category_id}:{page}”, page_products, time=3600)
>
> print(f”商品列表分页存储完成: category_id={category_id}, 总数={total}, 页数={page_count}”)
>
> def get_product_list(self, category_id, page=0):
> “””获取指定页的商品列表”””
> meta = self.mc.get(f”fgedu:product:meta:{category_id}”)
> if not meta:
> return None
>
> page_data = self.mc.get(f”fgedu:product:page:{category_id}:{page}”)
> return {
> ‘total’: meta[‘total’],
> ‘page’: page,
> ‘page_size’: meta[‘page_size’],
> ‘products’: page_data
> }
>
> def set_product_detail(self, product_id, detail):
> “””存储商品详情,使用压缩”””
> import gzip
> import base64
>
> detail_str = json.dumps(detail)
> detail_bytes = detail_str.encode(‘utf-8’)
>
> if len(detail_bytes) > 10240: # 大于10KB压缩
> compressed = gzip.compress(detail_bytes)
> data = {
> ‘compressed’: True,
> ‘data’: base64.b64encode(compressed).decode(‘utf-8’)
> }
> print(f”商品详情压缩存储: product_id={product_id}, 原始={len(detail_bytes)}B, 压缩后={len(compressed)}B”)
> else:
> data = detail
>
> self.mc.set(f”fgedu:product:detail:{product_id}”, data, time=3600)
>
> def get_product_detail(self, product_id):
> “””获取商品详情”””
> import gzip
> import base64
>
> data = self.mc.get(f”fgedu:product:detail:{product_id}”)
> if not data:
> return None
>
> if isinstance(data, dict) and data.get(‘compressed’):
> compressed = base64.b64decode(data[‘data’])
> detail_bytes = gzip.decompress(compressed)
> return json.loads(detail_bytes.decode(‘utf-8’))
> return data
>
> if __name__ == “__main__”:
> optimizer = ProductCacheOptimizer()
>
> # 测试商品列表分页存储
> products = [{“id”: i, “name”: f”商品{i}”, “price”: i * 10} for i in range(500)]
> optimizer.set_product_list(1001, products)
>
> # 获取第一页数据
> result = optimizer.get_product_list(1001, page=0)
> print(f”获取第1页数据,商品数: {len(result[‘products’])}”)
>
> # 测试商品详情压缩存储
> detail = {
> “id”: 10001,
> “name”: “测试商品”,
> “description”: “x” * 50000, # 大文本
> “specs”: {“color”: “red”, “size”: “L”}
> }
> optimizer.set_product_detail(10001, detail)
> result = optimizer.get_product_detail(10001)
> print(f”获取商品详情成功: {result[‘name’]}”)
> EOF

[root@fgedu101 ~]# chmod +x /memcached/app/scripts/product_cache_optimize.py

Part05-风哥经验总结与分享

5.1 Memcached大Key优化经验

在多年生产环境实践中,总结出以下大Key优化经验:首先,从设计阶段就要考虑数据大小,避免产生大Key;其次,对于已有的大Key,优先考虑拆分方案,其次考虑压缩方案;第三,建立大Key监控机制,及时发现和处理问题;第四,制定大Key治理规范,纳入代码审查流程。更多学习教程公众号风哥教程itpux_com。

# 大Key治理检查清单
# cat > /memcached/app/docs/bigkey_checklist.md << 'EOF'
# Memcached大Key治理检查清单
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

## 设计阶段
– [ ] 评估单个Key的Value大小
– [ ] 设计合理的数据分片策略
– [ ] 选择合适的序列化方式
– [ ] 设置合理的过期时间

## 开发阶段
– [ ] 实现大Key检测逻辑
– [ ] 使用压缩存储大Value
– [ ] 分页存储列表数据
– [ ] 编写单元测试验证

## 测试阶段
– [ ] 执行大Key扫描
– [ ] 验证压缩效果
– [ ] 测试分片读取性能
– [ ] 压力测试验证

## 上线阶段
– [ ] 配置大Key告警
– [ ] 启用定期扫描
– [ ] 监控内存使用
– [ ] 记录优化效果
EOF

[root@fgedu101 ~]# cat > /memcached/app/docs/bigkey_checklist.md << 'EOF'
> # Memcached大Key治理检查清单
> # from:www.itpux.com.qq113257174.wx:itpux-com
> # web: http://www.fgedu.net.cn
>
> ## 设计阶段
> – [ ] 评估单个Key的Value大小
> – [ ] 设计合理的数据分片策略
> – [ ] 选择合适的序列化方式
> – [ ] 设置合理的过期时间
>
> ## 开发阶段
> – [ ] 实现大Key检测逻辑
> – [ ] 使用压缩存储大Value
> – [ ] 分页存储列表数据
> – [ ] 编写单元测试验证
>
> ## 测试阶段
> – [ ] 执行大Key扫描
> – [ ] 验证压缩效果
> – [ ] 测试分片读取性能
> – [ ] 压力测试验证
>
> ## 上线阶段
> – [ ] 配置大Key告警
> – [ ] 启用定期扫描
> – [ ] 监控内存使用
> – [ ] 记录优化效果
> EOF

5.2 Memcached大Key监控方案

建立完善的大Key监控体系,实现问题的及时发现和处理。学习交流加群风哥QQ113257174。

# 大Key监控脚本
# cat > /memcached/app/scripts/bigkey_monitor.sh << 'EOF'
#!/bin/bash
# bigkey_monitor.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn

HOST=”192.168.1.101″
PORT=”11211″
WARNING_THRESHOLD=100
CRITICAL_THRESHOLD=500
LOG_FILE=”/memcached/app/logs/bigkey_monitor.log”
ALERT_LOG=”/memcached/app/logs/bigkey_alert.log”

# 获取大Key统计
SLABS=$(echo “stats slabs” | nc $HOST $PORT)

BIGKEY_COUNT=0
BIGKEY_SIZE=0

for slab_id in $(echo “$SLABS” | grep “chunk_size” | awk -F: ‘{print $1}’ | awk ‘{print $2}’ | sort -n); do
CHUNK_SIZE=$(echo “$SLABS” | grep “STAT ${slab_id}:chunk_size” | awk ‘{print $3}’)
USED_CHUNKS=$(echo “$SLABS” | grep “STAT ${slab_id}:used_chunks” | awk ‘{print $3}’)

if [ -n “$CHUNK_SIZE” ] && [ $CHUNK_SIZE -gt 102400 ] && [ $USED_CHUNKS -gt 0 ]; then
BIGKEY_COUNT=$((BIGKEY_COUNT + USED_CHUNKS))
BIGKEY_SIZE=$((BIGKEY_SIZE + CHUNK_SIZE * USED_CHUNKS))
fi
done

TIMESTAMP=$(date ‘+%Y-%m-%d %H:%M:%S’)
echo “$TIMESTAMP,BIGKEY_COUNT=$BIGKEY_COUNT,BIGKEY_SIZE=$BIGKEY_SIZE” >> $LOG_FILE

# 告警判断
if [ $BIGKEY_COUNT -gt $CRITICAL_THRESHOLD ]; then
echo “[$TIMESTAMP] [CRITICAL] 大Key数量: $BIGKEY_COUNT, 总大小: $((BIGKEY_SIZE/1024/1024))MB” >> $ALERT_LOG
# 发送告警
# curl -X POST “webhook_url” -d “{\”text\”:\”Memcached大Key告警: 数量$BIGKEY_COUNT\”}”
elif [ $BIGKEY_COUNT -gt $WARNING_THRESHOLD ]; then
echo “[$TIMESTAMP] [WARNING] 大Key数量: $BIGKEY_COUNT, 总大小: $((BIGKEY_SIZE/1024/1024))MB” >> $ALERT_LOG
fi

echo “监控完成: 大Key数量=$BIGKEY_COUNT, 总大小=$((BIGKEY_SIZE/1024/1024))MB”
EOF

# chmod +x /memcached/app/scripts/bigkey_monitor.sh

# 添加定时任务
# echo “*/30 * * * * /memcached/app/scripts/bigkey_monitor.sh” >> /var/spool/cron/root

[root@fgedu101 ~]# cat > /memcached/app/scripts/bigkey_monitor.sh << 'EOF'
> #!/bin/bash
> # bigkey_monitor.sh
> # from:www.itpux.com.qq113257174.wx:itpux-com
> # web: http://www.fgedu.net.cn
>
> HOST=”192.168.1.101″
> PORT=”11211″
> WARNING_THRESHOLD=100
> CRITICAL_THRESHOLD=500
> LOG_FILE=”/memcached/app/logs/bigkey_monitor.log”
> ALERT_LOG=”/memcached/app/logs/bigkey_alert.log”
>
> # 获取大Key统计
> SLABS=$(echo “stats slabs” | nc $HOST $PORT)
>
> BIGKEY_COUNT=0
> BIGKEY_SIZE=0
>
> for slab_id in $(echo “$SLABS” | grep “chunk_size” | awk -F: ‘{print $1}’ | awk ‘{print $2}’ | sort -n); do
> CHUNK_SIZE=$(echo “$SLABS” | grep “STAT ${slab_id}:chunk_size” | awk ‘{print $3}’)
> USED_CHUNKS=$(echo “$SLABS” | grep “STAT ${slab_id}:used_chunks” | awk ‘{print $3}’)
>
> if [ -n “$CHUNK_SIZE” ] && [ $CHUNK_SIZE -gt 102400 ] && [ $USED_CHUNKS -gt 0 ]; then
> BIGKEY_COUNT=$((BIGKEY_COUNT + USED_CHUNKS))
> BIGKEY_SIZE=$((BIGKEY_SIZE + CHUNK_SIZE * USED_CHUNKS))
> fi
> done
>
> TIMESTAMP=$(date ‘+%Y-%m-%d %H:%M:%S’)
> echo “$TIMESTAMP,BIGKEY_COUNT=$BIGKEY_COUNT,BIGKEY_SIZE=$BIGKEY_SIZE” >> $LOG_FILE
>
> # 告警判断
> if [ $BIGKEY_COUNT -gt $CRITICAL_THRESHOLD ]; then
> echo “[$TIMESTAMP] [CRITICAL] 大Key数量: $BIGKEY_COUNT, 总大小: $((BIGKEY_SIZE/1024/1024))MB” >> $ALERT_LOG
> # 发送告警
> # curl -X POST “webhook_url” -d “{\”text\”:\”Memcached大Key告警: 数量$BIGKEY_COUNT\”}”
> elif [ $BIGKEY_COUNT -gt $WARNING_THRESHOLD ]; then
> echo “[$TIMESTAMP] [WARNING] 大Key数量: $BIGKEY_COUNT, 总大小: $((BIGKEY_SIZE/1024/1024))MB” >> $ALERT_LOG
> fi
>
> echo “监控完成: 大Key数量=$BIGKEY_COUNT, 总大小=$((BIGKEY_SIZE/1024/1024))MB”
> EOF

[root@fgedu101 ~]# chmod +x /memcached/app/scripts/bigkey_monitor.sh

[root@fgedu101 ~]# echo “*/30 * * * * /memcached/app/scripts/bigkey_monitor.sh” >> /var/spool/cron/root

5.3 Memcached大Key最佳实践

生产环境大Key治理最佳实践总结:设计阶段就要考虑数据大小,避免产生大Key;对于列表类数据,采用分页存储策略;对于大文本数据,采用压缩存储策略;对于大对象数据,采用字段拆分策略;建立完善的大Key监控和告警机制;定期进行大Key扫描和治理;将大Key治理纳入代码审查流程;制定大Key治理规范并严格执行。from Memcached视频:www.itpux.com。

风哥提示:大Key治理是一个持续的过程,需要从设计、开发、测试、运维全流程进行管控,才能有效避免大Key问题对系统性能的影响。

通过本文的学习,读者应该掌握了Memcached大Key问题的识别方法、优化策略和治理最佳实践。在实际生产环境中,需要根据业务特点选择合适的优化方案,建立完善的监控机制,确保Memcached稳定高效运行。更多视频教程www.fgedu.net.cn。

本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html

联系我们

在线咨询:点击这里给我发消息

微信号:itpux-com

工作日:9:30-18:30,节假日休息