# TiDB关键指标配置

## 1. QPS指标
– 指标名称：tidb_server_qps学习交流加群风哥QQ113257174
– 采集频率：15秒
– 监控目标：所有TiDB实例
– 告警阈值：
– 警告：QPS > 8000
– 严重：QPS > 10000

## 2. 延迟指标
– 指标名称：tidb_server_handle_query_duration_seconds
– 采集频率：15秒
– 监控目标：所有TiDB实例
– 告警阈值：
– 警告：P99 > 0.5秒
– 严重：P99 > 1秒

## 3. 错误率指标
– 指标名称：tidb_server_errors_total
– 采集频率：15秒
– 监控目标：所有TiDB实例
– 告警阈值：
– 警告：错误率 > 0.5%
– 严重：错误率 > 1%

## 4. 连接数指标
– 指标名称：tidb_server_connections
– 采集频率：30秒
– 监控目标：所有TiDB实例
– 告警阈值：
– 警告：连接数 > 800
– 严重：连接数 > 1000

# TiKV关键指标配置

## 1. 存储使用指标
– 指标名称：tikv_store_capacity, tikv_store_available_size
– 采集频率：1分钟
– 监控目标：所有TiKV实例
– 告警阈值：
– 警告：使用率 > 70%
– 严重：使用率 > 85%

## 2. 读写延迟指标
– 指标名称：tikv_server_request_duration_seconds
– 采集频率：15秒
– 监控目标：所有TiKV实例
– 告警阈值：
– 警告：P99 > 0.1秒
– 严重：P99 > 0.2秒

## 3. 并发请求指标
– 指标名称：tikv_server_current_requests
– 采集频率：15秒
– 监控目标：所有TiKV实例
– 告警阈值：
– 警告：并发请求 > 1000
– 严重：并发请求 > 2000

## 4. RocksDB指标
– 指标名称：tikv_rocksdb_write_amplification, tikv_rocksdb_read_amplification
– 采集频率：1分钟
– 监控目标：所有TiKV实例
– 告警阈值：
– 警告：写放大 > 10
– 严重：写放大 > 20

# PD关键指标配置

## 1. 集群状态指标
– 指标名称：pd_cluster_status
– 采集频率：15秒
– 监控目标：所有PD实例
– 告警阈值：
– 严重：状态 != 健康

## 2. 调度指标
– 指标名称：pd_scheduler_schedule_counter
– 采集频率：30秒
– 监控目标：所有PD实例
– 告警阈值：
– 警告：调度失败率 > 10%
– 严重：调度失败率 > 20%

## 3. 存储容量指标
– 指标名称：pd_cluster_total_capacity, pd_cluster_available_capacity
– 采集频率：1分钟
– 监控目标：所有PD实例
– 告警阈值：
– 警告：使用率 > 70%
– 严重：使用率 > 85%

## 4. 性能指标
– 指标名称：pd_server_handle_request_duration_seconds
– 采集频率：30秒
– 监控目标：所有PD实例
– 告警阈值：
– 警告：P99 > 0.1秒
– 严重：P99 > 0.2秒

# 创建告警规则文件
$ mkdir -p /tidb/app/prometheus/alerts
$ cat > /tidb/app/prometheus/alerts/tidb_alerts.yml << EOF groups: - name: tidb rules: - alert: TiDBDown expr: up{job="tidb"} == 0 for: 5m labels: severity: critical annotations: summary: "TiDB down" description: "TiDB instance {{ $labels.instance }} is down for more than 5 minutes" - alert: TiDBHighQPS expr: sum(rate(tidb_server_qps[5m])) by (instance) > 10000
for: 5m
labels:
severity: warning
annotations:
summary: “TiDB high QPS”
description: “TiDB instance {{ $labels.instance }} has high QPS: {{ $value }}”

– alert: TiDBHighErrorRate
expr: sum(rate(tidb_server_errors_total[5m])) by (instance) / sum(rate(tidb_server_qps[5m])) by (instance) > 0.01
for: 5m
labels:
severity: warning
annotations:
summary: “TiDB high error rate”
description: “TiDB instance {{ $labels.instance }} has high error rate: {{ $value }}”

– alert: TiDBHighConnections
expr: tidb_server_connections{job=”tidb”} > 1000
for: 5m
labels:
severity: warning
annotations:
summary: “TiDB high connections”
description: “TiDB instance {{ $labels.instance }} has high connections: {{ $value }}”
EOF

$ cat > /tidb/app/prometheus/alerts/tikv_alerts.yml << EOF groups: - name: tikv rules: - alert: TiKVDown expr: up{job="tikv"} == 0 for: 5m labels: severity: critical annotations: summary: "TiKV down" description: "TiKV instance {{ $labels.instance }} is down for more than 5 minutes" - alert: TiKVHighStorageUsage expr: (1 - tikv_store_available_size / tikv_store_capacity) > 0.85
for: 10m
labels:
severity: critical
annotations:
summary: “TiKV high storage usage”
description: “TiKV instance {{ $labels.instance }} has high storage usage: {{ $value | humanizePercentage }}”

– alert: TiKVHighLatency
expr: histogram_quantile(0.99, sum(rate(tikv_server_request_duration_seconds_bucket[5m])) by (instance, le)) > 0.2
for: 5m
labels:
severity: warning
annotations:
summary: “TiKV high latency”
description: “TiKV instance {{ $labels.instance }} has high latency: {{ $value }}s”

– alert: TiKVHighConcurrency
expr: sum(tikv_server_current_requests) by (instance) > 2000
for: 5m
labels:
severity: warning
annotations:
summary: “TiKV high concurrency”
description: “TiKV instance {{ $labels.instance }} has high concurrency: {{ $value }}”
EOF

$ cat > /tidb/app/prometheus/alerts/pd_alerts.yml << EOF groups: - name: pd rules: - alert: PDDown expr: up{job="pd"} == 0 for: 5m labels: severity: critical annotations: summary: "PD down" description: "PD instance {{ $labels.instance }} is down for more than 5 minutes" - alert: PDClusterUnhealthy expr: pd_cluster_status{type="health"} != 1 for: 5m labels: severity: critical annotations: summary: "PD cluster unhealthy" description: "PD cluster is unhealthy" - alert: PDHighStorageUsage expr: (1 - pd_cluster_available_capacity / pd_cluster_total_capacity) > 0.85
for: 10m
labels:
severity: critical
annotations:
summary: “PD high storage usage”
description: “PD cluster has high storage usage: {{ $value | humanizePercentage }}”

– alert: PDScheduleFailure
expr: rate(pd_scheduler_schedule_counter{type=”fail”}[5m]) / rate(pd_scheduler_schedule_counter{type=”total”}[5m]) > 0.2
for: 5m
labels:
severity: warning
annotations:
summary: “PD schedule failure”
description: “PD schedule failure rate is high: {{ $value | humanizePercentage }}”
EOF

# 重新加载配置
$ curl -X POST http://localhost:9090/-/reload

# 创建AlertManager配置文件
$ cat > /tidb/app/alertmanager/alertmanager.yml << EOF global: resolve_timeout: 5m smtp_smarthost: 'smtp.example.com:587' smtp_from: 'alertmanager@fgedu.net.cn' smtp_auth_username: 'alertmanager' smtp_auth_password: 'password' wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/' wechat_api_corp_id: 'your_corp_id' wechat_api_secret: 'your_secret' route: group_by: ['alertname', 'cluster', 'service'] group_wait: 30s group_interval: 5m repeat_interval: 1h receiver: 'default-receiver' routes: - match: severity: critical receiver: 'critical-receiver' - match: severity: warning receiver: 'warning-receiver' receivers: - name: 'default-receiver' email_configs: - to: 'admin@fgedu.net.cn' send_resolved: true - name: 'critical-receiver' email_configs: - to: 'admin@fgedu.net.cn' send_resolved: true wechat_configs: - corp_id: 'your_corp_id' api_url: 'https://qyapi.weixin.qq.com/cgi-bin/' to_party: '1' agent_id: '1000002' api_secret: 'your_secret' message: '{{ template "wechat.default.message" . }}' - name: 'warning-receiver' email_configs: - to: 'admin@fgedu.net.cn' send_resolved: true wechat_configs: - corp_id: 'your_corp_id' api_url: 'https://qyapi.weixin.qq.com/cgi-bin/' to_party: '1' agent_id: '1000002' api_secret: 'your_secret' message: '{{ template "wechat.default.message" . }}' inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname', 'cluster', 'service'] EOF # 启动AlertManager $ systemctl restart alertmanager # 验证AlertManager状态 $ systemctl status alertmanager # 输出示例 ● alertmanager.service - AlertManager Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2026-04-09 10:00:00 CST; 5min ago Main PID: 12346 (alertmanager) Tasks: 5 Memory: 64.0M CPU: 2% CGroup: /system.slice/alertmanager.service └─12346 /tidb/app/alertmanager/alertmanager --config.file=/tidb/app/alertmanager/alertmanager.yml

# 查看告警状态
http://192.168.1.50:9093

# 告警状态
– firing：正在触发的告警
– resolved：已解决的告警

# 告警分组
– 按alertname分组：相同名称的告警分组
– 按cluster分组：相同集群的告警分组
– 按service分组：相同服务的告警分组

# 告警抑制
– 高级别告警抑制低级别告警
– 同一问题的多个告警只显示一个

# 告警路由
– 根据标签将告警路由到不同的接收器
– 支持多级路由

# 告警通知配置
– 邮件通知：配置SMTP服务器
– 微信通知：配置企业微信应用
– 短信通知：配置短信网关
– 电话通知：配置电话网关
– Slack通知：配置Slack webhook

# 通知模板
$ cat > /tidb/app/alertmanager/templates/default.tmpl << EOF {{ define "wechat.default.message" }} {{ range .Alerts }} 告警级别: {{ .Labels.severity }} 告警类型: {{ .Labels.alertname }} 告警主机: {{ .Labels.instance }} 告警详情: {{ .Annotations.description }} 触发时间: {{ .StartsAt.Format "2006-01-02 15:04:05" }} {{ end }} {{ end }} EOF # 测试告警通知 $ curl -X POST http://localhost:9093/api/v1/alerts -d '[{ "labels": { "alertname": "TestAlert", "severity": "critical", "instance": "test.fgedu.net.cn", "service": "tidb" }, "annotations": { "summary": "Test alert", "description": "This is a test alert" } }]'