GoldenGate教程FG065-OGG容器化部署(Docker+K8s)运维实战
本文详细介绍OGG容器化部署的完整方案,包括Docker容器化部署和Kubernetes编排管理。风哥教程参考GoldenGate官方文档GoldenGate Docker部署指南、GoldenGate Kubernetes集成手册等内容,帮助读者掌握OGG在容器化环境中的部署、运维和最佳实践。
目录大纲
Part01-基础概念与理论知识
1.1 OGG容器化架构概述
1.2 Docker容器化技术原理
1.3 Kubernetes编排管理原理
Part02-生产环境规划与建议
2.1 OGG容器化环境规划
2.2 Docker环境准备与配置
2.3 Kubernetes集群规划
Part03-生产环境项目实施方案
3.1 OGG Docker镜像制作
3.2 OGG Docker单节点部署
3.3 OGG Kubernetes集群部署
3.4 OGG Kubernetes服务配置
Part04-生产案例与实战讲解
4.1 OGG Docker生产部署案例
4.2 OGG Kubernetes集群运维案例
4.3 OGG容器化故障处理案例
Part05-风哥经验总结与分享
5.1 OGG容器化最佳实践
5.2 OGG容器化常见问题总结
5.3 OGG容器化运维脚本工具
Part01-基础概念与理论知识
1.1 OGG容器化架构概述
OGG容器化部署是将Oracle GoldenGate运行在Docker容器或Kubernetes集群中的技术方案,可以提供更高的灵活性和可管理性,学习交流加群风哥微信: itpux-com。
容器化部署优势:
1. 快速部署:一键部署,分钟级启动
2. 环境一致性:开发、测试、生产环境统一
3. 资源隔离:进程级别资源隔离,避免相互影响
4. 弹性伸缩:根据负载自动扩缩容
5. 易于迁移:环境打包,跨平台迁移简单
6. 高可用保障:Kubernetes提供自动故障恢复
1.2 Docker容器化技术原理
Docker是一种轻量级的容器化技术,通过 namespaces 和 cgroups 实现进程隔离和资源控制。
docker –version
systemctl status docker
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2024-01-15 10:00:00 CST; 5 days ago
Docs: https://docs.docker.com
Main PID: 1234 (dockerd)
Tasks: 15
Memory: 45.2M
CGroup: /system.slice/docker.service
└─1234 /usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock
docker info
Context: default
Debug Mode: false
Server:
Containers: 8
Running: 5
Paused: 0
Stopped: 3
Images: 25
Server Version: 24.0.7
Storage Driver: overlay2
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan ipvlan overlay
Operating System: Oracle Linux Server 9.3
Kernel Version: 5.15.0-generic
1.3 Kubernetes编排管理原理
Kubernetes是Google开源的容器编排平台,提供自动化部署、扩缩容、负载均衡等能力。
kubectl version –short
Kustomize Version: v5.1.1
Server Version: v1.28.3
kubectl get nodes
master01 Ready control-plane 30d v1.28.3
worker01 Ready worker 30d v1.28.3
worker02 Ready worker 30d v1.28.3
worker03 Ready worker 30d v1.28.3
kubectl top nodes
master01 450m 5% 2048Mi 26%
worker01 1230m 15% 8192Mi 52%
worker02 980m 12% 8192Mi 48%
worker03 1100m 13% 8192Mi 55%
Part02-生产环境规划与建议
2.1 OGG容器化环境规划
容器化部署前需要做好充分的环境规划,包括资源评估、网络配置、存储规划等。
lscpu | grep “CPU(s):”
free -h
Mem: 125Gi 35Gi 70Gi 1Gi 20Gi 85Gi
Swap: 16Gi 0B 16Gi
df -h /GoldenGate/fgdata
/dev/sdb1 1.0T 200G 800G 20% /GoldenGate/fgdata
容器化环境规划建议:
1. 资源配置:每个OGG容器建议4核CPU、8GB内存
2. 存储规划:Trail文件、日志、配置文件需要持久化存储
3. 网络配置:容器间通信需要配置专用网络
4. 高可用:Kubernetes部署需要多副本配置
5. 监控告警:集成容器监控和日志收集系统
2.2 Docker环境准备与配置
Docker环境是容器化部署的基础,需要正确配置才能保证OGG正常运行。
mkdir -p /etc/docker
cat > /etc/docker/daemon.json << 'EOF'
{
“storage-driver”: “overlay2”,
“log-driver”: “json-file”,
“log-opts”: {
“max-size”: “100m”,
“max-file”: “3”
},
“data-root”: “/var/lib/docker”,
“exec-opts”: [“native.cgroupdriver=cgroupfs”],
“default-ulimits”: {
“nofile”: {
“Name”: “nofile”,
“Hard”: 65536,
“Soft”: 65536
}
}
}
EOF
systemctl restart docker
docker network create –driver=bridge –subnet=172.20.0.0/16 ogg-network
Network created successfully
docker network ls
abc123… bridge bridge local
def456… host host local
ghi789… none null local
5a7c9d8e… ogg-network bridge local
2.3 Kubernetes集群规划
Kubernetes集群规划需要考虑高可用、性能和可维护性要求。
kubectl get namespaces
default Active 30d
kube-system Active 30d
kube-public Active 30d
kube-node-lease Active 30d
kubectl create namespace ogg-system
cat > ogg-storageclass.yaml << 'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ogg-nfs
provisioner: nfs.io/provisioner
parameters:
archiveOnDelete: “false”
reclaimPolicy: Retain
volumeBindingMode: Immediate
EOF
kubectl apply -f ogg-storageclass.yaml
cat > ogg-pv.yaml << 'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: ogg-pv
labels:
type: ogg-data
spec:
capacity:
storage: 500Gi
accessModes:
– ReadWriteMany
nfs:
server: 192.168.1.100
path: /nfs/ogg/data
—
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ogg-pvc
namespace: ogg-system
spec:
accessModes:
– ReadWriteMany
resources:
requests:
storage: 500Gi
selector:
matchLabels:
type: ogg-data
EOF
kubectl apply -f ogg-pv.yaml
persistentvolumeclaim/ogg-pvc created
kubectl get pvc -n ogg-system
ogg-pvc Bound ogg-pv 500Gi RWX ogg-nfs 5s
Part03-生产环境项目实施方案
3.1 OGG Docker镜像制作
OGG Docker镜像是容器化部署的基础,需要基于Oracle官方镜像或从头构建。
mkdir -p /GoldenGate/docker/ogg
cd /GoldenGate/docker/ogg
cat > Dockerfile << 'EOF'
FROM oraclelinux:8
# 设置环境变量
ENV ORACLE_BASE=/GoldenGate/app \
OGG_HOME=/GoldenGate/app \
PATH=$OGG_HOME:$PATH \
LD_LIBRARY_PATH=$OGG_HOME:$LD_LIBRARY_PATH
# 安装依赖包
RUN dnf -y install \
hostname vi less unzip \
openssl libc-client procps && \
dnf clean all
# 创建目录结构
RUN mkdir -p $OGG_HOME/{dirprm,dirdat,dirrpt,dirsql,dirdef,dirpcs,logs} && \
mkdir -p /GoldenGate/fgdata/{trail,work}
# 复制OGG安装文件
COPY ogg_binaries.tar.gz /tmp/ogg_binaries.tar.gz
RUN tar -xzf /tmp/ogg_binaries.tar.gz -C $OGG_HOME && \
rm -f /tmp/ogg_binaries.tar.gz && \
chmod -R 755 $OGG_HOME
# 复制配置文件
COPY dirprm/* $OGG_HOME/dirprm/
COPY GLOBALS $OGG_HOME/GLOBALS
# 设置工作目录
WORKDIR $OGG_HOME
# 暴露端口
EXPOSE 7809 7808
# 健康检查
HEALTHCHECK –interval=30s –timeout=10s –start-period=60s –retries=3 \
CMD ps aux | grep mgr || exit 1
# 启动命令
CMD [“./ggsci”]
EOF
docker build -t ogg:23ai -f Dockerfile .
Step 1/15 : FROM oraclelinux:8
—> 4e2d6c22a2f1
Step 2/15 : ENV ORACLE_BASE=/GoldenGate/app
—> Using cache
—> abc123def456
…
Step 15/15 : CMD [“./ggsci”]
—> Running in a1b2c3d4e5f6
Removing intermediate container a1b2c3d4e5f6
—> g7h8i9j0k1l2
Successfully built g7h8i9j0k1l2
Successfully tagged ogg:23ai
docker images | grep ogg
ogg 23ai g7h8i9j0k1l2 5 seconds ago 2.5GB
3.2 OGG Docker单节点部署
Docker单节点部署适合开发测试环境或小规模生产环境。
mkdir -p /GoldenGate/docker/data/{dirprm,dirdat,dirrpt,dirsql,dirdef,dirpcs,logs}
chmod -R 777 /GoldenGate/docker/data
cat > /GoldenGate/docker/data/dirprm/MGR.prm << 'EOF'
PORT 7809
— 自动启动Extract和Replicat
AUTOSTART EXTRACT E_FGEDU_*
AUTORESTART EXTRACT *, RETRIES 5, WAITMINUTES 2
AUTORESTART REPLICAT *, RETRIES 5, WAITMINUTES 2
— Trail文件配置
PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 3
— 端口配置
DYNAMICPORTLIST 7810-7820
— 启动GGSCI
STARTUPVALIDATIONTYPE LOG STDOUT
EOF
docker run -d \
–name ogg-manager \
–network ogg-network \
–hostname ogg.fgedu.net.cn \
-p 7809:7809 \
-v /GoldenGate/docker/data/dirprm:/GoldenGate/app/dirprm \
-v /GoldenGate/docker/data/dirdat:/GoldenGate/app/dirdat \
-v /GoldenGate/docker/data/dirrpt:/GoldenGate/app/dirrpt \
-v /GoldenGate/docker/data/logs:/GoldenGate/app/logs \
-e TZ=Asia/Shanghai \
ogg:23ai \
mgr
Container started successfully
docker ps | grep ogg
9a8b7c6d5e4f ogg:23ai “mgr” 5 seconds ago Up 4 seconds 0.0.0.0:7809->7809/tcp ogg-manager
docker exec -it ogg-manager ./ggsci
Version 23.0.0.0.0
Build 001
GGSCI (ogg) 1> INFO ALL
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
GGSCI (ogg) 2> EXIT
docker logs ogg-manager –tail 50
2024-01-15T10:00:00.000000Z INFO OGG Manager started (PID 1234)
2024-01-15T10:00:00.000000Z INFO OGG listening on port 7809
2024-01-15T10:00:00.000000Z INFO OGG Auto-start configured for extract and replicat groups
3.3 OGG Kubernetes集群部署
Kubernetes集群部署适合生产环境,提供高可用和弹性伸缩能力。
cat > ogg-configmap.yaml << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: ogg-config
namespace: ogg-system
data:
MGR.prm: |
PORT 7809
AUTOSTART EXTRACT E_FGEDU_*
AUTORESTART EXTRACT *, RETRIES 5, WAITMINUTES 2
AUTORESTART REPLICAT *, RETRIES 5, WAITMINUTES 2
PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 3
DYNAMICPORTLIST 7810-7820
STARTUPVALIDATIONTYPE LOG STDOUT
EOF
kubectl apply -f ogg-configmap.yaml
cat > ogg-deployment.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: ogg-manager
namespace: ogg-system
labels:
app: ogg
component: manager
spec:
replicas: 1
selector:
matchLabels:
app: ogg
component: manager
template:
metadata:
labels:
app: ogg
component: manager
spec:
containers:
– name: ogg-manager
image: ogg:23ai
ports:
– containerPort: 7809
name: manager
– containerPort: 7808
name: ai
env:
– name: TZ
value: Asia/Shanghai
– name: OGG_HOME
value: /GoldenGate/app
resources:
requests:
memory: “4Gi”
cpu: “2”
limits:
memory: “8Gi”
cpu: “4”
volumeMounts:
– name: ogg-config
mountPath: /GoldenGate/app/dirprm/MGR.prm
subPath: MGR.prm
– name: ogg-data
mountPath: /GoldenGate/app/dirdat
– name: ogg-logs
mountPath: /GoldenGate/app/logs
– name: ogg-reports
mountPath: /GoldenGate/app/dirrpt
livenessProbe:
exec:
command:
– ps
– aux
– grep
– mgr
initialDelaySeconds: 60
periodSeconds: 30
readinessProbe:
exec:
command:
– sh
– -c
– “./ggsci INFO ALL | grep MANAGER”
initialDelaySeconds: 30
periodSeconds: 10
volumes:
– name: ogg-config
configMap:
name: ogg-config
– name: ogg-data
persistentVolumeClaim:
claimName: ogg-pvc
– name: ogg-logs
emptyDir: {}
– name: ogg-reports
emptyDir: {}
EOF
kubectl apply -f ogg-deployment.yaml
kubectl get deployments -n ogg-system
ogg-manager 1/1 1 1 10s
kubectl get pods -n ogg-system
ogg-manager-7d8f9c6b5-x4m7n 1/1 Running 0 30s
3.4 OGG Kubernetes服务配置
Kubernetes Service用于暴露OGG服务,提供负载均衡和服务发现能力。
cat > ogg-service.yaml << 'EOF'
apiVersion: v1
kind: Service
metadata:
name: ogg-manager-svc
namespace: ogg-system
labels:
app: ogg
spec:
type: NodePort
ports:
– name: manager
port: 7809
targetPort: 7809
nodePort: 30001
– name: ai
port: 7808
targetPort: 7808
nodePort: 30008
selector:
app: ogg
component: manager
—
apiVersion: v1
kind: Service
metadata:
name: ogg-manager-cluster
namespace: ogg-system
labels:
app: ogg
spec:
type: ClusterIP
ports:
– name: manager
port: 7809
targetPort: 7809
selector:
app: ogg
component: manager
EOF
kubectl apply -f ogg-service.yaml
service/ogg-manager-cluster created
kubectl get svc -n ogg-system
ogg-manager-svc NodePort 10.96.0.100 7809:30001/TCP,7808:30008/TCP 10s
ogg-manager-cluster ClusterIP 10.96.0.101 7809/TCP 10s
kubectl exec -it ogg-manager-7d8f9c6b5-x4m7n -n ogg-system — ./ggsci
Version 23.0.0.0.0
GGSCI (ogg-manager-7d8f9c6b5-x4m7n) 1> INFO ALL
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING\n
GGSCI (ogg-manager-7d8f9c6b5-x4m7n) 2> EXIT
Part04-生产案例与实战讲解
4.1 OGG Docker生产部署案例
本案例展示一个完整的OGG Docker生产环境部署过程,包括Extract和Replicat配置。
docker run -d \
–name ogg-extract \
–network ogg-network \
–hostname ogg-extract.fgedu.net.cn \
-v /GoldenGate/docker/data/dirprm:/GoldenGate/app/dirprm \
-v /GoldenGate/docker/data/dirdat:/GoldenGate/app/dirdat \
-v /GoldenGate/docker/data/logs:/GoldenGate/app/logs \
-e OGG_PROC=EXTRACT \
ogg:23ai \
extract E_FGEDU01
Container started successfully
docker run -d \
–name ogg-replicat \
–network ogg-network \
–hostname ogg-replicat.fgedu.net.cn \
-v /GoldenGate/docker/data/dirprm:/GoldenGate/app/dirprm \
-v /GoldenGate/docker/data/dirdat:/GoldenGate/app/dirdat \
-v /GoldenGate/docker/data/logs:/GoldenGate/app/logs \
-e OGG_PROC=REPLICAT \
ogg:23ai \
replicat R_FGEDU01
Container started successfully
docker ps –filter “name=ogg” –format “table {{.Names}}\t{{.Status}}\t{{.Ports}}”
ogg-manager Up 10 minutes 0.0.0.0:7809->7809/tcp
ogg-extract Up 2 minutes
ogg-replicat Up 1 minute
4.2 OGG Kubernetes集群运维案例
展示Kubernetes环境下OGG的运维操作,包括扩缩容、故障恢复等。
kubectl scale deployment ogg-extract –replicas=3 -n ogg-system
kubectl get pods -n ogg-system -l component=extract
ogg-extract-7d8f9c6b5-x4m7n 1/1 Running 0 5m
ogg-extract-7d8f9c6b5-y5n8o 1/1 Running 0 30s
ogg-extract-7d8f9c6b5-z6o9p 1/1 Running 0 30s
kubectl delete pod ogg-extract-7d8f9c6b5-x4m7n -n ogg-system
kubectl get pods -n ogg-system -l component=extract -w
ogg-extract-7d8f9c6b5-y5n8o 1/1 Running 0 1m
ogg-extract-7d8f9c6b5-z6o9p 1/1 Running 0 1m
ogg-extract-7d8f9c6b5-x4m7n 1/1 Running 0 30s
Pod ogg-extract-7d8f9c6b5-x4m7n was deleted, replaced by new pod
kubectl set image deployment/ogg-manager ogg-manager=ogg:23ai-v2 -n ogg-system
kubectl rollout status deployment/ogg-manager -n ogg-system
Waiting for deployment “ogg-manager” rollout to finish: 2 out of 3 new replicas have been updated…
deployment “ogg-manager” successfully rolled out
4.3 OGG容器化故障处理案例
容器化环境下的常见故障及处理方法。
docker logs ogg-manager –tail 100 –since 10m
2024-01-15T14:00:00.000000Z ERROR OGG Error: Unable to connect to source database
2024-01-15T14:00:00.000000Z ERROR OGG Retrying in 60 seconds…
2024-01-15T14:01:00.000000Z INFO OGG Connection restored
2024-01-15T14:01:00.000000Z INFO OGG Extract process restarted
docker exec -it ogg-manager /bin/bash
[root@ogg-manager /]# ./ggsci
GGSCI (ogg-manager) 1> INFO EXTRACT E_FGEDU01
EXTRACT E_FGEDU01 Last Started 2024-01-15 14:01:00
Status RUNNING
Lag 00:00:02
Checkpoint 00:00:02
GGSCI (ogg-manager) 2> EXIT
[root@ogg-manager /]# EXIT
exit
kubectl describe pod ogg-manager-7d8f9c6b5-x4m7n -n ogg-system
Namespace: ogg-system
Node: worker01/192.168.1.101
Status: Running
…
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
…
Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal Scheduled 5m default-scheduler Successfully assigned ogg-system/ogg-manager-7d8f9c6b5-x4m7n to worker01
Normal Pulling 5m kubelet Pulling image “ogg:23ai”
Normal Created 5m kubelet Created container ogg-manager
Normal Started 5m kubelet Started container ogg-manager
Part05-风哥经验总结与分享
5.1 OGG容器化最佳实践
基于实际项目经验,总结OGG容器化部署的最佳实践。
最佳实践建议:
1. 使用持久化存储:Trail文件和配置文件必须持久化存储
2. 资源限制:合理设置CPU和内存限制,避免资源争抢
3. 健康检查:配置liveness和readiness探针
4. 日志管理:集成日志收集系统,统一管理日志
5. 镜像优化:使用多阶段构建,减小镜像体积
6. 网络配置:使用专用网络,避免端口冲突
vi /GoldenGate/docker/scripts/ogg_container_check.sh
# ogg_container_check.sh
# from:www.itpux.com.qq113257174.wx:itpux-com
# web: http://www.fgedu.net.cn
echo “========== OGG Container Deployment Checklist ==========”
echo “Check Time: $(date)”
echo “”
# 1. 检查Docker状态
echo “1. Docker Status:”
docker info 2>/dev/null | grep -E “Server Version|Containers|Running” || echo “Docker is not running!”
# 2. 检查OGG容器状态
echo “”
echo “2. OGG Container Status:”
docker ps -a –filter “name=ogg” –format “table {{.Names}}\t{{.Status}}\t{{.Ports}}”
# 3. 检查资源使用
echo “”
echo “3. Container Resource Usage:”
docker stats –no-stream $(docker ps –filter “name=ogg” -q)
# 4. 检查存储卷
echo “”
echo “4. Volume Mounts:”
docker inspect –format='{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}’ $(docker ps –filter “name=ogg” -q)
# 5. 检查网络连接
echo “”
echo “5. Network Status:”
docker network inspect ogg-network 2>/dev/null | grep -E “Name|Containers” || echo “Network not found!”
echo “”
echo “Check completed!”
5.2 OGG容器化常见问题总结
汇总OGG容器化部署和使用过程中的常见问题及解决方案。
常见问题与解决方案:
1. 容器无法启动:检查镜像兼容性和资源限制
2. 进程通信失败:检查网络配置和端口映射
3. 数据丢失:确认持久化存储配置正确
4. 性能下降:调整资源限制和存储驱动
5. 日志无法查看:检查日志驱动配置
6. 镜像拉取失败:配置私有镜像仓库
5.3 OGG容器化运维脚本工具
提供一套完整的OGG容器化运维脚本,方便日常管理和监控。
cat > /GoldenGate/docker/docker-compose.yml << 'EOF'
version: ‘3.8’
services:
ogg-manager:
image: ogg:23ai
container_name: ogg-manager
hostname: ogg-manager.fgedu.net.cn
ports:
– “7809:7809”
volumes:
– ./data/dirprm:/GoldenGate/app/dirprm
– ./data/dirdat:/GoldenGate/app/dirdat
– ./data/logs:/GoldenGate/app/logs
– ./data/dirrpt:/GoldenGate/app/dirrpt
networks:
– ogg-net
restart: unless-stopped
command: mgr
ogg-extract:
image: ogg:23ai
container_name: ogg-extract
hostname: ogg-extract.fgedu.net.cn
depends_on:
– ogg-manager
volumes:
– ./data/dirprm:/GoldenGate/app/dirprm
– ./data/dirdat:/GoldenGate/app/dirdat
– ./data/logs:/GoldenGate/app/logs
networks:
– ogg-net
restart: unless-stopped
command: extract E_FGEDU01
ogg-replicat:
image: ogg:23ai
container_name: ogg-replicat
hostname: ogg-replicat.fgedu.net.cn
depends_on:
– ogg-manager
volumes:
– ./data/dirprm:/GoldenGate/app/dirprm
– ./data/dirdat:/GoldenGate/app/dirdat
– ./data/logs:/GoldenGate/app/logs
networks:
– ogg-net
restart: unless-stopped
command: replicat R_FGEDU01
networks:
ogg-net:
driver: bridge
EOF
cd /GoldenGate/docker
docker-compose up -d
Creating ogg-extract … done
Creating ogg-replicat … done
docker-compose ps
————————————————–
ogg-manager “/bin/bash mgr” ogg-manager running
ogg-extract “/bin/bash … ” ogg-extract running
ogg-replicat “/bin/bash … ” ogg-replicat running
chmod +x /GoldenGate/docker/scripts/ogg_container_check.sh
/GoldenGate/docker/scripts/ogg_container_check.sh
Check Time: Mon Jan 15 10:00:00 CST 2024
1. Docker Status:
Server Version: 24.0.7
Containers: 8
Running: 5
2. OGG Container Status:
NAMES STATUS PORTS
ogg-manager Up 5 minutes 0.0.0.0:7809->7809/tcp
ogg-extract Up 4 minutes
ogg-replicat Up 3 minutes
3. Container Resource Usage:
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O
ogg-manager 0.50% 256MiB / 8GiB 3.12% 1.2MB / 1.5MB
ogg-extract 1.20% 512MiB / 8GiB 6.25% 2.5MB / 3.2MB
ogg-replicat 1.10% 480MiB / 8GiB 5.86% 2.1MB / 2.8MB
Check completed!
本文由风哥教程整理发布,仅用于学习测试使用,转载注明出处:http://www.fgedu.net.cn/10327.html
