k8s Paas实战——K8S集成Apollo配置中心

k8s Paas实战——K8S集成Apollo配置中心

Administrator 795 2022-02-25

之前交付的dubbo和monitor服务，它们的配置都是写死在容器里的，同时配置也存在格式散乱、多副本集下配置修改困难，测试生产环境配置混用等问题，所以就需要引入配置中心来对配置进行管理。

常见的配置中心有：

  • SpringCloudConfig
  • K8S ConfigMap
  • Apollo，基于SpringCloudConfig

configmap使用

为了让镜像和配置文件解耦，以便实现镜像的可移植性和可复用性，因为一个configmap就是一系列配置信息的集合，后续可以直接注入到pod中的容器使用。

使用configmap管理应用配置，需要先拆分环境，拆分为test和pro环境来模拟实际场景。

  1. 将dubbo服务（消费者/服务站/监视器）都scale为0，这样就没有容器在运行了;

  2. 拆分zk，分为测试和生产环境，配置如下：

    主机名角色ip
    hdss7-11.host.comzk1.od.com(test环境)10.4.7.11
    hdss7-12.host.comzk2.od.com(prod环境)10.4.7.12

    之前是3台zk，需要停止11/12/21机器上的zk，并删掉data和log文件：

    [root@hdss7-11 ~]# cd /opt/zookeeper
[root@hdss7-11 zookeeper]# ./bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
[root@hdss7-11 zookeeper]# ps -aux|grep zoo
root      60940  0.0  0.0 112720   964 pts/0    S+   10:21   0:00 grep --color=auto zoo
[root@hdss7-11 zookeeper]# rm -rf /data/zookeeper/data/* /data/zookeeper/logs/*

  3. 修改11/12机器zk的zoo.cfg配置，删除最后3行的集群配置，最终配置如下，然后启动zk：

    tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181

    [root@hdss7-11 zookeeper]# ./bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

# standalone模式
[root@hdss7-11 zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: standalone

  4. 200机器，在/data/k8s-yaml/dubbo-monitor/目录下，准备资源配置清单：

    cm.yaml

    apiVersion: v1
kind: ConfigMap
metadata:
  name: dubbo-monitor-cm
  namespace: infra
data:
  dubbo.properties: |
    dubbo.container=log4j,spring,registry,jetty
    dubbo.application.name=dubbo-monitor
    dubbo.application.owner=evobot.com
    dubbo.registry.address=zookeeper://zk1.od.com:2181
    dubbo.protocol.port=20880
    dubbo.jetty.port=8080
    dubbo.jetty.directory=/dubbo-monitor-simple/monitor
    dubbo.charts.directory=/dubbo-monitor-simple/charts
    dubbo.statistics.directory=/dubbo-monitor-simple/statistics
    dubbo.log4j.file=logs/dubbo-monitor-simple.log
    dubbo.log4j.level=WARN

    dp2.yaml

    kind: Deployment
apiVersion: apps/v1
metadata:
  name: dubbo-monitor
  namespace: infra
  labels:
    name: dubbo-monitor
spec:
  replicas: 1
  selector:
    matchLabels:
      name: dubbo-monitor
  template:
    metadata:
      labels:
        app: dubbo-monitor
        name: dubbo-monitor
    spec:
      containers:
      - name: dubbo-monitor
        image: harbor.od.com/infra/dubbo-monitor:latest
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - name: configmap-volume
            mountPath: /dubbo-monitor-simple/conf
      volumes:
        - name: configmap-volume
          configMap:
            name: dubbo-monitor-cm
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

    可以使用vimdiff dp.yaml dp2.yaml比较两个文件的差异。

  5. 替换原来的dp.yaml，然后在node节点应用配置清单：

    [root@hdss7-200 dubbo-monitor]# mv dp.yaml dp1.yaml
[root@hdss7-200 dubbo-monitor]# mv dp2.yaml dp.yaml

[root@hdss7-21 zookeeper]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/cm.yaml
configmap/dubbo-monitor-cm created
[root@hdss7-21 zookeeper]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/dp.yaml
deployment.apps/dubbo-monitor configured

  6. 查看dubbo-monitor：

  7. 在dashboard的configmap菜单修改dubbo-monitor-cm，将zk改为zk2，然后删除dubbo-monitor的pod让其重启应用新配置：

  8. 新pod启动后，再次访问dubbo-monitor，新配置已经生效：

Apollo交付

Apollo架构：

Meta Server：

  • Portal通过域名访问MetaServer获取AdminService的地址列表
  • Client通过域名访问MetaServer获取ConfigService的地址列表
  • 相当于一个Eureka Proxy
  • 逻辑角色，和ConfigService在一起部署

Eureka：

  • 用于服务发现和注册
  • Config/AdminService注册实例并定期报心跳
  • 和ConfigService在一起部署

其中Eureka和Zookeeper是一样的功能，所以不再搭建重复的功能服务。

Client通过推拉结合和ConfigService交互，然后ConfigService去拿ConfigDB里面的配置给Client端返回去。

Portal是Apollo的一个仪表盘，通过调用AdminService去同步修改ConfigDB里面的配置

先交付ConfigService，然后交付AdminService，最后加portal。

Apollo-ConfigService交付

安装MySQL

  1. 在11机器安装MySQL数据库，可以参考之前的文章；

  2. 修改my.cnf，添加下面的配置：

    [mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
init_connect = "SET NAMES 'utf8mb4'"

[mysql]
default-character-set = utf8mb4

  3. 下载apollo数据库并导入MySQL：

    wget https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql -O apolloconfig.sql

mysql -uroot -p < apolloconfig.sql

  4. 创建数据库用户：

    set global validate_password.policy=LOW;
set global validate_password.length=4;
create user 'apolloconfig'@'10.4.7.%' identified by "123456";
grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to 'apolloconfig'@'10.4.7.%';

  5. 修改初始数据：

    use ApolloConfigDB;
show tables;
select * from ServerConfig \G
update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
select * from ServerConfig \G

  6. 11机器配置域名解析：

    2021121010 ; serial
config             A    10.4.7.10

    2021121011 ; serial
mysql              A    10.4.7.11

    # 22机器上测试域名解析
[root@hdss7-21 conf]# dig -t A config.od.com @192.168.0.2 +short
10.4.7.10

镜像制作

  1. 200机器下载apollo v1.5.1安装包：

    image-20220224164214702

    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-configservice
[root@hdss7-200 src]# unzip -o apollo-configservice-1.5.1-github.zip -d /data/dockerfile/apollo-configservice/
[root@hdss7-200 src]# cd /data/dockerfile/apollo-configservice/
[root@hdss7-200 apollo-configservice]# rm -rf apollo-configservice-1.5.1-sources.jar

  2. 编辑apollo-configserver的账号密码，/data/dockerfile/apollo-configservice/config/application-github.properties文件：

    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456

    [root@hdss7-200 scripts]# pwd
/data/dockerfile/apollo-configservice/scripts
[root@hdss7-200 scripts]# rm shutdown.sh
[root@hdss7-200 scripts]# vi startup.sh

    startup.sh文件清空，改为以下内容：

    #!/bin/bash
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
export JAVA_OPTS="-Xms128m -Xmx128m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

  3. 在/data/dockerfile/apollo-configservice目录下编辑Dockerfile文件，生成镜像：

    FROM 909336740/jre8:8u112
ENV VERSION 1.5.1

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
ADD config/ /apollo-configservice/config
ADD scripts/ /apollo-configservice/scripts

CMD [ "/apollo-configservice/scripts/startup.sh" ]

    [root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1
[root@hdss7-200 apollo-configservice]# docker push !$

服务交付

  1. 在200机器上，创建/data/k8s-yaml/apollo-configservice目录，编辑资源配置清单：

    cm.yaml

    apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-configservice-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
  app.properties: |
    appId=100003171

    dp.yaml

    kind: Deployment
apiVersion: apps/v1
metadata:
  name: apollo-configservice
  namespace: infra
  labels:
    name: apollo-configservice
spec:
  replicas: 1
  selector:
    matchLabels:
      name: apollo-configservice
  template:
    metadata:
      labels:
        app: apollo-configservice
        name: apollo-configservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-configservice-cm
      containers:
      - name: apollo-configservice
        image: harbor.od.com/infra/apollo-configservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-configservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

    svc.yaml

    kind: Service
apiVersion: v1
metadata:
  name: apollo-configservice
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app: apollo-configservice

    ingress.yaml

    kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: apollo-configservice
  namespace: infra
spec:
  rules:
  - host: config.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: apollo-configservice
          servicePort: 8080

  2. 在node节点上用资源配置清单，然后访问config.od.com：image-20220224203945269

  3. 查看数据库连接：

    image-20220224205742947

    image-20220224205846913

    可以看到，apollo-configservice连接到数据库的IP是做了NAT转换的。

Apollo-adminservice交付

镜像制作

  1. 代码解压

    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-adminservice
[root@hdss7-200 src]# unzip -o apollo-adminservice-1.5.1-github.zip -d /data/dockerfile/apollo-adminservice/
[root@hdss7-200 src]# cd /data/dockerfile/apollo-adminservice/
[root@hdss7-200 apollo-adminservice]# rm -rf apollo-adminservice-1.5.1-sources.jar
[root@hdss7-200 apollo-adminservice]# rm -f apollo-adminservice.conf
[root@hdss7-200 apollo-adminservice]# rm -f scripts/shutdown.sh
[root@hdss7-200 apollo-adminservice]# echo '' > scripts/startup.sh

  2. 编辑startup.sh文件，内容如下：

    #!/bin/bash
SERVICE_NAME=apollo-adminservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-admin-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
# SERVER_URL="http://localhost:${SERVER_PORT}"
SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

  3. 修改config下的application-github.properties配置文件：

    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456

  4. 在/data/dockerfile/apollo-adminservice目录下，编辑Dockerfile文件，构建镜像：

    FROM 909336740/jre8:8u112
ENV VERSION 1.5.1

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
ADD config/ /apollo-adminservice/config
ADD scripts/ /apollo-adminservice/scripts

CMD ["/apollo-adminservice/scripts/startup.sh"]

    [root@hdss7-200 apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.5.1
[root@hdss7-200 apollo-adminservice]# docker push !$

服务交付

  1. 在200机器的/data/k8s-yaml/apollo-adminservice目录下，编辑资源配置清单：

    cm.yaml

    apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-adminservice-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
  app.properties: |
    appId=100003172

    dp.yaml

    kind: Deployment
apiVersion: apps/v1
metadata:
  name: apollo-adminservice
  namespace: infra
  labels: 
    name: apollo-adminservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-adminservice
  template:
    metadata:
      labels: 
        app: apollo-adminservice 
        name: apollo-adminservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-adminservice-cm
      containers:
      - name: apollo-adminservice
        image: harbor.od.com/infra/apollo-adminservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-adminservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

  2. 应用资源配置清单，查看config.od.com，已经增加了adminservice：

    image-20220224222829626

    [root@hdss7-22 ~]# curl http://172.7.21.5:8080/info
{"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}

Apollo-Portal交付

数据库初始化

  1. 下载数据库文件;

  2. 导入及配置数据：

    mysql> source ./apolloportaldb.sql;
mysql> show databases;
mysql> use ApolloPortalDB;
mysql> create user "apolloportal"@"10.4.7.%" identified by "123456";
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"10.4.7.%";

mysql> update ServerConfig set Value='[{"orgId":"evobot","orgName":"evobot.cn"},{"orgId":""evobot02","orgName":"www.evobot.cn"},{"orgId":"evobot03","orgName":"evobot-blog"}]' where
e Id=2;

镜像制作

  1. 代码整理，启动脚本startup.sh编辑：

    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-portal
[root@hdss7-200 src]# unzip -o apollo-portal-1.5.1-github.zip -d /data/dockerfile/apollo-portal/

[root@hdss7-200 src]# cd /data/dockerfile/apollo-portal/
[root@hdss7-200 apollo-portal]# rm -f apollo-portal-1.5.1-sources.jar apollo-portal.conf
scripts/shutdown.sh
[root@hdss7-200 apollo-portal]# echo '' > scripts/startup.sh

    startup.sh

    #!/bin/bash
SERVICE_NAME=apollo-portal
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
# SERVER_URL="http://localhost:$SERVER_PORT"
SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null

  2. 修改config/application-github.properties文件：

    # DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username = apolloportal
spring.datasource.password = 123456

  3. 创建Dockerfile文件，构建镜像：

    FROM 909336740/jre8:8u112
ENV VERSION 1.5.1

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts

CMD ["/apollo-portal/scripts/startup.sh"]

    [root@hdss7-200 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.5.1
[root@hdss7-200 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.5.1

服务交付

  1. 在200机器的/data/k8s-yaml/apollo-portal目录下，编辑资源配置清单文件：

    cm.yaml

    apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-portal-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
    spring.datasource.username = apolloportal
    spring.datasource.password = 123456
  app.properties: |
    appId=100003173
  apollo-env.properties: |
    dev.meta=http://config.od.com

    dp.yaml

    kind: Deployment
apiVersion: apps/v1
metadata:
  name: apollo-portal
  namespace: infra
  labels: 
    name: apollo-portal
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-portal
  template:
    metadata:
      labels: 
        app: apollo-portal 
        name: apollo-portal
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-portal-cm
      containers:
      - name: apollo-portal
        image: harbor.od.com/infra/apollo-portal:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-portal/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

    svc.yaml

    kind: Service
apiVersion: v1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: apollo-portal

    ingress.yaml

    kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  rules:
  - host: portal.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-portal
          servicePort: 8080

  2. 添加域名：

    2021121012 ; serial
portal             A    10.4.7.10

  3. 应用资源配置清单，访问portal.od.com，用户名密码为apollo/admin:

    image-20220225103042598

  4. 修改默认密码：

    image-20220225103359420

  5. 查询key及保存key：

    image-20220225104405794

    image-20220225104517487

    image-20220225104629195

  6. 创建项目，填入AppId:dubbo-demo-service并提交：

    image-20220225104753785

    image-20220225104900088

dubbo连接Apollo

服务提供者连接apollo

  1. 在原dubbo-demo-web-master创建apollo分支，然后放入新的代码并上传到仓库：

    上传项目的文件夹打开，然后cd到附近的新建的apollo文件夹，克隆项目并切换身份：
$ git clone http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
$ cd dubbo-demo-web/
$ git branch apollo
$ git checkout apollo

    $ git add .
$ git commit -m "apollo commit#1"
$ git push -u origin apollo

  2. 在Apollo portal里创建两个配置项，与代码dubbo-server/src/main/resources/目录下的config.properties相同：

    [root@hdss7-200 resources]# cat config.properties
dubbo.registry=${dubbo.registry}
dubbo.port=${dubbo.port}

    image-20220225113930619

    image-20220225114013336

  3. 发布，只有发布过的配置才会被客户端获取到，现在发布只会作用于当前DEV环境：

    image-20220225114343127

  4. 进入Jenkins制作镜像：

    # 填入对应参数
app_name: dubbo-demo-service
image_name: app/dubbo-demo-service
git_repo: git@gitee.com:evobot/dubbo-demo-web.git
git_ver: apollo
add_tag: 220225_1145
target_dir: ./dubbo-server/target
base_image: base/jre8:8u112
maven: 3.6.1-8u242

  5. 在200机器上，修改dubbo-demo-service目录下的dp.yaml资源配置清单并在node节点上应用：

            image: harbor.od.com/app/dubbo-demo-service:apollo_220225_1145
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config.od.com
        imagePullPolicy: IfNotPresent

    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/dp.yaml
deployment.apps/dubbo-demo-service configured

    在pod日志中可以看到已经连接了apollo：

    image-20220225141032980

    image-20220225141134766

  6. 修改apollo里的dubbo.port端口号，然后删除dubbo-demo-service的两个pod让其重启应用新的配置：

    image-20220225143208406

    可以看到端口已经变成了20881：

    image-20220225143505689

    也可以更改dubbo.registry的值：

    image-20220225143551977

    修改dubbo-monitor的cm参数为zk2，然后应用cm并删除运行中的pod：

    image-20220225143807081

    image-20220225143849201

    最后删除dubbo-demo-service的运行中的pod，让其运行在zk2上。

服务消费者连接apollo

  1. 推送代码到git仓库后，在apollo创建项目：

    image-20220225145420354

    image-202202251455254752. Jenkins构建dubbo消费者镜像：

    # 填入对应参数
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: git@gitee.com:evobot/dubbo-demo-web.git
git_ver: apollo
add_tag: 220225_1500
target_dir: ./dubbo-client/target
base_image: base/jre8:8u112

    image-20220225145941128

    image-20220225150117792

  2. 200机器上修改dubbo-demo-consumer目录下的dp.yaml资源配置清单文件，然后应用资源配置清单：

     image: harbor.od.com/app/dubbo-demo-consumer:apollo-web_220225_1500
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config.od.com
        imagePullPolicy: IfNotPresent

  3. 完成后访问demo.od.com/hello?name=apollo：

    image-20220225150727682

实现代码迭代

  1. 编辑apollo-web分支下的/opt/src/dubbo-demo-web-master/dubbo-client/src/main/java/com/od/dubbotest/action/HelloAction.java文件，并推送到git仓库：

    public String say(String name) {
                SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss ");
                String ds = df.format(new Date());
                System.out.println(ds + "HelloAction接收到请求:"+name);
                String str = ds + "<h1>这是Dubbo 消费者端(Apollo)</h1><h2>Apollo版本修改 v0.2</h2>";
                str+=helloService.hello(name);
                System.out.println(ds + "HelloService返回到结果:"+str);
                return str;

  2. Jenkins进行镜像构建：

    # 填入对应参数
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: git@gitee.com:evobot/dubbo-demo-web.git
git_ver: apollo
add_tag: 220225_1524
target_dir: ./dubbo-client/target
base_image: base/jre8:8u112

  3. 修改dp.yaml中镜像的tag，然后应用资源配置清单：

    - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-consumer:apollo-web_220225_1524

    image-20220225153806442

Apollo分环境管理

分环境实现生产和测试环境只需要一个打包镜像。

交付apollo-configservice

数据准备

  1. 配置域名解析：

    2021121013 ; serial
zk-test            A    10.4.7.11
zk-prod            A    10.4.7.12

    [root@hdss7-11 ~]# dig -t A zk-test.od.com @10.4.7.11 +short
10.4.7.11
[root@hdss7-11 ~]# dig -t A zk-prod.od.com @10.4.7.11 +short
10.4.7.12

  2. 关闭app命名空间deployment里的消费者和服务者，将scale改为0，然后创建两个命名空间：

    [root@hdss7-21 ~]# kubectl create ns test

[root@hdss7-21 ~]# kubectl create ns prod

[root@hdss7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --doccker-username=admin --docker-password=Harbor12345 -n test

[root@hdss7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n prod

  3. 将infra命名空间内的admin、config、portal的deployment都scale为0；

  4. 到MySQL服务器修改apolloconig.sql，增加Test关键字并导入数据库：

    CREATE DATABASE IF NOT EXISTS ApolloConfigTestDB DEFAULT CHARACTER SET = utf8mb4;

Use ApolloConfigTestDB;

    image-20220225155928952

    mysql> use ApolloConfigTestDB;
mysql> update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.Key="eureka.service.url";

mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%";

  5. 再次修改apolloconfig.sql，将数据库名改为Prod，然后导入数据库：

    mysql> use ApolloConfigProdDB
mysql> update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.Key="eureka.service.url";
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%";

  6. 修改MySQL数据库中ApolloPortalDB：

    mysql> use ApolloPortalDB;
mysql> update ServerConfig set Value='dev,fat,uat,pro' where Id=1;
mysql> select * from ServerConfig\G

  7. 修改apollo-portal的cm.yaml资源配置清单，然后在node节点应用：

      apollo-env.properties: |
    fat.meta=http://config-test.od.com
    pro.meta=http://config-prod.od.com

交付服务

  1. 创建目录，修改资源配置清单：

    [root@hdss7-200 k8s-yaml]# mkdir -pv test/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}

[root@hdss7-200 k8s-yaml]# mkdir -pv prod/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}

[root@hdss7-200 k8s-yaml]# cp /data/k8s-yaml/apollo-configservice/* test/apollo-configservice/

[root@hdss7-200 k8s-yaml]# cp /data/k8s-yaml/apollo-configservice/* prod/apollo-configservice/

    进入test/apollo-configservice目录，修改资源配置清单：

    # cm.yaml修改以下两处
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
    spring.service.url = http://config-test.od.com/eureka

    # 其余yaml文件批量修改namespace为test
sed -i 's/namespace: infra/namespace: test/g' `grep namespace . -rl`

# ingress修改host
sed -i 's/host: config.od.com/host: config-test.od.com/g' ingress.yaml

    进入prod/apollo-configservice目录，修改资源配置清单：

    # cm.yaml修改以下两处
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
    spring.service.url = http://config-prod.od.com/eureka

    sed -i 's/namespace: infra/namespace: prod/g' `grep namespace . -rl`
sed -i 's/host: config.od.com/host: config-prod.od.com/g' ingress.yaml

  2. 解析域名：

    2021121014 ; serial
config-test        A    10.4.7.10
config-prod        A    10.4.7.10

  3. 应用资源配置清单：

    for i in cm dp svc ingress;do kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/$i.yaml;done
for i in cm dp svc ingress;do kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/$i.yaml;done

  4. 访问config-test.od.com和config-prod.od.com

    image-20220225171921071

    image-20220225171947513

交付apollo-portal和adminservice

  1. 复制资源配置清单并进行修改：

    cp -a /data/k8s-yaml/apollo-adminservice/* prod/apollo-adminservice/
cp -a /data/k8s-yaml/apollo-adminservice/* test/apollo-adminservice/

    修改test/apollo-adminservice目录下的资源配置清单：

    # cm.yaml修改以下两处
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
    spring.service.url = http://config-test.od.com/eureka

    # 修改namespace
sed -i 's/namespace: infra/namespace: test/g' `grep namespace . -rl`

    修改prod/apollo-adminservice目录下的资源配置清单：

    # cm.yaml修改以下两处
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
    spring.service.url = http://config-prod.od.com/eureka

    # 修改namespace
sed -i 's/namespace: infra/namespace: prod/g' `grep namespace . -rl`

  2. 应用资源配置清单：

    # shell执行
for i in cm dp;do for s in test prod; do kubectl apply -f http://k8s-yaml.od.com/$s/apollo-adminservice/$i.yaml;done;done

  3. MySQL清除ApolloPortalDB中的数据：

    mysql> use ApolloPortalDB
mysql> select * from App;
mysql> select * from AppNamespace;
mysql> truncate table App;
mysql> truncate table AppNamespace;

  4. 将Portal scale为1：

    image-20220225204236419

    image-20220225204325905

发布dubbo到不同环境

测试环境

  1. Apollo上创建项目：

    image-20220225204455521

  2. 在测试环境新增配置项并发布：

    image-20220225204620283 image-20220225204740752

  3. 生产环境新增配置项并发布：

    image-20220225204934316

    image-20220225205020082

  4. 创建consumer项目，添加测试和生产配置：

    image-20220225205116700

    image-20220225205210283

    image-20220225205310850

  5. 200机器，制作测试环境的service并在node节点应用：

    [root@hdss7-200 ~]# cp -a /data/k8s-yaml/dubbo-demo-service/* /data/k8s-yaml/test/dubbo-demo-service/
[root@hdss7-200 ~]# cd !$
cd /data/k8s-yaml/test/dubbo-demo-service/

    # 修改dp.yaml以下三处
  namespace: test
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config-test.od.com

    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-service/dp.yaml
deployment.apps/dubbo-demo-service created

    image-20220225210040691

  6. 制作测试环境consumer的资源配置清单，然后在node节点应用：

    [root@hdss7-200 test]# cp /data/k8s-yaml/dubbo-demo-consumer/* /data/k8s-yaml/test/dubbo-demo-consumer/
[root@hdss7-200 test]# cd !$
cd /data/k8s-yaml/test/dubbo-demo-consumer/

    # dp.yaml修改以下几处
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://config-test.od.com

    # 批量修改
sed -i 's/namespace: app/namespace: test/g' `grep namespace . -rl`
sed -i 's/host: demo.od.com/host: demo-test.od.com/g' ingress.yaml

  7. 新增域名解析：

    2021121015 ; serial
demo-test          A    10.4.7.10

  8. 应用资源配置清单

    # 批量应用
for i in dp svc ingress; do kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/$i.yaml;done

  9. 修改dubbo-mintor的configmap，监控zk-test，然后删除dubbo-monitor重建pod：

    image-20220225211052378

    image-20220225211206099

    image-20220225211300752

生产环境

  1. 域名配置：

    2021121016 ; serial
demo-prod          A    10.4.7.10

  2. service资源配置清单配置：

    [root@hdss7-200 k8s-yaml]# cp -a /data/k8s-yaml/dubbo-demo-service/* /data/k8s-yaml/prod/dubbo-demo-service/
[root@hdss7-200 k8s-yaml]# cp -a /data/k8s-yaml/dubbo-demo-consumer/* /data/k8s-yaml/prod/dubbo-demo-consumer/

    # prod/dubbo-demo-service/dp.yaml
  namespace: prod
        image: harbor.od.com/app/dubbo-demo-service:apollo_220225_1145
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080
# 这里用了servicename和端口

  3. consumer资源配置清单：

    # prod/dubbo-demo-consumer/dp.yaml
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080

    # 批量修改
sed -i 's/namespace: app/namespace: prod/g' `grep namespace . -rl`
sed -i 's/host: demo.od.com/host: demo-prod.od.com/g' ingress.yaml

    # 批量应用
[root@hdss7-21 ~]# for i in dp svc ingress; do kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/$i.yaml;done

    image-20220225213640183

项目提测发版流程

  1. 修改apollo-web分支下的dubbo-client/src/main/java/com/od/dubbotest/action/HelloAction.java文件，然后提交到远程仓库：

    image-20220225214204480

  2. Jenkins构建：

    # 填入对应参数，然后build
app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: git@gitee.com:evobot/dubbo-demo-web.git
git_ver: 34b84388cd36319767bcc2fdd306269904323ad7
add_tag: 220225_2145
target_dir: ./dubbo-client/target
base_image: base/jre8:8u112

  3. 获取新镜像的tag，然后发布到测试环境，修改test namespace的deployment：

    image-20220225215011512

    image-20220225215105070

  4. 查看生产环境的demo-prod.od.com，版本还未更新：

    image-20220225215153091

  5. 这时候不需要再到Jenkins打包镜像，直接修改deployment的配置即可：

    image-20220225215335731

    image-20220225215400832

    完成。

k8s