k8s Paas实战——K8S集成Apollo配置中心

k8s Paas实战——K8S集成Apollo配置中心

evobot 770 2022-02-25

之前交付的dubbo和monitor服务,它们的配置都是写死在容器里的,同时配置也存在格式散乱、多副本集下配置修改困难,测试生产环境配置混用等问题,所以就需要引入配置中心来对配置进行管理。

常见的配置中心有:

  • SpringCloudConfig
  • K8S ConfigMap
  • Apollo,基于SpringCloudConfig

configmap使用

为了让镜像和配置文件解耦,以便实现镜像的可移植性和可复用性,因为一个configmap就是一系列配置信息的集合,后续可以直接注入到pod中的容器使用。

使用configmap管理应用配置,需要先拆分环境,拆分为test和pro环境来模拟实际场景。

  1. 将dubbo服务(消费者/服务站/监视器)都scale为0,这样就没有容器在运行了;

  2. 拆分zk,分为测试和生产环境,配置如下:

    主机名角色ip
    hdss7-11.host.comzk1.od.com(test环境)10.4.7.11
    hdss7-12.host.comzk2.od.com(prod环境)10.4.7.12

    之前是3台zk,需要停止11/12/21机器上的zk,并删掉data和log文件:

    [root@hdss7-11 ~]# cd /opt/zookeeper
    [root@hdss7-11 zookeeper]# ./bin/zkServer.sh stop
    ZooKeeper JMX enabled by default
    Using config: /opt/zookeeper/bin/../conf/zoo.cfg
    Stopping zookeeper ... STOPPED
    [root@hdss7-11 zookeeper]# ps -aux|grep zoo
    root      60940  0.0  0.0 112720   964 pts/0    S+   10:21   0:00 grep --color=auto zoo
    [root@hdss7-11 zookeeper]# rm -rf /data/zookeeper/data/* /data/zookeeper/logs/*
    
  3. 修改11/12机器zk的zoo.cfg配置,删除最后3行的集群配置,最终配置如下,然后启动zk:

    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/data/zookeeper/data
    dataLogDir=/data/zookeeper/logs
    clientPort=2181
    
    [root@hdss7-11 zookeeper]# ./bin/zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /opt/zookeeper/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    
    # standalone模式
    [root@hdss7-11 zookeeper]# ./bin/zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /opt/zookeeper/bin/../conf/zoo.cfg
    Mode: standalone
    
    
  4. 200机器,在/data/k8s-yaml/dubbo-monitor/目录下,准备资源配置清单:

    cm.yaml

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: dubbo-monitor-cm
      namespace: infra
    data:
      dubbo.properties: |
        dubbo.container=log4j,spring,registry,jetty
        dubbo.application.name=dubbo-monitor
        dubbo.application.owner=evobot.com
        dubbo.registry.address=zookeeper://zk1.od.com:2181
        dubbo.protocol.port=20880
        dubbo.jetty.port=8080
        dubbo.jetty.directory=/dubbo-monitor-simple/monitor
        dubbo.charts.directory=/dubbo-monitor-simple/charts
        dubbo.statistics.directory=/dubbo-monitor-simple/statistics
        dubbo.log4j.file=logs/dubbo-monitor-simple.log
        dubbo.log4j.level=WARN
    

    dp2.yaml

    kind: Deployment
    apiVersion: apps/v1
    metadata:
      name: dubbo-monitor
      namespace: infra
      labels:
        name: dubbo-monitor
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: dubbo-monitor
      template:
        metadata:
          labels:
            app: dubbo-monitor
            name: dubbo-monitor
        spec:
          containers:
          - name: dubbo-monitor
            image: harbor.od.com/infra/dubbo-monitor:latest
            ports:
            - containerPort: 8080
              protocol: TCP
            - containerPort: 20880
              protocol: TCP
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - name: configmap-volume
                mountPath: /dubbo-monitor-simple/conf
          volumes:
            - name: configmap-volume
              configMap:
                name: dubbo-monitor-cm
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext:
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600
    

    可以使用vimdiff dp.yaml dp2.yaml比较两个文件的差异。

  5. 替换原来的dp.yaml,然后在node节点应用配置清单:

    [root@hdss7-200 dubbo-monitor]# mv dp.yaml dp1.yaml
    [root@hdss7-200 dubbo-monitor]# mv dp2.yaml dp.yaml
    
    [root@hdss7-21 zookeeper]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/cm.yaml
    configmap/dubbo-monitor-cm created
    [root@hdss7-21 zookeeper]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/dp.yaml
    deployment.apps/dubbo-monitor configured
    
    
  6. 查看dubbo-monitor:

  7. 在dashboard的configmap菜单修改dubbo-monitor-cm,将zk改为zk2,然后删除dubbo-monitor的pod让其重启应用新配置:

  8. 新pod启动后,再次访问dubbo-monitor,新配置已经生效:

Apollo交付

Apollo架构:

Meta Server:

  • Portal通过域名访问MetaServer获取AdminService的地址列表
  • Client通过域名访问MetaServer获取ConfigService的地址列表
  • 相当于一个Eureka Proxy
  • 逻辑角色,和ConfigService在一起部署

Eureka:

  • 用于服务发现和注册
  • Config/AdminService注册实例并定期报心跳
  • 和ConfigService在一起部署

其中Eureka和Zookeeper是一样的功能,所以不再搭建重复的功能服务。

Client通过推拉结合和ConfigService交互,然后ConfigService去拿ConfigDB里面的配置给Client端返回去。

Portal是Apollo的一个仪表盘,通过调用AdminService去同步修改ConfigDB里面的配置

先交付ConfigService,然后交付AdminService,最后加portal。

Apollo-ConfigService交付

安装MySQL

  1. 在11机器安装MySQL数据库,可以参考之前的文章;

  2. 修改my.cnf,添加下面的配置:

    [mysqld]
    character_set_server = utf8mb4
    collation_server = utf8mb4_general_ci
    init_connect = "SET NAMES 'utf8mb4'"
    
    [mysql]
    default-character-set = utf8mb4
    
  3. 下载apollo数据库并导入MySQL:

    wget https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1/scripts/db/migration/configdb/V1.0.0__initialization.sql -O apolloconfig.sql
    
    mysql -uroot -p < apolloconfig.sql
    
  4. 创建数据库用户:

    set global validate_password.policy=LOW;
    set global validate_password.length=4;
    create user 'apolloconfig'@'10.4.7.%' identified by "123456";
    grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to 'apolloconfig'@'10.4.7.%';
    
  5. 修改初始数据:

    use ApolloConfigDB;
    show tables;
    select * from ServerConfig \G
    update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
    select * from ServerConfig \G
    
  6. 11机器配置域名解析:

    2021121010 ; serial
    config             A    10.4.7.10
    
    2021121011 ; serial
    mysql              A    10.4.7.11
    
    # 22机器上测试域名解析
    [root@hdss7-21 conf]# dig -t A config.od.com @192.168.0.2 +short
    10.4.7.10
    

镜像制作

  1. 200机器下载apollo v1.5.1安装包

    image-20220224164214702

    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-configservice
    [root@hdss7-200 src]# unzip -o apollo-configservice-1.5.1-github.zip -d /data/dockerfile/apollo-configservice/
    [root@hdss7-200 src]# cd /data/dockerfile/apollo-configservice/
    [root@hdss7-200 apollo-configservice]# rm -rf apollo-configservice-1.5.1-sources.jar
    
  2. 编辑apollo-configserver的账号密码,/data/dockerfile/apollo-configservice/config/application-github.properties文件:

    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    
    [root@hdss7-200 scripts]# pwd
    /data/dockerfile/apollo-configservice/scripts
    [root@hdss7-200 scripts]# rm shutdown.sh
    [root@hdss7-200 scripts]# vi startup.sh
    

    startup.sh文件清空,改为以下内容:

    #!/bin/bash
    SERVICE_NAME=apollo-configservice
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-config-server
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
    SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"
    
    ## Adjust memory settings if necessary
    export JAVA_OPTS="-Xms128m -Xmx128m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:SurvivorRatio=8"
    
    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
    
    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
    
    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi
    
    if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi
    
    printf "$(date) ==== Starting ==== \n"
    
    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start
    
    rc=$?;
    
    if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi
    
    tail -f /dev/null
    
  3. /data/dockerfile/apollo-configservice目录下编辑Dockerfile文件,生成镜像:

    FROM 909336740/jre8:8u112
    ENV VERSION 1.5.1
    
    RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
        echo "Asia/Shanghai" > /etc/timezone
    
    ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
    ADD config/ /apollo-configservice/config
    ADD scripts/ /apollo-configservice/scripts
    
    CMD [ "/apollo-configservice/scripts/startup.sh" ]
    
    [root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.5.1
    [root@hdss7-200 apollo-configservice]# docker push !$
    

服务交付

  1. 在200机器上,创建/data/k8s-yaml/apollo-configservice目录,编辑资源配置清单:

    cm.yaml

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apollo-configservice-cm
      namespace: infra
    data:
      application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
        spring.datasource.username = apolloconfig
        spring.datasource.password = 123456
      app.properties: |
        appId=100003171
    

    dp.yaml

    kind: Deployment
    apiVersion: apps/v1
    metadata:
      name: apollo-configservice
      namespace: infra
      labels:
        name: apollo-configservice
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: apollo-configservice
      template:
        metadata:
          labels:
            app: apollo-configservice
            name: apollo-configservice
        spec:
          volumes:
          - name: configmap-volume
            configMap:
              name: apollo-configservice-cm
          containers:
          - name: apollo-configservice
            image: harbor.od.com/infra/apollo-configservice:v1.5.1
            ports:
            - containerPort: 8080
              protocol: TCP
            volumeMounts:
            - name: configmap-volume
              mountPath: /apollo-configservice/config
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext: 
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate: 
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600
    

    svc.yaml

    kind: Service
    apiVersion: v1
    metadata:
      name: apollo-configservice
      namespace: infra
    spec:
      ports:
      - protocol: TCP
        port: 8080
        targetPort: 8080
      selector:
        app: apollo-configservice
    

    ingress.yaml

    kind: Ingress
    apiVersion: extensions/v1beta1
    metadata:
      name: apollo-configservice
      namespace: infra
    spec:
      rules:
      - host: config.od.com
        http:
          paths:
          - path: /
            backend:
              serviceName: apollo-configservice
              servicePort: 8080
    
  2. 在node节点上用资源配置清单,然后访问config.od.com:image-20220224203945269

  3. 查看数据库连接:

    image-20220224205742947

    image-20220224205846913

    可以看到,apollo-configservice连接到数据库的IP是做了NAT转换的。

Apollo-adminservice交付

镜像制作

  1. 代码解压

    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-adminservice
    [root@hdss7-200 src]# unzip -o apollo-adminservice-1.5.1-github.zip -d /data/dockerfile/apollo-adminservice/
    [root@hdss7-200 src]# cd /data/dockerfile/apollo-adminservice/
    [root@hdss7-200 apollo-adminservice]# rm -rf apollo-adminservice-1.5.1-sources.jar
    [root@hdss7-200 apollo-adminservice]# rm -f apollo-adminservice.conf
    [root@hdss7-200 apollo-adminservice]# rm -f scripts/shutdown.sh
    [root@hdss7-200 apollo-adminservice]# echo '' > scripts/startup.sh
    
  2. 编辑startup.sh文件,内容如下:

    #!/bin/bash
    SERVICE_NAME=apollo-adminservice
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-admin-server
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
    # SERVER_URL="http://localhost:${SERVER_PORT}"
    SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"
    
    ## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
    
    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
    
    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
    
    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi
    
    if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi
    
    printf "$(date) ==== Starting ==== \n"
    
    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start
    
    rc=$?;
    
    if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi
    
    tail -f /dev/null
    
  3. 修改config下的application-github.properties配置文件:

    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    
  4. 在/data/dockerfile/apollo-adminservice目录下,编辑Dockerfile文件,构建镜像:

    FROM 909336740/jre8:8u112
    ENV VERSION 1.5.1
    
    RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
        echo "Asia/Shanghai" > /etc/timezone
    
    ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
    ADD config/ /apollo-adminservice/config
    ADD scripts/ /apollo-adminservice/scripts
    
    CMD ["/apollo-adminservice/scripts/startup.sh"]
    
    [root@hdss7-200 apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.5.1
    [root@hdss7-200 apollo-adminservice]# docker push !$
    

服务交付

  1. 在200机器的/data/k8s-yaml/apollo-adminservice目录下,编辑资源配置清单:

    cm.yaml

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apollo-adminservice-cm
      namespace: infra
    data:
      application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
        spring.datasource.username = apolloconfig
        spring.datasource.password = 123456
      app.properties: |
        appId=100003172
    

    dp.yaml

    kind: Deployment
    apiVersion: apps/v1
    metadata:
      name: apollo-adminservice
      namespace: infra
      labels: 
        name: apollo-adminservice
    spec:
      replicas: 1
      selector:
        matchLabels: 
          name: apollo-adminservice
      template:
        metadata:
          labels: 
            app: apollo-adminservice 
            name: apollo-adminservice
        spec:
          volumes:
          - name: configmap-volume
            configMap:
              name: apollo-adminservice-cm
          containers:
          - name: apollo-adminservice
            image: harbor.od.com/infra/apollo-adminservice:v1.5.1
            ports:
            - containerPort: 8080
              protocol: TCP
            volumeMounts:
            - name: configmap-volume
              mountPath: /apollo-adminservice/config
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext: 
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate: 
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600
    
  2. 应用资源配置清单,查看config.od.com,已经增加了adminservice:

    image-20220224222829626

    [root@hdss7-22 ~]# curl http://172.7.21.5:8080/info
    {"git":{"commit":{"time":{"seconds":1573275854,"nanos":0},"id":"c9eae54"},"branch":"1.5.1"}}
    

Apollo-Portal交付

数据库初始化

  1. 下载数据库文件;

  2. 导入及配置数据:

    mysql> source ./apolloportaldb.sql;
    mysql> show databases;
    mysql> use ApolloPortalDB;
    mysql> create user "apolloportal"@"10.4.7.%" identified by "123456";
    mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"10.4.7.%";
    
    mysql> update ServerConfig set Value='[{"orgId":"evobot","orgName":"evobot.cn"},{"orgId":""evobot02","orgName":"www.evobot.cn"},{"orgId":"evobot03","orgName":"evobot-blog"}]' where
    e Id=2;
    

镜像制作

  1. 代码整理,启动脚本startup.sh编辑:

    [root@hdss7-200 src]# mkdir /data/dockerfile/apollo-portal
    [root@hdss7-200 src]# unzip -o apollo-portal-1.5.1-github.zip -d /data/dockerfile/apollo-portal/
    
    [root@hdss7-200 src]# cd /data/dockerfile/apollo-portal/
    [root@hdss7-200 apollo-portal]# rm -f apollo-portal-1.5.1-sources.jar apollo-portal.conf
    scripts/shutdown.sh
    [root@hdss7-200 apollo-portal]# echo '' > scripts/startup.sh
    

    startup.sh

    #!/bin/bash
    SERVICE_NAME=apollo-portal
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-portal-server
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
    # SERVER_URL="http://localhost:$SERVER_PORT"
    SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"
    
    ## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
    
    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
    
    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"
    
    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi
    
    if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi
    
    printf "$(date) ==== Starting ==== \n"
    
    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start
    
    rc=$?;
    
    if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi
    
    tail -f /dev/null
    
  2. 修改config/application-github.properties文件:

    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
    spring.datasource.username = apolloportal
    spring.datasource.password = 123456
    
  3. 创建Dockerfile文件,构建镜像:

    FROM 909336740/jre8:8u112
    ENV VERSION 1.5.1
    
    RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
        echo "Asia/Shanghai" > /etc/timezone
    
    ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
    ADD config/ /apollo-portal/config
    ADD scripts/ /apollo-portal/scripts
    
    CMD ["/apollo-portal/scripts/startup.sh"]
    
    [root@hdss7-200 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.5.1
    [root@hdss7-200 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.5.1
    

服务交付

  1. 在200机器的/data/k8s-yaml/apollo-portal目录下,编辑资源配置清单文件:

    cm.yaml

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apollo-portal-cm
      namespace: infra
    data:
      application-github.properties: |
        # DataSource
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
        spring.datasource.username = apolloportal
        spring.datasource.password = 123456
      app.properties: |
        appId=100003173
      apollo-env.properties: |
        dev.meta=http://config.od.com
    

    dp.yaml

    kind: Deployment
    apiVersion: apps/v1
    metadata:
      name: apollo-portal
      namespace: infra
      labels: 
        name: apollo-portal
    spec:
      replicas: 1
      selector:
        matchLabels: 
          name: apollo-portal
      template:
        metadata:
          labels: 
            app: apollo-portal 
            name: apollo-portal
        spec:
          volumes:
          - name: configmap-volume
            configMap:
              name: apollo-portal-cm
          containers:
          - name: apollo-portal
            image: harbor.od.com/infra/apollo-portal:v1.5.1
            ports:
            - containerPort: 8080
              protocol: TCP
            volumeMounts:
            - name: configmap-volume
              mountPath: /apollo-portal/config
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
          imagePullSecrets:
          - name: harbor
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          securityContext: 
            runAsUser: 0
          schedulerName: default-scheduler
      strategy:
        type: RollingUpdate
        rollingUpdate: 
          maxUnavailable: 1
          maxSurge: 1
      revisionHistoryLimit: 7
      progressDeadlineSeconds: 600
    

    svc.yaml

    kind: Service
    apiVersion: v1
    metadata: 
      name: apollo-portal
      namespace: infra
    spec:
      ports:
      - protocol: TCP
        port: 8080
        targetPort: 8080
      selector: 
        app: apollo-portal
    

    ingress.yaml

    kind: Ingress
    apiVersion: extensions/v1beta1
    metadata: 
      name: apollo-portal
      namespace: infra
    spec:
      rules:
      - host: portal.od.com
        http:
          paths:
          - path: /
            backend: 
              serviceName: apollo-portal
              servicePort: 8080
    
  2. 添加域名:

    2021121012 ; serial
    portal             A    10.4.7.10
    
  3. 应用资源配置清单,访问portal.od.com,用户名密码为apollo/admin:

    image-20220225103042598

  4. 修改默认密码:

    image-20220225103359420

  5. 查询key及保存key:

    image-20220225104405794

    image-20220225104517487

    image-20220225104629195

  6. 创建项目,填入AppId:dubbo-demo-service并提交:

    image-20220225104753785

    image-20220225104900088

dubbo连接Apollo

服务提供者连接apollo

  1. 在原dubbo-demo-web-master创建apollo分支,然后放入新的代码并上传到仓库:

    上传项目的文件夹打开,然后cd到附近的新建的apollo文件夹,克隆项目并切换身份:
    $ git clone http://gitlab.od.com:10000/909336740/dubbo-demo-web.git
    $ cd dubbo-demo-web/
    $ git branch apollo
    $ git checkout apollo
    
    $ git add .
    $ git commit -m "apollo commit#1"
    $ git push -u origin apollo
    
  2. 在Apollo portal里创建两个配置项,与代码dubbo-server/src/main/resources/目录下的config.properties相同:

    [root@hdss7-200 resources]# cat config.properties
    dubbo.registry=${dubbo.registry}
    dubbo.port=${dubbo.port}
    

    image-20220225113930619

    image-20220225114013336

  3. 发布,只有发布过的配置才会被客户端获取到,现在发布只会作用于当前DEV环境:

    image-20220225114343127

  4. 进入Jenkins制作镜像:

    # 填入对应参数
    app_name: dubbo-demo-service
    image_name: app/dubbo-demo-service
    git_repo: git@gitee.com:evobot/dubbo-demo-web.git
    git_ver: apollo
    add_tag: 220225_1145
    target_dir: ./dubbo-server/target
    base_image: base/jre8:8u112
    maven: 3.6.1-8u242
    
  5. 在200机器上,修改dubbo-demo-service目录下的dp.yaml资源配置清单并在node节点上应用:

            image: harbor.od.com/app/dubbo-demo-service:apollo_220225_1145
            ports:
            - containerPort: 20880
              protocol: TCP
            env:
            - name: JAR_BALL
              value: dubbo-server.jar
            - name: C_OPTS
              value: -Denv=dev -Dapollo.meta=http://config.od.com
            imagePullPolicy: IfNotPresent
    
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/dp.yaml
    deployment.apps/dubbo-demo-service configured
    

    在pod日志中可以看到已经连接了apollo:

    image-20220225141032980

    image-20220225141134766

  6. 修改apollo里的dubbo.port端口号,然后删除dubbo-demo-service的两个pod让其重启应用新的配置:

    image-20220225143208406

    可以看到端口已经变成了20881:

    image-20220225143505689

    也可以更改dubbo.registry的值:

    image-20220225143551977

    修改dubbo-monitor的cm参数为zk2,然后应用cm并删除运行中的pod:

    image-20220225143807081

    image-20220225143849201

    最后删除dubbo-demo-service的运行中的pod,让其运行在zk2上。

服务消费者连接apollo

  1. 推送代码到git仓库后,在apollo创建项目:

    image-20220225145420354

    image-202202251455254752. Jenkins构建dubbo消费者镜像:

    # 填入对应参数
    app_name: dubbo-demo-consumer
    image_name: app/dubbo-demo-consumer
    git_repo: git@gitee.com:evobot/dubbo-demo-web.git
    git_ver: apollo
    add_tag: 220225_1500
    target_dir: ./dubbo-client/target
    base_image: base/jre8:8u112
    

    image-20220225145941128

    image-20220225150117792

  2. 200机器上修改dubbo-demo-consumer目录下的dp.yaml资源配置清单文件,然后应用资源配置清单:

     image: harbor.od.com/app/dubbo-demo-consumer:apollo-web_220225_1500
            ports:
            - containerPort: 8080
              protocol: TCP
            - containerPort: 20880
              protocol: TCP
            env:
            - name: JAR_BALL
              value: dubbo-client.jar
            - name: C_OPTS
              value: -Denv=dev -Dapollo.meta=http://config.od.com
            imagePullPolicy: IfNotPresent
    
  3. 完成后访问demo.od.com/hello?name=apollo:

    image-20220225150727682

实现代码迭代

  1. 编辑apollo-web分支下的/opt/src/dubbo-demo-web-master/dubbo-client/src/main/java/com/od/dubbotest/action/HelloAction.java文件,并推送到git仓库:

    public String say(String name) {
                    SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss ");
                    String ds = df.format(new Date());
                    System.out.println(ds + "HelloAction接收到请求:"+name);
                    String str = ds + "<h1>这是Dubbo 消费者端(Apollo)</h1><h2>Apollo版本修改 v0.2</h2>";
                    str+=helloService.hello(name);
                    System.out.println(ds + "HelloService返回到结果:"+str);
                    return str;
    
  2. Jenkins进行镜像构建:

    # 填入对应参数
    app_name: dubbo-demo-consumer
    image_name: app/dubbo-demo-consumer
    git_repo: git@gitee.com:evobot/dubbo-demo-web.git
    git_ver: apollo
    add_tag: 220225_1524
    target_dir: ./dubbo-client/target
    base_image: base/jre8:8u112
    
  3. 修改dp.yaml中镜像的tag,然后应用资源配置清单:

    - name: dubbo-demo-consumer
            image: harbor.od.com/app/dubbo-demo-consumer:apollo-web_220225_1524
    

    image-20220225153806442

Apollo分环境管理

分环境实现生产和测试环境只需要一个打包镜像。

交付apollo-configservice

数据准备

  1. 配置域名解析:

    2021121013 ; serial
    zk-test            A    10.4.7.11
    zk-prod            A    10.4.7.12
    
    [root@hdss7-11 ~]# dig -t A zk-test.od.com @10.4.7.11 +short
    10.4.7.11
    [root@hdss7-11 ~]# dig -t A zk-prod.od.com @10.4.7.11 +short
    10.4.7.12
    
  2. 关闭app命名空间deployment里的消费者和服务者,将scale改为0,然后创建两个命名空间:

    [root@hdss7-21 ~]# kubectl create ns test
    
    [root@hdss7-21 ~]# kubectl create ns prod
    
    [root@hdss7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --doccker-username=admin --docker-password=Harbor12345 -n test
    
    [root@hdss7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n prod
    
    
  3. 将infra命名空间内的admin、config、portal的deployment都scale为0;

  4. 到MySQL服务器修改apolloconig.sql,增加Test关键字并导入数据库:

    CREATE DATABASE IF NOT EXISTS ApolloConfigTestDB DEFAULT CHARACTER SET = utf8mb4;
    
    Use ApolloConfigTestDB;
    

    image-20220225155928952

    mysql> use ApolloConfigTestDB;
    mysql> update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.Key="eureka.service.url";
    
    mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%";
    
  5. 再次修改apolloconfig.sql,将数据库名改为Prod,然后导入数据库:

    mysql> use ApolloConfigProdDB
    mysql> update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.Key="eureka.service.url";
    mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%";
    
  6. 修改MySQL数据库中ApolloPortalDB:

    mysql> use ApolloPortalDB;
    mysql> update ServerConfig set Value='dev,fat,uat,pro' where Id=1;
    mysql> select * from ServerConfig\G
    
  7. 修改apollo-portal的cm.yaml资源配置清单,然后在node节点应用:

      apollo-env.properties: |
        fat.meta=http://config-test.od.com
        pro.meta=http://config-prod.od.com
    

交付服务

  1. 创建目录,修改资源配置清单:

    [root@hdss7-200 k8s-yaml]# mkdir -pv test/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
    
    [root@hdss7-200 k8s-yaml]# mkdir -pv prod/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
    
    [root@hdss7-200 k8s-yaml]# cp /data/k8s-yaml/apollo-configservice/* test/apollo-configservice/
    
    [root@hdss7-200 k8s-yaml]# cp /data/k8s-yaml/apollo-configservice/* prod/apollo-configservice/
    
    

    进入test/apollo-configservice目录,修改资源配置清单:

    # cm.yaml修改以下两处
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
        spring.service.url = http://config-test.od.com/eureka
    
    # 其余yaml文件批量修改namespace为test
    sed -i 's/namespace: infra/namespace: test/g' `grep namespace . -rl`
    
    # ingress修改host
    sed -i 's/host: config.od.com/host: config-test.od.com/g' ingress.yaml
    

    进入prod/apollo-configservice目录,修改资源配置清单:

    # cm.yaml修改以下两处
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
        spring.service.url = http://config-prod.od.com/eureka
    
    sed -i 's/namespace: infra/namespace: prod/g' `grep namespace . -rl`
    sed -i 's/host: config.od.com/host: config-prod.od.com/g' ingress.yaml
    
  2. 解析域名:

    2021121014 ; serial
    config-test        A    10.4.7.10
    config-prod        A    10.4.7.10
    
  3. 应用资源配置清单:

    for i in cm dp svc ingress;do kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/$i.yaml;done
    for i in cm dp svc ingress;do kubectl apply -f http://k8s-yaml.od.com/test/apollo-configservice/$i.yaml;done
    
  4. 访问config-test.od.com和config-prod.od.com

    image-20220225171921071

    image-20220225171947513

交付apollo-portal和adminservice

  1. 复制资源配置清单并进行修改:

    cp -a /data/k8s-yaml/apollo-adminservice/* prod/apollo-adminservice/
    cp -a /data/k8s-yaml/apollo-adminservice/* test/apollo-adminservice/
    

    修改test/apollo-adminservice目录下的资源配置清单:

    # cm.yaml修改以下两处
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
        spring.service.url = http://config-test.od.com/eureka
    
    # 修改namespace
    sed -i 's/namespace: infra/namespace: test/g' `grep namespace . -rl`
    

    修改prod/apollo-adminservice目录下的资源配置清单:

    # cm.yaml修改以下两处
        spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
        spring.service.url = http://config-prod.od.com/eureka
    
    # 修改namespace
    sed -i 's/namespace: infra/namespace: prod/g' `grep namespace . -rl`
    
  2. 应用资源配置清单:

    # shell执行
    for i in cm dp;do for s in test prod; do kubectl apply -f http://k8s-yaml.od.com/$s/apollo-adminservice/$i.yaml;done;done
    
  3. MySQL清除ApolloPortalDB中的数据:

    mysql> use ApolloPortalDB
    mysql> select * from App;
    mysql> select * from AppNamespace;
    mysql> truncate table App;
    mysql> truncate table AppNamespace;
    
  4. 将Portal scale为1:

    image-20220225204236419

    image-20220225204325905

发布dubbo到不同环境

测试环境

  1. Apollo上创建项目:

    image-20220225204455521

  2. 在测试环境新增配置项并发布:

    image-20220225204620283 image-20220225204740752
  3. 生产环境新增配置项并发布:

    image-20220225204934316

    image-20220225205020082

  4. 创建consumer项目,添加测试和生产配置:

    image-20220225205116700

    image-20220225205210283

    image-20220225205310850

  5. 200机器,制作测试环境的service并在node节点应用:

    [root@hdss7-200 ~]# cp -a /data/k8s-yaml/dubbo-demo-service/* /data/k8s-yaml/test/dubbo-demo-service/
    [root@hdss7-200 ~]# cd !$
    cd /data/k8s-yaml/test/dubbo-demo-service/
    
    # 修改dp.yaml以下三处
      namespace: test
            env:
            - name: JAR_BALL
              value: dubbo-server.jar
            - name: C_OPTS
              value: -Denv=dev -Dapollo.meta=http://config-test.od.com
    
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-service/dp.yaml
    deployment.apps/dubbo-demo-service created
    

    image-20220225210040691

  6. 制作测试环境consumer的资源配置清单,然后在node节点应用:

    [root@hdss7-200 test]# cp /data/k8s-yaml/dubbo-demo-consumer/* /data/k8s-yaml/test/dubbo-demo-consumer/
    [root@hdss7-200 test]# cd !$
    cd /data/k8s-yaml/test/dubbo-demo-consumer/
    
    # dp.yaml修改以下几处
            env:
            - name: JAR_BALL
              value: dubbo-client.jar
            - name: C_OPTS
              value: -Denv=fat -Dapollo.meta=http://config-test.od.com
    
    # 批量修改
    sed -i 's/namespace: app/namespace: test/g' `grep namespace . -rl`
    sed -i 's/host: demo.od.com/host: demo-test.od.com/g' ingress.yaml
    
  7. 新增域名解析:

    2021121015 ; serial
    demo-test          A    10.4.7.10
    
  8. 应用资源配置清单

    # 批量应用
    for i in dp svc ingress; do kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/$i.yaml;done
    
  9. 修改dubbo-mintor的configmap,监控zk-test,然后删除dubbo-monitor重建pod:

    image-20220225211052378

    image-20220225211206099

    image-20220225211300752

生产环境

  1. 域名配置:

    2021121016 ; serial
    demo-prod          A    10.4.7.10
    
  2. service资源配置清单配置:

    [root@hdss7-200 k8s-yaml]# cp -a /data/k8s-yaml/dubbo-demo-service/* /data/k8s-yaml/prod/dubbo-demo-service/
    [root@hdss7-200 k8s-yaml]# cp -a /data/k8s-yaml/dubbo-demo-consumer/* /data/k8s-yaml/prod/dubbo-demo-consumer/
    
    # prod/dubbo-demo-service/dp.yaml
      namespace: prod
            image: harbor.od.com/app/dubbo-demo-service:apollo_220225_1145
            ports:
            - containerPort: 20880
              protocol: TCP
            env:
            - name: JAR_BALL
              value: dubbo-server.jar
            - name: C_OPTS
              value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080
    # 这里用了servicename和端口
    
  3. consumer资源配置清单:

    # prod/dubbo-demo-consumer/dp.yaml
            env:
            - name: JAR_BALL
              value: dubbo-client.jar
            - name: C_OPTS
              value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080
    
    # 批量修改
    sed -i 's/namespace: app/namespace: prod/g' `grep namespace . -rl`
    sed -i 's/host: demo.od.com/host: demo-prod.od.com/g' ingress.yaml
    
    # 批量应用
    [root@hdss7-21 ~]# for i in dp svc ingress; do kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/$i.yaml;done
    

    image-20220225213640183

项目提测发版流程

  1. 修改apollo-web分支下的dubbo-client/src/main/java/com/od/dubbotest/action/HelloAction.java文件,然后提交到远程仓库:

    image-20220225214204480

  2. Jenkins构建:

    # 填入对应参数,然后build
    app_name: dubbo-demo-consumer
    image_name: app/dubbo-demo-consumer
    git_repo: git@gitee.com:evobot/dubbo-demo-web.git
    git_ver: 34b84388cd36319767bcc2fdd306269904323ad7
    add_tag: 220225_2145
    target_dir: ./dubbo-client/target
    base_image: base/jre8:8u112
    
  3. 获取新镜像的tag,然后发布到测试环境,修改test namespace的deployment:

    image-20220225215011512

    image-20220225215105070

  4. 查看生产环境的demo-prod.od.com,版本还未更新:

    image-20220225215153091

  5. 这时候不需要再到Jenkins打包镜像,直接修改deployment的配置即可:

    image-20220225215335731

    image-20220225215400832

    完成。



# k8s