k8s Paas实战——CI&CD持续交付

k8s Paas实战——CI&CD持续交付

Administrator 832 2022-02-23

交付Dubbo服务到k8s，Dubbo是阿里巴巴开源的一个高性能优秀的服务框架，使得应用可通过高性能的 RPC 实现服务的输出和输入功能，可以和 Spring框架无缝集成。我们将其当作我们生产中的APP业务，交付到PaaS里。

交付架构图

架构详情

zk（zookeeper)：dubbo服务的注册中心是通过zk来做的，我们用3个zk组成一个集群，就和etcd一样，有一个leader，两个slave，leader出问题的时候由其他来决定谁成为leader，因为zk是有状态的服务，所以将其放在集群外，集群内都是无状态的。

dubbo微服务：在集群内通过图形化操作扩容（dashboard），即当有秒杀或其他需求的时候就可以扩展， 过后则进行缩容。

git：开发将代码上传到git上，这里使用gitee，也可以使用github。

jenkins：用jenkins将git代码拉下来并编译打包成镜像，然后推送到harbor。

OPS服务器（200机器）：将harbor的镜像通过yaml应用到k8s里，现在使用yaml文件，后续使用spinnaker做成图形化的形式。

用户（笑脸）：外部访问通过ingress转发到集群内的dubbo消费者（web服务），然后就可以访问。

最终目标，图形化完成所有配置。

服务器角色分配

主机名角色IP
HDSS7-11.host.comk8s代理节点1，zk110.4.7.11
HDSS7-12.host.comk8s代理节点2，zk210.4.7.12
HDSS7-21.host.comk8s运算节点1，zk310.4.7.21
HDSS7-22.host.comk8s运算节点2，jenkins10.4.7.22
HDSS7-200.host.comk8s运维节点（docker仓库）10.4.7.200

zookeeper部署

zookeeper主要用来解决分布式应用中经常遇到的数据管理问题，如：统一命名服务，状态同步服务，集群管理，分布式应用配置项的管理等，简单来说zookeeper=文件系统+监听通知机制。

dubbo服务要注册到zk里，把配置放在zk上，一旦配置信息发生变化，zk将获取到的最新配置应用到系统中。

java环境安装

  1. 在11/12/21机器上，安装java环境，这里使用jdk8，可以前往官网进行下载：

    [root@hdss7-21 ~]# cd /opt/src
[root@hdss7-21 src]# mkdir -p /opt/src
[root@hdss7-21 src]# mkdir /usr/java
[root@hdss7-21 src]# scp root@hdss7-200:/opt/src/jdk-8u211-linux-x64.tar.gz .
[root@hdss7-21 src]# tar xf jdk-8u211-linux-x64.tar.gz -C /usr/java
[root@hdss7-21 src]# ln -s /usr/java/jdk1.8.0_211 /usr/java/jdk

  2. 添加java环境变量，编辑/etc/profile文件，在文件末尾添加以下内容：

    export JAVA_HOME=/usr/java/jdk
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

    [root@hdss7-11 ~]# source /etc/profile
[root@hdss7-11 ~]# java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

zookeeper安装

  1. 下载解压zookeeper：

    [root@hdss7-12 src]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz

[root@hdss7-12 src]# tar xf zookeeper-3.4.14.tar.gz -C /opt
[root@hdss7-12 src]# cd /opt
[root@hdss7-12 opt]# ln -s /opt/zookeeper-3.4.14 /opt/zookeeper

[root@hdss7-12 opt]# mkdir -pv /data/zookeeper/data /data/zookeeper/logs

  2. 创建并编辑zookeeper配置/opt/zookeeper/conf/zoo.cfg，配置如下：

    tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
server.1=zk1.od.com:2888:3888
server.2=zk2.od.com:2888:3888
server.3=zk3.od.com:2888:3888

  3. 在11机器添加解析，编辑/var/named/od.com.zone，serial前滚一位，最下面添加以下内容：

    zk1                A    10.4.7.11
zk2                A    10.4.7.12
zk3                A    10.4.7.21

    #重启named
[root@hdss7-11 opt]# systemctl restart named
[root@hdss7-11 opt]# dig -t A zk1.od.com @10.4.7.11 +short
10.4.7.11

  4. 在11/12/21机器配置myid，编辑/data/zookeeper/data/myid文件，写入myid，11机器配置1，12配置2，21配置3：

    [root@hdss7-11 opt]# echo '1' > /data/zookeeper/data/myid

[root@hdss7-12 opt]# echo '2' > /data/zookeeper/data/myid

[root@hdss7-21 opt]# echo '3' > /data/zookeeper/data/myid

  5. 启动zookeeper：

    [root@hdss7-11 opt]#  /opt/zookeeper/bin/zkServer.sh start

[root@hdss7-11 opt]# ps aux|grep zoo

[root@hdss7-11 opt]# netstat -tlunp|grep 2181

Jenkins部署

镜像准备

  1. 在200机器拉取jenkins镜像：

    [root@hdss7-200 src]# docker pull jenkins/jenkins:2.263.1

[root@hdss7-200 src]# docker images|grep jenkins

[root@hdss7-200 src]# docker tag 22b8b9a84dbe harbor.od.com/public/jenkins:v2.263.1

[root@hdss7-200 src]# docker push !$

  2. 生成密钥，创建目录：

    [root@hdss7-200 src]# ssh-keygen -t rsa -b 2048 -C "evobot@foxmail.com" -N "" -f /root/.ssh/id_rsa

[root@hdss7-200 src]# mkdir /data/dockerfile
[root@hdss7-200 src]# cd /data/dockerfile/

[root@hdss7-200 dockerfile]# mkdir jenkins
[root@hdss7-200 dockerfile]# cd jenkins/

  3. 创建jenkins的Dockerfile文件，内容如下：

    FROM harbor.od.com/public/jenkins:v2.263.1
USER root
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo 'Asia/Shanghai' > /etc/timezone
ADD id_rsa /root/.ssh/id_rsa
ADD config.json /root/.docker/config.json
ADD get-docker.sh /get-docker.sh
RUN echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config && \
    /get-docker.sh

  4. 准备文件：

    [root@hdss7-200 jenkins]# cp /root/.ssh/id_rsa .
[root@hdss7-200 jenkins]# cp /root/.docker/config.json .
[root@hdss7-200 jenkins]# curl -fsSL get.docker.com -o get-docker.sh
[root@hdss7-200 jenkins]# chmod +x get-docker.sh

gitee仓库配置

  1. 到gitee上新建一个dubbo-demo-web仓库：

  2. 下载dubbo源码包，然后使用git提交到gitee仓库：

    [root@hdss7-200 dubbo-demo-web-master]# git add .
[root@hdss7-200 dubbo-demo-web-master]# git commit -m 'first commit'
[root@hdss7-200 dubbo-demo-web-master]# git remote add origin https://gitee.com/evobot/dubbo-demo-web.git
[root@hdss7-200 dubbo-demo-web-master]# git push -u origin "master"

  3. gitee添加公钥

    [root@hdss7-200 dubbo-demo-web-master]# cat /root/.ssh/id_rsa.pub
ssh-rsa xxxxx evobot@foxmail.com

  4. harbor上新建infra私有仓库：

镜像构建

  1. 在200机器上，开始使用dockerfile构建jenkins的镜像：

    [root@hdss7-200 ~]# cd /data/dockerfile/jenkins/
[root@hdss7-200 jenkins]# docker build . -t harbor.od.com/infra/jenkins:v2.263.1
[root@hdss7-200 jenkins]# docker push harbor.od.com/infra/jenkins:v2.263.1

#测试git仓库的密钥是否有效
[root@hdss7-200 jenkins]# docker run --rm harbor.od.com/infra/jenkins:v2.263.1 ssh -T git@gitee.com

  2. 如果build镜像时提示部分软件包不存在，也可以用下面的Dockerfile进行镜像构建：

    FROM harbor.od.com/public/jenkins:v2.263.1
USER root
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo 'Asia/Shanghai' > /etc/timezone
ADD id_rsa /root/.ssh/id_rsa
ADD config.json /root/.docker/config.json
RUN echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config
RUN apt-get update > /dev/null && apt install -y ca-certificates \
    curl \
    gnupg \
    apt-transport-https \
    lsb-release
RUN DEBIAN_FRONTEND=noninteractive curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
RUN  echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" |  tee /etc/apt/sources.list.d/docker.list
RUN apt-get update > /dev/null && apt-get install -y docker-ce docker-ce-cli containerd.io

命名空间与共享存储

  1. 到21机器，创建命名空间，对应私有化仓库：

    [root@hdss7-21 ~]# kubectl create ns infra
namespace/infra created

[root@hdss7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n infra

    kuberctl create secret创建私有仓库，对应参数分别是仓库地址、用户名、密码、仓库名称infra。

  2. 21/22/200机器配置nfs共享存储

    [root@hdss7-200 jenkins]# yum install nfs-utils -y

# 200机器，做nfs共享存储的客户端
[root@hdss7-200 jenkins]# vi /etc/exports
/data/nfs-volume 10.4.7.0/24(rw,no_root_squash)

    [root@hdss7-200 jenkins]# mkdir /data/nfs-volume
[root@hdss7-200 jenkins]# mkdir /data/nfs-volume/jenkins_home
[root@hdss7-200 jenkins]# systemctl start nfs
[root@hdss7-200 jenkins]# systemctl enable nfs

  3. 进入200机器，创建/data/k8s-yaml/jenkins目录，准备资源配置清单文件，kubectl api-resources |grep deployment可以查看资源配置类型的apiVersion：

    dp.yaml

    kind: Deployment
apiVersion: apps/v1
metadata:
  name: jenkins
  namespace: infra
  labels:
    name: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      name: jenkins
  template:
    metadata:
      labels:
        app: jenkins
        name: jenkins
    spec:
      volumes:
      - name: data
        nfs:
          server: hdss7-200
          path: /data/nfs-volume/jenkins_home
      - name: docker
        hostPath:
          path: /run/docker.sock
          type: ''
      containers:
      - name: jenkins
        image: harbor.od.com/infra/jenkins:v2.263.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
        - name: JAVA_OPTS
          value: -Xmx512m -Xms512m
        volumeMounts:
        - name: data
          mountPath: /var/jenkins_home
        - name: docker
          mountPath: /run/docker.sock
      imagePullSecrets:
      - name: harbor
      securityContext:
        runAsUser: 0
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

    svc.yaml

    kind: Service
apiVersion: v1
metadata:
  name: jenkins
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector:
    app: jenkins

    ingress.yaml

    kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: jenkins
  namespace: infra
spec:
  rules:
  - host: jenkins.od.com
    http:
      paths: 
      - path: /
        backend:
          serviceName: jenkins
          servicePort: 80

  4. 在21机器上应用资源配置清单：

    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/jenkins/dp.yaml

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/jenkins/svc.yaml

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/jenkins/ingress.yaml


    [root@hdss7-21 ~]# kubectl get pods -n infra
NAME                       READY   STATUS    RESTARTS   AGE
jenkins-546fbb867c-992l5   1/1     Running   0          3m27s
[root@hdss7-21 ~]# kubectl get all -n infra
NAME                           READY   STATUS    RESTARTS   AGE
pod/jenkins-546fbb867c-992l5   1/1     Running   0          3m35s

NAME              TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)   AGE
service/jenkins   ClusterIP   192.168.187.251   <none>        80/TCP    62s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jenkins   1/1     1            1           3m35s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/jenkins-546fbb867c   1         1         1       3m35s

  5. 11机器配置域名解析，serial前滚一位并添加下面的内容：

    # 2021121007 ; serial 前滚
jenkins            A    10.4.7.10

    [root@hdss7-11 ~]# systemctl restart named
[root@hdss7-11 ~]# dig -t A jenkins.od.com @10.4.7.11 +short
10.4.7.10

Jenkins配置

访问jenkins.od.com，首次访问需要输入初始密码，可以到200机器的/data/nfs-volume/jenkins_home下，查看secrets/initialAdminPassword文件获取初始密码。

  1. 跳过插件安装，配置管理员账号：

    # jenkins账号密码设置，一定要跟我的一样，后面要用到的：
账号：admin
密码：admin123
full name：admin
# 然后save->save->start using Jenkins即可

  2. 调整安全配置，进入Manage Jenkins>Configure Global Security菜单，调整下面两个安全选项：

    allow anonymous read acces 允许匿名用户访问(勾选)
Prevent cross site request forgery exploits 允许跨域（不勾选）

安装蓝海插件

  1. 蓝海插件可以从仪表板到各个Pipeline运行的查看分支和结果，使用可视编辑器修改Pipeline作为代码，连续交付（CD）Pipeline的复杂可视化，允许快速和直观地了解Pipeline的状态（下面回顾构建镜像流程的时候有使用到），进入Manage Plugins菜单，点击Avaliable，没有内容的话点击check out刷新，然后ctrl+f查找Blue Ocean，勾选之后开始安装：

  2. 安装完成后，在主菜单可以看到blue ocean菜单：

Maven安装

maven是一个项目管理工具，可以对JAVA项目进行构建、依赖管理。可以到官网下载对应版本的maven，这里使用3.6.1版本。

  1. 到dashboard或命令行，进入jenkins容器，查看java版本：

  2. 在200机器的jenkins nfs目录内创建maven目录：

    # 目录名的8u242是根据上图java版本号确定的
[root@hdss7-200 src]# mkdir /data/nfs-volume/jenkins_home/maven-3.6.1-8u242

  3. 在jenkins容器内确认能够登录harbor仓库和git仓库：

    # 进入harbo
docker login harbor.od.com
# 是否能连接gitee
ssh -i /root/.ssh/id_rsa -T git@gitee.com

  4. 解压maven到指定目录：

    [root@hdss7-200 src]# tar zxvf apache-maven-3.6.1-bin.tar.gz -C /data/nfs-volume/jenkins_home/maven-3.6.1-8u242/

[root@hdss7-200 src]# cd !$
cd /data/nfs-volume/jenkins_home/maven-3.6.1-8u242/
[root@hdss7-200 maven-3.6.1-8u242]# ls
apache-maven-3.6.1
[root@hdss7-200 maven-3.6.1-8u242]# mv apache-maven-3.6.1/ ../
[root@hdss7-200 maven-3.6.1-8u242]# mv ../apache-maven-3.6.1/* .
[root@hdss7-200 maven-3.6.1-8u242]# ll
总用量 28
drwxr-xr-x 2 root root     97 2月  22 16:15 bin
drwxr-xr-x 2 root root     42 2月  22 16:15 boot
drwxr-xr-x 3  501 games    63 4月   5 2019 conf
drwxr-xr-x 4  501 games  4096 2月  22 16:15 lib
-rw-r--r-- 1  501 games 13437 4月   5 2019 LICENSE
-rw-r--r-- 1  501 games   182 4月   5 2019 NOTICE
-rw-r--r-- 1  501 games  2533 4月   5 2019 README.txt

  5. 修改conf/settings.xml文件内容，将其中的镜像源改为阿里源：

        <mirror>
      <id>nexus-aliyun</id>
      <mirrorOf>*</mirrorOf>
      <name>Nexus aliyun</name>
      <url>https://maven.aliyun.com/repository/public</url>
      <!-- 旧地址：http://maven.allyun.com/nexus/content/groups/public --> 
    </mirror>

dubbo镜像制作

  1. 到200机器，拉取jre镜像：

    [root@hdss7-200 jenkins_home]# docker pull docker.io/909336740/jre8:8u112

[root@hdss7-200 jenkins_home]# docker images|grep jre

[root@hdss7-200 jenkins_home]# docker tag fa3a085d6ef1 harbor.od.com/public/jre:8u112

[root@hdss7-200 jenkins_home]# docker push !$

  2. 创建/data/dockerfile/jre8目录，创建Dockerfile文件，内容如下：

    FROM harbor.od.com/public/jre:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo 'Asia/Shanghai' > /etc/timezone
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/
WORKDIR /opt/project_dir
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]

  3. 下载和创建所需文件：

    # 下载javaagent-0.3.1.jar
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar

    config.yml

    ---
rules:
  - pattern: '.*'

    entrypoint.sh

    #!/bin/sh

M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=${C_OPTS}
JAR_BALL=${JAR_BALL}
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}

    [root@hdss7-200 jre8]# chmod +x entrypoint.sh
[root@hdss7-200 jre8]# ll
总用量 368
-rw-r--r-- 1 root root     29 2月  22 16:48 config.yml
-rwxr-xr-x 1 root root    236 2月  22 16:48 entrypoint.sh
-rw-r--r-- 1 root root 367417 5月  10 2018 jmx_javaagent-0.3.1.jar

  4. harbor创建base仓库：

  5. 200机器，开始build镜像：

    [root@hdss7-200 jre8]# docker build . -t harbor.od.com/base/jre8:8u112

[root@hdss7-200 jre8]# docker push !$
docker push harbor.od.com/base/jre8:8u112

dubbo持续交付构建

Jenkins pipeline构建

  1. jenkins按照下图顺序创建pipeline：

    构建10个参数：

  2. 把以下内容填写下面的Adnanced Project Options：

    pipeline {
  agent any 
    stages {
      stage('pull') { //get project code from repo 
        steps {
          sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
        }
      }
      stage('build') { //exec mvn cmd
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER}  && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
        }
      }
      stage('package') { //move jar file into project_dir
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
        }
      }
      stage('image') { //build image and push to registry
        steps {
          writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/project_dir"""
          sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
        }
      }
    }
}

    注释（流水线脚本）：

    pull: 把项目克隆到仓库

    build: 到指定的地方创建

    package: 用完mvn后打包到project_dir

    image: 弄到我们的docker仓库

执行构建

  1. harbor创建app项目；

  2. 点击build with Parameters，填入对应的参数：

    填入对应的参数：
app_name:       dubbo-demo-service
image_name:     app/dubbo-demo-service
git_repo:       git@gitee.com:evobot/dubbo-demo-web.git
git_ver:        master
add_tag:        202202_1716
mvn_dir:        ./
target_dir:     ./dubbo-server/target
mvn_cmd:        mvn clean package -Dmaven.test.skip=true
base_image:     base/jre8:8u112
maven:          3.6.1-8u232
# 注意看脚注，点击Build进行构建，等待构建完成。

  3. 执行build：

  4. 查看harbor仓库：

dubbo交付

  1. 在200服务器，创建/data/k8s-yaml/dubbo-demo-service目录，创建对应的资源清单文件：

    dp.yaml

    kind: Deployment
apiVersion: apps/v1
metadata:
  name: dubbo-demo-service
  namespace: app
  labels:
    name: dubbo-demo-service
spec:
  replicas: 1
  selector:
    matchLabels:
      name: dubbo-demo-service
  template:
    metadata:
      labels:
        app: dubbo-demo-service
        name: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:master_202202_2014
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

  2. 21机器创建命名空间和harbor secret：

    [root@hdss7-21 ~]# kubectl create ns app

[root@hdss7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n app

  3. 到11机器查看zookeeper状态：

    [root@hdss7-11 ~]# cd /opt/zookeeper
[root@hdss7-11 zookeeper]# bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Mode: follower

# 连接到zk，里面目前只有zk，还没有dubbo
[root@hdss7-11 zookeeper]# bin/zkCli.sh -server localhost:2181
[zk: localhost:2181(CONNECTED) 1] ls /
[zookeeper]

  4. 22机器，应用dubbo资源配置清单：

    [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/dp.yaml
deployment.apps/dubbo-demo-service created

  5. 查看dubbo的pod：

  6. 到11机器查看zk，已经有了dubbo，此时dubbo服务已经注册到了了ZK交付中心，项目交付成功：

    [zk: localhost:2181(CONNECTED) 2] ls /
[dubbo, zookeeper]
[zk: localhost:2181(CONNECTED) 3]

BlueOcean插件回顾流水线构建

  1. 打开BlueOcean：

  2. 点击构建成功的编号：

  3. 查看具体步骤，jenkins流水线构建就这5步：

    拉取代码——》编译代码——》到指定目录打包jar——》构建镜像——》结束

交付dubbo-monitor

为了方便查看zk注册情况，需要一个图形化界面来查看，也就是dubbo-monitor。

  1. 到200机器，下载dubbo monitor安装包：

    [root@hdss7-200 dubbo-demo-service]# cd /opt/src/
[root@hdss7-200 src]# wget https://github.com/Jeromefromcn/dubbo-monitor/archive/master.zip
[root@hdss7-200 src]# unzip master.zip
[root@hdss7-200 src]# mv dubbo-monitor-master/ /opt/src/dubbo-monitor

  2. 修改dubbo monitor配置，编辑dubbo-monitor/dubbo-monitor-simple/conf/dubbo_origin.properties：

    dubbo.container=log4j,spring,registry,jetty
dubbo.application.name=dubbo-monitor
dubbo.application.owner=evobot
dubbo.registry.address=zookeeper://zk1.od.com:2181?backup=zk2.od.com:2181,zk3.od.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.log4j.file=logs/dubbo-monitor-simple.log
dubbo.log4j.level=WARN

  3. 修改dubbo-monitor/dubbo-monitor-simple/bin/start.sh,修改为下面的内容：

    if [ -n "$BITS" ]; then
    JAVA_MEM_OPTS=" -server -Xmx128m -Xms128m -Xmn32m -XX:PermSize=16m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
else
    JAVA_MEM_OPTS=" -server -Xms128m -Xmx128m -XX:PermSize=16m -XX:SurvivorRatio=2 -XX:+UseParallelGC "
fi

echo -e "Starting the $SERVER_NAME ...\c"
exec java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1

# 原脚本下面的内容全部删除

  4. 200机器，构建镜像：

    [root@hdss7-200 src]# cp -a dubbo-monitor/ /data/dockerfile/
[root@hdss7-200 src]# cd /data/dockerfile/dubbo-monitor/
[root@hdss7-200 dubbo-monitor]# docker build . -t harbor.od.com/infra/dubbo-monitor:latest
[root@hdss7-200 dubbo-monitor]# docker push !$

  5. 200机器，创建/data/k8s-yaml/dubbo-monitor目录，编辑资源清单：

    dp.yaml

    kind: Deployment
apiVersion: apps/v1
metadata:
  name: dubbo-monitor
  namespace: infra
  labels:
    name: dubbo-monitor
spec:
  replicas: 1
  selector:
    matchLabels:
      name: dubbo-monitor
  template:
    metadata:
      labels:
        app: dubbo-monitor
        name: dubbo-monitor
    spec:
      containers:
      - name: dubbo-monitor
        image: harbor.od.com/infra/dubbo-monitor:latest
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

    svc.yaml

    kind: Service
apiVersion: v1
metadata:
  name: dubbo-monitor
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app: dubbo-monitor

    ingress.yaml

    apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dubbo-monitor
  namespace: infra
spec:
  rules:
  - host: dubbo-monitor.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: dubbo-monitor
          servicePort: 8080

  6. 11机器，域名配置，serial前滚，增加dubbo-monitor A 10.4.7.10:

  7. 22机器，应用资源配置清单：

    [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/dp.yaml
deployment.apps/dubbo-monitor created
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/svc.yaml
service/dubbo-monitor created
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/ingress.yaml
ingress.extensions/dubbo-monitor created

  8. 访问dubbo-monitor.od.com，正常访问即交付完成：

交付dubbo服务消费者

  1. 切换到dubbo client分支，登录Jenkins，重新填入构建参数，参数如下：

    # 填入指定参数
app_name:       dubbo-demo-consumer
image_name:     app/dubbo-demo-consumer
git_repo:       git@gitee.com:evobot/dubbo-demo-web.git
git_ver:        client
add_tag:        220223_2016
mvn_dir:        ./
target_dir:     ./dubbo-client/target
mvn_cmd:        mvn clean package -e -q -Dmaven.test.skip=true
base_image:     base/jre8:8u112
maven:          3.6.1-8u232
# 点击Build进行构建，等待构建完成，mvn_cmd 里的 -e -q是让输出输出的多点，可以看里面的内容

  2. 构建成功后，harbor内可以看到新的镜像：

  3. 200机器，创建/data/k8s-yaml/dubbo-demo-consumer准备资源配置清单：

    dp.yaml

    kind: Deployment
apiVersion: apps/v1
metadata:
  name: dubbo-demo-consumer
  namespace: app
  labels:
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      name: dubbo-demo-consumer
  template:
    metadata:
      labels:
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-consumer:client_220223_2016
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

    svc.yaml

    kind: Service
apiVersion: v1
metadata:
  name: dubbo-demo-consumer
  namespace: app
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app: dubbo-demo-consumer

    ingress.yaml

    kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: dubbo-demo-consumer
  namespace: app
spec:
  rules:
  - host: demo.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: dubbo-demo-consumer
          servicePort: 8080

  4. 11机器解析域名：

    # serial前滚一位
2021121009 ; serial
demo               A    10.4.7.10

  5. node节点应用资源清单：

    [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/dp.yaml
deployment.apps/dubbo-demo-consumer created
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/svc.yaml
service/dubbo-demo-consumer created
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/ingress.ya
ml
ingress.extensions/dubbo-demo-consumer created

  6. 刷新dubbo-monitor页面：

  7. 在浏览器访问http://demo.od.com/hello?name=evobot，出现以下页面则交付成功：

  8. dubbo最重要的是能够软负载均衡及扩容，也就是可以随意扩容后端和前端，在dashboard中将dubbo-demo-service deployment的副本增加到3个，把消费者pod改为2个：

  9. 同样的操作也可以随时进行缩容，当访问量高时扩容，访问量少时缩容；

dubbo集群日常维护

在日常维护中，肯定会有代码迭代的情况，下面模拟代码更新时的迭代维护。

  1. 进入dubbo-demo-web的client分支，修改dubbo-client/src/main/java/com/od/dubbotest/action目录下的HelloAction.java文件中的版本号为0.2如下：

    public class HelloAction {

        @Reference HelloService helloService;

        @RequestMapping
        public String say(String name) {
                System.out.println("HelloAction接收到请求:"+name);
                String str="<h1>这是Dubbo 消费者端(springboot)</h1>";
                str+="<h2>版本更新 v0.2</h2>";
                str+=helloService.hello(name);
                System.out.println("HelloService返回到结果:"+str);
                return str;
        }
}

  2. 然后提交更改到仓库：

    [root@hdss7-200 action]# git add .
[root@hdss7-200 action]# git status
# 位于分支 client
# 要提交的变更：
#   （使用 "git reset HEAD <file>..." 撤出暂存区）
#
#       修改：      HelloAction.java
#
[root@hdss7-200 action]# git commit -m 'v0.2'
[client 4ff957e] v0.2
 1 file changed, 1 insertion(+), 1 deletion(-)
[root@hdss7-200 action]# git push

  3. 获取提交的tag：

  4. jenkins中填入构建参数：

    # 填入指定参数,git_repo中的-b client表示clone指定分支
app_name:       dubbo-demo-consumer
image_name:     app/dubbo-demo-consumer
git_repo:       -b client git@gitee.com:evobot/dubbo-demo-web.git
git_ver:        4ff957ed4ae8e5a378ad765dae7d44035b366646
add_tag:        2202023_2125
mvn_dir:        ./
target_dir:     ./dubbo-client/target
mvn_cmd:        mvn clean package -e -q -Dmaven.test.skip=true
base_image:     base/jre8:8u112
maven:          3.6.1-8u232
# 点击Build进行构建，等待构建完成，mvn_cmd 里的 -e -q是让输出输出的多点，可以看里面的内容

  5. build完成后，harbor中可以看到新的镜像：

  6. 然后到dashboard修改dubbo-demo-consumer的yaml文件，将其中的镜像名改为最新的镜像：

  7. 更新之后，再次访问http://demo.od.com/helle?name=evobot，查看版本变化，这样就完成了版本的迭代：

K8S集群毁灭测试

节点宕机

  1. 将consumer和service扩容到2个，查看pod所在节点：

    [root@hdss7-21 ~]# kubectl get pod -n app -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP           NODE                NOMINATED NODE   READINESS GATES
dubbo-demo-consumer-5567b64d55-5zztx   1/1     Running   0          5m52s   172.7.21.5   hdss7-21.host.com   <none>           <none>
dubbo-demo-consumer-5567b64d55-vhwnc   1/1     Running   0          27m     172.7.22.7   hdss7-22.host.com   <none>           <none>
dubbo-demo-service-74d457fc7b-vtpm6    1/1     Running   0          4h38m   172.7.22.5   hdss7-22.host.com   <none>           <none>
dubbo-demo-service-74d457fc7b-w86tr    1/1     Running   0          5m47s   172.7.21.6   hdss7-21.host.com   <none>           <none>

    每个节点上有2个pod

  2. 模拟一个节点服务器宕机，关闭21机器，然后会发现dashboard和demo网址都已经无法访问，接着删除离线主机(不删除的话，k8s会认为是网络抖动等问题，会尝试不断重连节点）：

    [root@hdss7-22 ~]# kubectl get nodes
NAME                STATUS     ROLES    AGE     VERSION
hdss7-21.host.com   NotReady   <none>   5h38m   v1.16.10
hdss7-22.host.com   Ready      <none>   5h42m   v1.16.10
[root@hdss7-22 ~]# kubectl delete nodes hdss7-21.host.com
node "hdss7-21.host.com" deleted

  3. 删除后，k8s的自愈机制会把pod在22节点重新创建，这时候再去访问demo和dashboard就可以正常访问到：

    [root@hdss7-22 ~]# kubectl get pod -n app -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP            NODE                NOMINATED NODE   READINESS GATES
dubbo-demo-consumer-5567b64d55-hxrfj   1/1     Running   0          2m15s   172.7.22.10   hdss7-22.host.com   <none>           <none>
dubbo-demo-consumer-5567b64d55-vhwnc   1/1     Running   0          33m     172.7.22.7    hdss7-22.host.com   <none>           <none>
dubbo-demo-service-74d457fc7b-vbhjg    1/1     Running   0          2m15s   172.7.22.9    hdss7-22.host.com   <none>           <none>
dubbo-demo-service-74d457fc7b-vtpm6    1/1     Running   0          4h44m   172.7.22.5    hdss7-22.host.com   <none>           <none>

  4. 到11和12机器，将nginx中的21节点注释掉，然后重启nginx，这样就完成了服务器节点宕机的应急处理。

集群恢复

  1. 重启21机器，查看服务状态，重新加入node节点：

    [root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 959, uptime 0:00:38
flanneld-7-21                    RUNNING   pid 951, uptime 0:00:38
kube-apiserver-7-21              RUNNING   pid 954, uptime 0:00:38
kube-controller-manager-7-21     RUNNING   pid 956, uptime 0:00:38
kube-kubelet-7-21                RUNNING   pid 950, uptime 0:00:38
kube-proxy-7-21                  RUNNING   pid 957, uptime 0:00:38
kube-scheduler-7-21              RUNNING   pid 953, uptime 0:00:38

[root@hdss7-21 ~]# kubectl label nodes hdss7-21.host.com node-role.kubernetes.io/master=
node/hdss7-21.host.com labeled

[root@hdss7-21 ~]# kubectl label nodes hdss7-21.host.com node-role.kubernetes.io/node=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl get node
NAME                STATUS   ROLES         AGE     VERSION
hdss7-21.host.com   Ready    master,node   72s     v1.16.10
hdss7-22.host.com   Ready    master,node   5h52m   v1.16.10

  2. 修改11/12的nginx，将21的注释取消，查看pod所在机器：

    [root@hdss7-21 ~]# kubectl get pod -n app -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP NODE                NOMINATED NODE   READINESS GATES
dubbo-demo-consumer-5567b64d55-hxrfj   1/1     Running   0          10m     172.7.22.10 hdss7-22.host.com   <none>           <none>
dubbo-demo-consumer-5567b64d55-vhwnc   1/1     Running   0          41m     172.7.22.7 hdss7-22.host.com   <none>           <none>
dubbo-demo-service-74d457fc7b-vbhjg    1/1     Running   0          10m     172.7.22.9 hdss7-22.host.com   <none>           <none>
dubbo-demo-service-74d457fc7b-vtpm6    1/1     Running   0          4h52m   172.7.22.5 hdss7-22.host.com   <none>           <none>

    可以看到pod仍然在22机器上，接下来做一下调度，让资源使用平衡。

  3. 删除一个consumer和service：

    [root@hdss7-21 ~]# kubectl delete pod dubbo-demo-consumer-5567b64d55-hxrfj -n app
pod "dubbo-demo-consumer-5567b64d55-hxrfj" deleted
[root@hdss7-21 ~]# kubectl delete pod dubbo-demo-service-74d457fc7b-vbhjg -n app
pod "dubbo-demo-service-74d457fc7b-vbhjg" deleted

  4. 再查看pod情况，已经分别在两个node上了：

    [root@hdss7-21 ~]# kubectl get pod -n app -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IPNODE                NOMINATED NODE   READINESS GATES
dubbo-demo-consumer-5567b64d55-jcwbx   1/1     Running   0          86s     172.7.21.3hdss7-21.host.com   <none>           <none>
dubbo-demo-consumer-5567b64d55-vhwnc   1/1     Running   0          44m     172.7.22.7hdss7-22.host.com   <none>           <none>
dubbo-demo-service-74d457fc7b-j9lrt    1/1     Running   0          56s     172.7.21.4hdss7-21.host.com   <none>           <none>
dubbo-demo-service-74d457fc7b-vtpm6    1/1     Running   0          4h55m   172.7.22.5hdss7-22.host.com   <none>           <none>

  5. 21机器查看iptables规则：

    [root@hdss7-21 ~]# iptables-save |grep -i postrouting
:POSTROUTING ACCEPT [1:76]
:KUBE-POSTROUTING - [0:0]
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.7.21.2/32 -d 172.7.21.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE

# 确实172.7.0.0/16,删除规则
[root@hdss7-21 ~]# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
# 增加172.7.0.0/16规则
[root@hdss7-21 ~]# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE

[root@hdss7-21 ~]# iptables-save |grep -i postrouting
:POSTROUTING ACCEPT [0:0]
:KUBE-POSTROUTING - [0:0]
-A POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.7.21.2/32 -d 172.7.21.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
k8s