k8s Paas实战——dashboard部署及实战交付

k8s Paas实战——dashboard部署及实战交付

Administrator 841 2022-02-18

K8S的交付服务的形式交付,流程是:准备镜像->准备清单->解析域名->应用配置清单->完成

Dashboard安装部署

dashboard是向企业展示度量信息和关键业务指标现状的数据虚拟化工具,我们之前的资源部署都是登录到机器上执行命令,是非常不安全的方式,并且实际应用中,开发人员也需要看到pod的情况,不能让他们也登录主机去查看,所以需要一个有权限控制的界面展示和控制的工具。

部署步骤

  1. 在200机器上准备镜像:

    [root@hdss7-200 traefik]# cd /data/k8s-yaml/
    [root@hdss7-200 k8s-yaml]# docker pull k8scn/kubernetes-dashboard-amd64:v1.8.3
    
    [root@hdss7-200 k8s-yaml]# docker images |grep dashboard
    
    [root@hdss7-200 k8s-yaml]# docker tag fcac9aa03fd6 harbor.od.com/public/dashboard:v1.8.3
    
    [root@hdss7-200 k8s-yaml]# docker push !$
    
    
  2. 创建/data/k8s-yaml/dashboard目录,创建资源配置清单文件:

    rbac.yaml

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
        addonmanager.kubernetes.io/mode: Reconcile
      name: kubernetes-dashboard-admin
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: kubernetes-dashboard-admin
      namespace: kube-system
      labels:
        k8s-app: kubernetes-dashboard
        addonmanager.kubernetes.io/mode: Reconcile
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
    - kind: ServiceAccount
      name: kubernetes-dashboard-admin
      namespace: kube-system
    

    dp.yaml

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kubernetes-dashboard
      namespace: kube-system
      labels:
        k8s-app: kubernetes-dashboard
        kubernetes.io/cluster-service: "true"
        addonmanager.kubernetes.io/mode: Reconcile
    spec:
      selector:
        matchLabels:
          k8s-app: kubernetes-dashboard
      template:
        metadata:
          labels:
            k8s-app: kubernetes-dashboard
          annotations:
            scheduler.alpha.kubernetes.io/critical-pod: ''
        spec:
          priorityClassName: system-cluster-critical
          containers:
          - name: kubernetes-dashboard
            image: harbor.od.com/public/dashboard:v1.8.3
            resources:
              limits:
                cpu: 100m
                memory: 300Mi
              requests:
                cpu: 50m
                memory: 100Mi
            ports:
            - containerPort: 8443
              protocol: TCP
            args:
              # PLATFORM-SPECIFIC ARGS HERE
              - --auto-generate-certificates
            volumeMounts:
            - name: tmp-volume
              mountPath: /tmp
            livenessProbe:
              httpGet:
                scheme: HTTPS
                path: /
                port: 8443
              initialDelaySeconds: 30
              timeoutSeconds: 30
          volumes:
          - name: tmp-volume
            emptyDir: {}
          serviceAccountName: kubernetes-dashboard-admin
          tolerations:
          - key: "CriticalAddonsOnly"
            operator: "Exists"
    

    svc.yaml

    apiVersion: v1
    kind: Service
    metadata:
      name: kubernetes-dashboard
      namespace: kube-system
      labels:
        k8s-app: kubernetes-dashboard
        kubernetes.io/cluster-service: "true"
        addonmanager.kubernetes.io/mode: Reconcile
    spec:
      selector:
        k8s-app: kubernetes-dashboard
      ports:
      - port: 443
        targetPort: 8443
    

    ingress.yaml

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: kubernetes-dashboard
      namespace: kube-system
      annotations:
        kubernetes.io/ingress.class: traefik
    spec:
      rules:
      - host: dashboard.od.com
        http:
          paths:
          - backend:
              serviceName: kubernetes-dashboard
              servicePort: 443
    
  3. 到11机器配置域名解析:

    $ORIGIN od.com.
    $TTL 600        ; 10 minutes
    @       IN SOA  dns.od.com. dnsadmin.od.com. (
                                    2021121005 ; serial
                                    10800      ; refresh (3 hours)
                                    900        ; retry (15 minutes)
                                    604800     ; expire (1 week)
                                    86400      ; minimum (1 day)
                                    )
                                    NS    dns.od.com.
    
    $TTL 60 ; 1 minute
    dns                A    10.4.7.11
    harbor             A    10.4.7.200
    k8s-yaml           A    10.4.7.200
    traefik            A    10.4.7.10
    dashboard          A    10.4.7.10
    
    [root@hdss7-11 ~]# systemctl restart named
    [root@hdss7-11 ~]# dig -t A dashboard.od.com @10.4.7.11 +short
    10.4.7.10
    
  4. 在任意node节点应用资源配置清单:

    [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/rbac.yaml
    serviceaccount/kubernetes-dashboard-admin created
    clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-admin created
    [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/dp.yaml
    deployment.apps/kubernetes-dashboard created
    [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/svc.yaml
    service/kubernetes-dashboard created
    [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/ingress.yaml
    ingress.extensions/kubernetes-dashboard created
    
    [root@hdss7-22 ~]# kubectl get pods -n kube-system
    NAME                                    READY   STATUS    RESTARTS   AGE
    coredns-6b6c4f9648-cmtnr                1/1     Running   0          29h
    kubernetes-dashboard-76dcdb4677-6v8n8   1/1     Running   0          38s
    traefik-ingress-pl2wp                   1/1     Running   0          47m
    traefik-ingress-wxt8b                   1/1     Running   0          47m
    [root@hdss7-22 ~]# kubectl get svc -n kube-system
    NAME                      TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                  AGE
    coredns                   ClusterIP   192.168.0.2       <none>        53/UDP,53/TCP,9153/TCP   29h
    kubernetes-dashboard      ClusterIP   192.168.194.192   <none>        443/TCP                  46s
    traefik-ingress-service   ClusterIP   192.168.171.12    <none>        80/TCP,8080/TCP          47m
    [root@hdss7-22 ~]# kubectl get ingresses -n kube-system
    NAME                   HOSTS              ADDRESS   PORTS   AGE
    kubernetes-dashboard   dashboard.od.com             80      58s
    traefik-web-ui         traefik.od.com               80      48m
    
    
  5. 访问dashboard.od.com:

    先选择跳过

k8s仪表盘鉴权

配置SSL

上面直接访问域名就可以进入dashboard,不需要登录,我们可以配置登录和权限,让管理员和普通用户权限分开。

  1. 在200机器的/opt/certs目录下,创建证书:

    [root@hdss7-200 certs]# (umask 077; openssl genrsa -out dashboard.od.com.key 2048)
    
    [root@hdss7-200 certs]# openssl req -new -key dashboard.od.com.key -out dashboard.od.com.csr -subj "/CN=dashboard.od.com/C=CN/ST=BJ/L=Beijing/O=ben1234560/OU=ops"
    
    [root@hdss7-200 certs]# openssl x509 -req -in dashboard.od.com.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out dashboard.od.com.crt -days 3650
    
    [root@hdss7-200 certs]# cfssl-certinfo -cert dashboard.od.com.crt
    
  2. 拷贝证书到11/12机器的nginx:

    [root@hdss7-11 ~]# cd /etc/nginx/
    [root@hdss7-11 nginx]# mkdir certs
    [root@hdss7-11 nginx]# cd certs/
    [root@hdss7-11 certs]# scp hdss7-200:/opt/certs/dashboard.od.com.key .
    [root@hdss7-11 certs]# scp hdss7-200:/opt/certs/dashboard.od.com.crt .
    

    创建/etc/nginx/conf.d/dashboard.od.com.conf文件,配置如下:

    server {
        listen 80;
        server_name dashboard.od.com;
    
        rewrite ^(.*)$ https://${server_name}$1 permanent;
    }
    
    server {
        listen  443 ssl;
        server_name dashboard.od.com;
    
        ssl_certificate "certs/dashboard.od.com.crt";
        ssl_certificate_key "certs/dashboard.od.com.key";
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout 10m;
        ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;
    
        location / {
            proxy_pass http://default_backend_traefik;
            proxy_set_header Host   $http_host;
            proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
        }
    }
    

升级dashboard

上面部署的是dashboard1.8版本,下面升级使用dashboard1.10版本,1.8版本的登陆授权不严格,而1.10版本在登陆授权上比较严格,更适用于生产环境。

  1. 到200机器拉去镜像:

    [root@hdss7-200 ~]# docker pull loveone/kubernetes-dashboard-amd64:v1.10.1
    [root@hdss7-200 ~]# docker images |grep dash
    [root@hdss7-200 ~]# docker tag f9aed6605b81 
    [root@hdss7-200 ~]# docker push !$
    
  2. 修改200机器上的dashboard资源配置文件dp.yaml,将其中的镜像更新为1.10.1,也可以在dashboard上进行修改:

     - name: kubernetes-dashboard
            image: harbor.od.com/public/dashboard:v1.10.1
    
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/dp.yaml
    deployment.apps/kubernetes-dashboard configured
    
  3. 1.10.1版本访问dashboard是强制登录的,所以需要先获取token再去登录:

    [root@hdss7-21 ~]# kubectl get secrets -n kube-system
    NAME                                     TYPE                                  DATA   AGE
    coredns-token-9qsmk                      kubernetes.io/service-account-token   3      2d19h
    default-token-msnhk                      kubernetes.io/service-account-token   3      21d
    kubernetes-dashboard-admin-token-mlrz2   kubernetes.io/service-account-token   3      37h
    kubernetes-dashboard-key-holder          Opaque                                2      37h
    traefik-ingress-controller-token-2snzm   kubernetes.io/service-account-token   3      38h
    
    [root@hdss7-21 ~]# kubectl describe secrets kubernetes-dashboard-admin-token-mlrz2 -n kube-system
    Name:         kubernetes-dashboard-admin-token-mlrz2
    Namespace:    kube-system
    Labels:       <none>
    Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard-admin
                  kubernetes.io/service-account.uid: 5b736b1d-7412-466d-93ac-31672569848f
    
    Type:  kubernetes.io/service-account-token
    
    Data
    ====
    ca.crt:     1298 bytes
    namespace:  11 bytes
    token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi1tbHJ6MiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjViNzM2YjFkLTc0MTItNDY2ZC05M2FjLTMxNjcyNTY5ODQ4ZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.rYN4_9yD-RObSIHBlSYsjONuVmPj7wVpEqTIpwc6O3YZeIzF1bJkFDz7TVWeu9oq91TZTv000fidG_qYr-7_Vl7EkfYYEwwiGrPwkocBS9rAO2ir_aex3hXYKz5pA7-6n3cYZLApYbFKLQHTNsvw3_V6EcOIvlagOLe5p5jCmB-AjH4AjqxTdn9ODe8xrILJASY-jXBeMkJsldzQpakFGcJFH8IraRA-INJ-tEFkzMwRutXrnoac79GY6WpXEH4w09FYFI-4iu-EM-Wws4KIGfEul7c1oDitmMnBnodjTpB04tgnaCYYCOAzFH-5cnqyiVZqoUcBgXWuZgQaHSjR8Q
    
  4. 再次访问新的dashboard,开始页已经没有跳过选项,使用上面的token登录:

配置普通用户权限

  1. 200机器的dashboard资源配置文件目录,创建rbac-minimal.yaml文件,内容如下:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
        addonmanager.kubernetes.io/mode: Reconcile
      name: kubernetes-dashboard
      namespace: kube-system
    ---
    kind: Role 
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
        addonmanager.kubernetes.io/mode: Reconcile
      name: kubernetes-dashboard-minimal
      namespace: kube-system
    rules:
      # Allow Dashboard to get,update and delete Dashboard exclusive secrets.
    - apiGroups: [""]
      resources: ["secrets"]
      resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
      verbs: ["get", "update", "delete"]
      # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
    - apiGroups: [""]
      resources: ["configmaps"]
      resourceNames: ["kubernetes-dashboard-settings"]
      verbs: ["get", "update"]
      # Allow Dashboard to get metrics from heapster.
    - apiGroups: [""]
      resources: ["services"]
      resourceNames: ["heapster"]
      verbs: ["proxy"]
    - apiGroups: [""]
      resources: ["services/proxy"]
      resourceNames: ["heapster", "http:heapster:","https:heapster:"]
      verbs: ["get"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: kubernetes-dashboard-minimal
      namespace: kube-system
      labels:
        k8s-app: kubernetes-dashboard
        addonmanager.kubernetes.io/mode: Reconcile
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: kubernetes-dashboard-minimal
    subjects:
    - kind: ServiceAccount
      name: kubernetes-dashboard
      namespace: kube-system
    
  2. 应用清单:

    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/rbac-minimal.yaml
    serviceaccount/kubernetes-dashboard created
    role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
    rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
    
  3. 修改dp.yaml,然后应用dp,就可以看见两个token:

    # 将serviceAccountName改为下面的内容
    serviceAccountName: kubernetes-dashboard
    
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/dp.yaml
    deployment.apps/kubernetes-dashboard configured
    
    [root@hdss7-21 ~]# kubectl get pods -n kube-system
    NAME                                   READY   STATUS    RESTARTS   AGE
    coredns-6b6c4f9648-cmtnr               1/1     Running   0          2d23h
    kubernetes-dashboard-bcb6785dd-qtlr5   1/1     Running   0          16s
    traefik-ingress-pl2wp                  1/1     Running   0          42h
    traefik-ingress-wxt8b                  1/1     Running   0          42h
    
    [root@hdss7-21 ~]# kubectl get secrets -n kube-system
    NAME                                     TYPE                                  DATA   AGE
    coredns-token-9qsmk                      kubernetes.io/service-account-token   3      2d23h
    default-token-msnhk                      kubernetes.io/service-account-token   3      22d
    kubernetes-dashboard-admin-token-mlrz2   kubernetes.io/service-account-token   3      41h
    kubernetes-dashboard-key-holder          Opaque                                2      41h
    kubernetes-dashboard-token-wqb4z         kubernetes.io/service-account-token   3      7m19s
    traefik-ingress-controller-token-2snzm   kubernetes.io/service-account-token   3      42h
    [root@hdss7-21 ~]# kubectl describe secrets kubernetes-dashboard-token-wqb4z -n kube-system
    
    
  4. 使用新的token登录dashboard:

    提示很多权限没有,新增用户权限也只需要配置rbac-xxx.yaml文件并应用即可。

dashboard-heapster

heapster可以让dashboard拥有更多图形化的小插件,更方便我们健康集群状态。

  1. 200机器准备镜像和资源配置清单:

    [root@hdss7-200 k8s-yaml]# cd /data/k8s-yaml/dashboard
    [root@hdss7-200 dashboard]# mkdir heapster
    [root@hdss7-200 dashboard]# cd heapster/
    [root@hdss7-200 heapster]# docker pull bitnami/heapster:1.5.4
    [root@hdss7-200 heapster]# docker images|grep heapster
    bitnami/heapster                     1.5.4           c359b95ad38b   3 years ago    136MB
    [root@hdss7-200 heapster]# docker tag c359b95ad38b harbor.od.com/public/heapster:1.5.4
    [root@hdss7-200 heapster]# docker push !$
    

    rbac.yaml

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: heapster
      namespace: kube-system
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: heapster
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:heapster
    subjects:
    - kind: ServiceAccount
      name: heapster
      namespace: kube-system
    

    dp.yaml

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: heapster
      namespace: kube-system
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            task: monitoring
            k8s-app: heapster
        spec:
          serviceAccountName: heapster
          containers:
          - name: heapster
            image: harbor.od.com/public/heapster:1.5.4
            imagePullPolicy: IfNotPresent
            command:
            - /opt/bitnami/heapster/bin/heapster
            - --source=kubernetes:https://kubernetes.default
    

    svc.yaml

    apiVersion: v1
    kind: Service
    metadata:
      labels:
        task: monitoring
        # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
        # If you are NOT using this as an addon, you should comment out this line.
        kubernetes.io/cluster-service: 'true'
        kubernetes.io/name: Heapster
      name: heapster
      namespace: kube-system
    spec:
      ports:
      - port: 80
        targetPort: 8082
      selector:
        k8s-app: heapster
    
  2. 应用资源清单:

    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/heapster/rbac.yaml
    
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/heapster/dp.yaml
    
    [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/heapster/svc.yaml
    
    

K8S平滑升级

  1. 进入21节点,查看pod状态:

    [root@hdss7-21 ~]# kubectl get node
    NAME                STATUS   ROLES         AGE   VERSION
    hdss7-21.host.com   Ready    master,node   10d   v1.15.12
    hdss7-22.host.com   Ready    master,node   10d   v1.15.12
    [root@hdss7-21 ~]# kubectl get pod -n kube-system -o wide
    NAME                                    READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
    coredns-6b6c4f9648-4dq7t                1/1     Running   0          20m   172.7.22.6   hdss7-22.host.com   <none>     <none>
    heapster-7cb6dc7b94-hjfv9               1/1     Running   0          18m   172.7.21.3   hdss7-21.host.com   <none>     <none>
    kubernetes-dashboard-76dcdb4677-wjr97   1/1     Running   0          74m   172.7.22.5   hdss7-22.host.com   <none>     <none>
    traefik-ingress-pl2wp                   1/1     Running   0          2d    172.7.22.4   hdss7-22.host.com   <none>     <none>
    traefik-ingress-wxt8b                   1/1     Running   0          2d    172.7.21.4   hdss7-21.host.com   <none>     <none>
    
    
  2. 在11机器上,进行nginx配置,然后重新加载nginx:

    # nginx.conf注释21机器配置
    stream {
        upstream kube-apiserver {
    #        server 10.4.7.21:6443     max_fails=3 fail_timeout=30s;
            server 10.4.7.22:6443     max_fails=3 fail_timeout=30s;
        }
        server {
            listen 7443;
            proxy_connect_timeout 2s;
            proxy_timeout 900s;
            proxy_pass kube-apiserver;
        }
    }
    
    
    # od.com.conf同样注释21机器
    upstream default_backend_traefik {
    #    server 10.4.7.21:81     max_fails=3 fail_timeout=10s;
        server 10.4.7.22:81     max_fails=3 fail_timeout=10s;
    }
    
    server {
        server_name *.od.com;
        listen 80;
    
        location / {
            proxy_pass http://default_backend_traefik;
            proxy_set_header Host   $http_host;
            proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
        }
    }
    
    
    # 重载nginx
    [root@hdss7-11 conf.d]# nginx -t
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful
    [root@hdss7-11 conf.d]# systemctl reload nginx
    
    
  3. 删除21机器node节点,并在22机器上查看pod调度:

    [root@hdss7-21 ~]# kubectl get node
    NAME                STATUS   ROLES         AGE   VERSION
    hdss7-21.host.com   Ready    master,node   10d   v1.15.12
    hdss7-22.host.com   Ready    master,node   10d   v1.15.12
    [root@hdss7-21 ~]# kubectl delete node hdss7-21.host.com
    node "hdss7-21.host.com" deleted
    
    [root@hdss7-22 ~]# kubectl get nodes
    NAME                STATUS   ROLES         AGE   VERSION
    hdss7-22.host.com   Ready    master,node   10d   v1.15.12
    [root@hdss7-22 ~]# kubectl get pod -n kube-system -o wide
    NAME                                    READY   STATUS    RESTARTS   AGE     IP           NODE                NOMINATED NODE   READINESS GATES
    coredns-6b6c4f9648-4dq7t                1/1     Running   0          17h     172.7.22.6   hdss7-22.host.com   <none>           <none>
    heapster-7cb6dc7b94-x27cq               1/1     Running   0          8s      172.7.22.7   hdss7-22.host.com   <none>           <none>
    kubernetes-dashboard-76dcdb4677-wjr97   1/1     Running   0          18h     172.7.22.5   hdss7-22.host.com   <none>           <none>
    traefik-ingress-pl2wp                   1/1     Running   0          2d18h   172.7.22.4   hdss7-22.host.com   <none>           <none>
    [root@hdss7-21 ~]# dig -t A kubernetes.default.svc.cluster.local @192.168.0.2 +short
    192.168.0.1
    
  4. 21/22机器上,下载新版本的kubernetes-v1.16.10安装包,放到/opt/src目录:

    [root@hdss7-21 src]# cd /opt/src/
    [root@hdss7-21 src]# wget https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.16.10.tar.gz -O kubernetes-v1.16.10.tar.gz
    [root@hdss7-21 src]# mkdir /opt/1.16.10
    [root@hdss7-21 src]# tar zxvf kubernetes-v1.16.10.tar.gz -C /opt/1.16.10
    
    # 下载kubernetes二进制包
    [root@hdss7-21 opt]# cd /opt/1.16.10/kubernetes-1.16.10/cluster
    [root@hdss7-21 cluster]# ./get-kube.sh
    
    # 解压二进制包
    [root@hdss7-21 server]# cd /opt/1.16.10/kubernetes-1.16.10/cluster/kubernetes/server
    [root@hdss7-21 server]# tar zxvf kubernetes-server-linux-amd64.tar.gz
    [root@hdss7-22 server]# mv kubernetes /opt/kubernetes-v1.16.10
    
    # 删除多余文件
    [root@hdss7-22 server]# cd /opt/kubernetes-v1.16.10/
    [root@hdss7-22 kubernetes-v1.16.10]# rm -rf kubernetes-src.tar.gz
    [root@hdss7-22 kubernetes-v1.16.10]# cd server/bin/
    [root@hdss7-22 bin]# rm -rf *.tar *_tag
    
    # 复制老版本cert和conf以及启动脚本
    [root@hdss7-22 bin]# cp -r /opt/kubernetes/server/bin/cert .
    [root@hdss7-22 bin]# cp -r /opt/kubernetes/server/bin/conf .
    [root@hdss7-22 bin]# cp /opt/kubernetes/server/bin/*.sh .
    
    # 删除老版本kubernetes目录软连接,创建新版本软连接
    [root@hdss7-22 bin]# cd /opt/
    [root@hdss7-22 opt]# rm -rf kubernetes
    [root@hdss7-22 opt]# ln -s /opt/kubernetes-v1.16.10 /opt/kubernetes
    
  5. 重启supervisor守护进程,生产环境需要一个一个重启,测试环境可以批量重启:

    [root@hdss7-22 bin]# supervisorctl stop all
    [root@hdss7-22 bin]# supervisorctl start all
    
    [root@hdss7-21 flannel]# kubectl get node
    NAME                STATUS   ROLES    AGE    VERSION
    hdss7-21.host.com   Ready    <none>   7m1s   v1.16.10
    hdss7-22.host.com   Ready    <none>   7m8s   v1.16.10
    

    get node可以看到版本已经更新。