K8S的交付服务的形式交付,流程是:准备镜像->准备清单->解析域名->应用配置清单->完成。
Dashboard安装部署
dashboard是向企业展示度量信息和关键业务指标现状的数据虚拟化工具,我们之前的资源部署都是登录到机器上执行命令,是非常不安全的方式,并且实际应用中,开发人员也需要看到pod的情况,不能让他们也登录主机去查看,所以需要一个有权限控制的界面展示和控制的工具。
部署步骤
-
在200机器上准备镜像:
[root@hdss7-200 traefik]# cd /data/k8s-yaml/ [root@hdss7-200 k8s-yaml]# docker pull k8scn/kubernetes-dashboard-amd64:v1.8.3 [root@hdss7-200 k8s-yaml]# docker images |grep dashboard [root@hdss7-200 k8s-yaml]# docker tag fcac9aa03fd6 harbor.od.com/public/dashboard:v1.8.3 [root@hdss7-200 k8s-yaml]# docker push !$
-
创建
/data/k8s-yaml/dashboard
目录,创建资源配置清单文件:rbac.yaml
apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard addonmanager.kubernetes.io/mode: Reconcile name: kubernetes-dashboard-admin namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard-admin namespace: kube-system labels: k8s-app: kubernetes-dashboard addonmanager.kubernetes.io/mode: Reconcile roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: kubernetes-dashboard-admin namespace: kube-system
dp.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: kubernetes-dashboard namespace: kube-system labels: k8s-app: kubernetes-dashboard kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: priorityClassName: system-cluster-critical containers: - name: kubernetes-dashboard image: harbor.od.com/public/dashboard:v1.8.3 resources: limits: cpu: 100m memory: 300Mi requests: cpu: 50m memory: 100Mi ports: - containerPort: 8443 protocol: TCP args: # PLATFORM-SPECIFIC ARGS HERE - --auto-generate-certificates volumeMounts: - name: tmp-volume mountPath: /tmp livenessProbe: httpGet: scheme: HTTPS path: / port: 8443 initialDelaySeconds: 30 timeoutSeconds: 30 volumes: - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard-admin tolerations: - key: "CriticalAddonsOnly" operator: "Exists"
svc.yaml
apiVersion: v1 kind: Service metadata: name: kubernetes-dashboard namespace: kube-system labels: k8s-app: kubernetes-dashboard kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: selector: k8s-app: kubernetes-dashboard ports: - port: 443 targetPort: 8443
ingress.yaml
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: kubernetes-dashboard namespace: kube-system annotations: kubernetes.io/ingress.class: traefik spec: rules: - host: dashboard.od.com http: paths: - backend: serviceName: kubernetes-dashboard servicePort: 443
-
到11机器配置域名解析:
$ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2021121005 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11 harbor A 10.4.7.200 k8s-yaml A 10.4.7.200 traefik A 10.4.7.10 dashboard A 10.4.7.10
[root@hdss7-11 ~]# systemctl restart named [root@hdss7-11 ~]# dig -t A dashboard.od.com @10.4.7.11 +short 10.4.7.10
-
在任意node节点应用资源配置清单:
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/rbac.yaml serviceaccount/kubernetes-dashboard-admin created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-admin created [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/dp.yaml deployment.apps/kubernetes-dashboard created [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/svc.yaml service/kubernetes-dashboard created [root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/ingress.yaml ingress.extensions/kubernetes-dashboard created
[root@hdss7-22 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6b6c4f9648-cmtnr 1/1 Running 0 29h kubernetes-dashboard-76dcdb4677-6v8n8 1/1 Running 0 38s traefik-ingress-pl2wp 1/1 Running 0 47m traefik-ingress-wxt8b 1/1 Running 0 47m [root@hdss7-22 ~]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE coredns ClusterIP 192.168.0.2 <none> 53/UDP,53/TCP,9153/TCP 29h kubernetes-dashboard ClusterIP 192.168.194.192 <none> 443/TCP 46s traefik-ingress-service ClusterIP 192.168.171.12 <none> 80/TCP,8080/TCP 47m [root@hdss7-22 ~]# kubectl get ingresses -n kube-system NAME HOSTS ADDRESS PORTS AGE kubernetes-dashboard dashboard.od.com 80 58s traefik-web-ui traefik.od.com 80 48m
-
访问dashboard.od.com:
先选择跳过
k8s仪表盘鉴权
配置SSL
上面直接访问域名就可以进入dashboard,不需要登录,我们可以配置登录和权限,让管理员和普通用户权限分开。
-
在200机器的
/opt/certs
目录下,创建证书:[root@hdss7-200 certs]# (umask 077; openssl genrsa -out dashboard.od.com.key 2048) [root@hdss7-200 certs]# openssl req -new -key dashboard.od.com.key -out dashboard.od.com.csr -subj "/CN=dashboard.od.com/C=CN/ST=BJ/L=Beijing/O=ben1234560/OU=ops" [root@hdss7-200 certs]# openssl x509 -req -in dashboard.od.com.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out dashboard.od.com.crt -days 3650 [root@hdss7-200 certs]# cfssl-certinfo -cert dashboard.od.com.crt
-
拷贝证书到11/12机器的nginx:
[root@hdss7-11 ~]# cd /etc/nginx/ [root@hdss7-11 nginx]# mkdir certs [root@hdss7-11 nginx]# cd certs/ [root@hdss7-11 certs]# scp hdss7-200:/opt/certs/dashboard.od.com.key . [root@hdss7-11 certs]# scp hdss7-200:/opt/certs/dashboard.od.com.crt .
创建
/etc/nginx/conf.d/dashboard.od.com.conf
文件,配置如下:server { listen 80; server_name dashboard.od.com; rewrite ^(.*)$ https://${server_name}$1 permanent; } server { listen 443 ssl; server_name dashboard.od.com; ssl_certificate "certs/dashboard.od.com.crt"; ssl_certificate_key "certs/dashboard.od.com.key"; ssl_session_cache shared:SSL:1m; ssl_session_timeout 10m; ssl_ciphers HIGH:!aNULL:!MD5; ssl_prefer_server_ciphers on; location / { proxy_pass http://default_backend_traefik; proxy_set_header Host $http_host; proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for; } }
升级dashboard
上面部署的是dashboard1.8版本,下面升级使用dashboard1.10版本,1.8版本的登陆授权不严格,而1.10版本在登陆授权上比较严格,更适用于生产环境。
-
到200机器拉去镜像:
[root@hdss7-200 ~]# docker pull loveone/kubernetes-dashboard-amd64:v1.10.1 [root@hdss7-200 ~]# docker images |grep dash [root@hdss7-200 ~]# docker tag f9aed6605b81 [root@hdss7-200 ~]# docker push !$
-
修改200机器上的dashboard资源配置文件
dp.yaml
,将其中的镜像更新为1.10.1,也可以在dashboard上进行修改:- name: kubernetes-dashboard image: harbor.od.com/public/dashboard:v1.10.1
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/dp.yaml deployment.apps/kubernetes-dashboard configured
-
1.10.1版本访问dashboard是强制登录的,所以需要先获取token再去登录:
[root@hdss7-21 ~]# kubectl get secrets -n kube-system NAME TYPE DATA AGE coredns-token-9qsmk kubernetes.io/service-account-token 3 2d19h default-token-msnhk kubernetes.io/service-account-token 3 21d kubernetes-dashboard-admin-token-mlrz2 kubernetes.io/service-account-token 3 37h kubernetes-dashboard-key-holder Opaque 2 37h traefik-ingress-controller-token-2snzm kubernetes.io/service-account-token 3 38h [root@hdss7-21 ~]# kubectl describe secrets kubernetes-dashboard-admin-token-mlrz2 -n kube-system Name: kubernetes-dashboard-admin-token-mlrz2 Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name: kubernetes-dashboard-admin kubernetes.io/service-account.uid: 5b736b1d-7412-466d-93ac-31672569848f Type: kubernetes.io/service-account-token Data ==== ca.crt: 1298 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi1tbHJ6MiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjViNzM2YjFkLTc0MTItNDY2ZC05M2FjLTMxNjcyNTY5ODQ4ZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.rYN4_9yD-RObSIHBlSYsjONuVmPj7wVpEqTIpwc6O3YZeIzF1bJkFDz7TVWeu9oq91TZTv000fidG_qYr-7_Vl7EkfYYEwwiGrPwkocBS9rAO2ir_aex3hXYKz5pA7-6n3cYZLApYbFKLQHTNsvw3_V6EcOIvlagOLe5p5jCmB-AjH4AjqxTdn9ODe8xrILJASY-jXBeMkJsldzQpakFGcJFH8IraRA-INJ-tEFkzMwRutXrnoac79GY6WpXEH4w09FYFI-4iu-EM-Wws4KIGfEul7c1oDitmMnBnodjTpB04tgnaCYYCOAzFH-5cnqyiVZqoUcBgXWuZgQaHSjR8Q
-
再次访问新的dashboard,开始页已经没有跳过选项,使用上面的token登录:
配置普通用户权限
-
200机器的dashboard资源配置文件目录,创建
rbac-minimal.yaml
文件,内容如下:apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard addonmanager.kubernetes.io/mode: Reconcile name: kubernetes-dashboard namespace: kube-system --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard addonmanager.kubernetes.io/mode: Reconcile name: kubernetes-dashboard-minimal namespace: kube-system rules: # Allow Dashboard to get,update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics from heapster. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:","https:heapster:"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubernetes-dashboard-minimal namespace: kube-system labels: k8s-app: kubernetes-dashboard addonmanager.kubernetes.io/mode: Reconcile roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboard-minimal subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kube-system
-
应用清单:
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/rbac-minimal.yaml serviceaccount/kubernetes-dashboard created role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
-
修改
dp.yaml
,然后应用dp,就可以看见两个token:# 将serviceAccountName改为下面的内容 serviceAccountName: kubernetes-dashboard
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/dp.yaml deployment.apps/kubernetes-dashboard configured [root@hdss7-21 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6b6c4f9648-cmtnr 1/1 Running 0 2d23h kubernetes-dashboard-bcb6785dd-qtlr5 1/1 Running 0 16s traefik-ingress-pl2wp 1/1 Running 0 42h traefik-ingress-wxt8b 1/1 Running 0 42h [root@hdss7-21 ~]# kubectl get secrets -n kube-system NAME TYPE DATA AGE coredns-token-9qsmk kubernetes.io/service-account-token 3 2d23h default-token-msnhk kubernetes.io/service-account-token 3 22d kubernetes-dashboard-admin-token-mlrz2 kubernetes.io/service-account-token 3 41h kubernetes-dashboard-key-holder Opaque 2 41h kubernetes-dashboard-token-wqb4z kubernetes.io/service-account-token 3 7m19s traefik-ingress-controller-token-2snzm kubernetes.io/service-account-token 3 42h [root@hdss7-21 ~]# kubectl describe secrets kubernetes-dashboard-token-wqb4z -n kube-system
-
使用新的token登录dashboard:
提示很多权限没有,新增用户权限也只需要配置rbac-xxx.yaml文件并应用即可。
dashboard-heapster
heapster可以让dashboard拥有更多图形化的小插件,更方便我们健康集群状态。
-
200机器准备镜像和资源配置清单:
[root@hdss7-200 k8s-yaml]# cd /data/k8s-yaml/dashboard [root@hdss7-200 dashboard]# mkdir heapster [root@hdss7-200 dashboard]# cd heapster/ [root@hdss7-200 heapster]# docker pull bitnami/heapster:1.5.4 [root@hdss7-200 heapster]# docker images|grep heapster bitnami/heapster 1.5.4 c359b95ad38b 3 years ago 136MB [root@hdss7-200 heapster]# docker tag c359b95ad38b harbor.od.com/public/heapster:1.5.4 [root@hdss7-200 heapster]# docker push !$
rbac.yaml
apiVersion: v1 kind: ServiceAccount metadata: name: heapster namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: heapster roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:heapster subjects: - kind: ServiceAccount name: heapster namespace: kube-system
dp.yaml
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: heapster namespace: kube-system spec: replicas: 1 template: metadata: labels: task: monitoring k8s-app: heapster spec: serviceAccountName: heapster containers: - name: heapster image: harbor.od.com/public/heapster:1.5.4 imagePullPolicy: IfNotPresent command: - /opt/bitnami/heapster/bin/heapster - --source=kubernetes:https://kubernetes.default
svc.yaml
apiVersion: v1 kind: Service metadata: labels: task: monitoring # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line. kubernetes.io/cluster-service: 'true' kubernetes.io/name: Heapster name: heapster namespace: kube-system spec: ports: - port: 80 targetPort: 8082 selector: k8s-app: heapster
-
应用资源清单:
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/heapster/rbac.yaml [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/heapster/dp.yaml [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/heapster/svc.yaml
K8S平滑升级
-
进入21节点,查看pod状态:
[root@hdss7-21 ~]# kubectl get node NAME STATUS ROLES AGE VERSION hdss7-21.host.com Ready master,node 10d v1.15.12 hdss7-22.host.com Ready master,node 10d v1.15.12 [root@hdss7-21 ~]# kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-6b6c4f9648-4dq7t 1/1 Running 0 20m 172.7.22.6 hdss7-22.host.com <none> <none> heapster-7cb6dc7b94-hjfv9 1/1 Running 0 18m 172.7.21.3 hdss7-21.host.com <none> <none> kubernetes-dashboard-76dcdb4677-wjr97 1/1 Running 0 74m 172.7.22.5 hdss7-22.host.com <none> <none> traefik-ingress-pl2wp 1/1 Running 0 2d 172.7.22.4 hdss7-22.host.com <none> <none> traefik-ingress-wxt8b 1/1 Running 0 2d 172.7.21.4 hdss7-21.host.com <none> <none>
-
在11机器上,进行nginx配置,然后重新加载nginx:
# nginx.conf注释21机器配置 stream { upstream kube-apiserver { # server 10.4.7.21:6443 max_fails=3 fail_timeout=30s; server 10.4.7.22:6443 max_fails=3 fail_timeout=30s; } server { listen 7443; proxy_connect_timeout 2s; proxy_timeout 900s; proxy_pass kube-apiserver; } }
# od.com.conf同样注释21机器 upstream default_backend_traefik { # server 10.4.7.21:81 max_fails=3 fail_timeout=10s; server 10.4.7.22:81 max_fails=3 fail_timeout=10s; } server { server_name *.od.com; listen 80; location / { proxy_pass http://default_backend_traefik; proxy_set_header Host $http_host; proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for; } }
# 重载nginx [root@hdss7-11 conf.d]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@hdss7-11 conf.d]# systemctl reload nginx
-
删除21机器node节点,并在22机器上查看pod调度:
[root@hdss7-21 ~]# kubectl get node NAME STATUS ROLES AGE VERSION hdss7-21.host.com Ready master,node 10d v1.15.12 hdss7-22.host.com Ready master,node 10d v1.15.12 [root@hdss7-21 ~]# kubectl delete node hdss7-21.host.com node "hdss7-21.host.com" deleted
[root@hdss7-22 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION hdss7-22.host.com Ready master,node 10d v1.15.12 [root@hdss7-22 ~]# kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-6b6c4f9648-4dq7t 1/1 Running 0 17h 172.7.22.6 hdss7-22.host.com <none> <none> heapster-7cb6dc7b94-x27cq 1/1 Running 0 8s 172.7.22.7 hdss7-22.host.com <none> <none> kubernetes-dashboard-76dcdb4677-wjr97 1/1 Running 0 18h 172.7.22.5 hdss7-22.host.com <none> <none> traefik-ingress-pl2wp 1/1 Running 0 2d18h 172.7.22.4 hdss7-22.host.com <none> <none> [root@hdss7-21 ~]# dig -t A kubernetes.default.svc.cluster.local @192.168.0.2 +short 192.168.0.1
-
21/22机器上,下载新版本的kubernetes-v1.16.10安装包,放到
/opt/src
目录:[root@hdss7-21 src]# cd /opt/src/ [root@hdss7-21 src]# wget https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.16.10.tar.gz -O kubernetes-v1.16.10.tar.gz [root@hdss7-21 src]# mkdir /opt/1.16.10 [root@hdss7-21 src]# tar zxvf kubernetes-v1.16.10.tar.gz -C /opt/1.16.10 # 下载kubernetes二进制包 [root@hdss7-21 opt]# cd /opt/1.16.10/kubernetes-1.16.10/cluster [root@hdss7-21 cluster]# ./get-kube.sh # 解压二进制包 [root@hdss7-21 server]# cd /opt/1.16.10/kubernetes-1.16.10/cluster/kubernetes/server [root@hdss7-21 server]# tar zxvf kubernetes-server-linux-amd64.tar.gz [root@hdss7-22 server]# mv kubernetes /opt/kubernetes-v1.16.10 # 删除多余文件 [root@hdss7-22 server]# cd /opt/kubernetes-v1.16.10/ [root@hdss7-22 kubernetes-v1.16.10]# rm -rf kubernetes-src.tar.gz [root@hdss7-22 kubernetes-v1.16.10]# cd server/bin/ [root@hdss7-22 bin]# rm -rf *.tar *_tag # 复制老版本cert和conf以及启动脚本 [root@hdss7-22 bin]# cp -r /opt/kubernetes/server/bin/cert . [root@hdss7-22 bin]# cp -r /opt/kubernetes/server/bin/conf . [root@hdss7-22 bin]# cp /opt/kubernetes/server/bin/*.sh . # 删除老版本kubernetes目录软连接,创建新版本软连接 [root@hdss7-22 bin]# cd /opt/ [root@hdss7-22 opt]# rm -rf kubernetes [root@hdss7-22 opt]# ln -s /opt/kubernetes-v1.16.10 /opt/kubernetes
-
重启supervisor守护进程,生产环境需要一个一个重启,测试环境可以批量重启:
[root@hdss7-22 bin]# supervisorctl stop all [root@hdss7-22 bin]# supervisorctl start all
[root@hdss7-21 flannel]# kubectl get node NAME STATUS ROLES AGE VERSION hdss7-21.host.com Ready <none> 7m1s v1.16.10 hdss7-22.host.com Ready <none> 7m8s v1.16.10
get node可以看到版本已经更新。