k8s Paas实战——部署准备

k8s Paas实战——部署准备

Administrator 841 2021-12-14

服务器规划

IPhostnamerole
10.4.7.21hdss7-21master+node 主控+运算节点(etcd,apiserver,kubelet,Controller-manager,ingress,scheduler)
10.4.7.22hdss7-22master+node 主控+运算节点(etcd,apiserver,kubelet,Controller-manager,ingress,scheduler)
10.4.7.11hdss7-11反向代理/bind9
10.4.7.12hdss7-12反向代理/etcd
10.4.7.200hdss7-200运维主机/harbor(资源存放)

准备工作

批量配置

在所有主机上执行命令关闭selinux和防火墙:

sed -i 's/enforcing/disabled/g' /etc/selinux/config
setenforce 0
systemctl disable firewalld && systemctl stop firewalld

基础软件安装

yum install epel-release -y
yum install wget net-tools telnet tree nmap sysstat lrzsz dos2unix bin-utils -y

bind9安装配置(DNS服务)

  1. bind9用于K8S ingress调度时我们能够直接使用DNS解析。bind9安装在10.4.7.11服务器上:

    yum install bind bind-utils -y
    
    # vi /etc/named.conf
    options {
            listen-on port 53 { 127.0.0.1; };	#改成10.4.7.11,指定监听IP
            listen-on-v6 port 53 { ::1; };	#删除
            directory       "/var/named";
            dump-file       "/var/named/data/cache_dump.db";
            statistics-file "/var/named/data/named_stats.txt";
            memstatistics-file "/var/named/data/named_mem_stats.txt";
            recursing-file  "/var/named/data/named.recursing";
            secroots-file   "/var/named/data/named.secroots";
            allow-query     { localhost; };	#改为any,指定哪些客户端能查询DNS
            forwarders       { 10.4.7.254; };	#新增,指定上级DNS
            recursion yes;
    
            dnssec-enable yes;	#改为no
            dnssec-validation yes;	#改为no
    
    
  2. named.conf最终配置如下:

    options {
            listen-on port 53 { 10.4.7.11; };
            directory       "/var/named";
            dump-file       "/var/named/data/cache_dump.db";
            statistics-file "/var/named/data/named_stats.txt";
            memstatistics-file "/var/named/data/named_mem_stats.txt";
            recursing-file  "/var/named/data/named.recursing";
            secroots-file   "/var/named/data/named.secroots";
            allow-query     { any; };
            forwarders      { 10.4.7.254; };
    
            /*
             - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
             - If you are building a RECURSIVE (caching) DNS server, you need to enable
               recursion.
             - If your recursive DNS server has a public IP address, you MUST enable access
               control to limit queries to your legitimate users. Failing to do so will
               cause your server to become part of large scale DNS amplification
               attacks. Implementing BCP38 within your network would greatly
               reduce such attack surface
            */
            recursion yes;
    
            dnssec-enable no;
            dnssec-validation no;
    

    使用named-checkconf命令检查bind9配置文件是否有错误,无输出则无报错。

  3. 编辑/etc/named.rfc1912.zones文件,配置主机域和业务域,主机域使用host.com,业务域使用od.com,在配置文件最后添加下面的内容:

    zone "host.com" IN {
            type master;
            file "host.com.zone";
            allow-update { 10.4.7.11; };
    };
    
    zone "od.com" IN {
            type master;
            file "od.com.zone";
            allow-update { 10.4.7.11; };
    };
    
  4. 编辑/var/named/host.com.zone,内容如下:

    $ORIGIN host.com.
    $TTL 600        ; 10 minutes
    @       IN SOA  dns.host.com. dnsadmin.host.com. (
                                    2021121001 ; serial
                                    10800      ; refresh (3 hours)
                                    900        ; retry (15 minutes)
                                    604800     ; expire (1 week)
                                    86400      ; minimum (1 day)
                                    )
                            NS    dns.host.com.
    
    $TTL 60 ; 1 minute
    dns                A    10.4.7.11
    HDSS7-11           A    10.4.7.11
    HDSS7-12           A    10.4.7.12
    HDSS7-21           A    10.4.7.21
    HDSS7-22           A    10.4.7.22
    HDSS7-200          A    10.4.7.200
    
  5. 编辑/var/named/od.com.zone文件。内容如下:

    $ORIGIN od.com.
    $TTL 600        ; 10 minutes
    @       IN SOA  dns.od.com. dnsadmin.od.com. (
                                    2021121001 ; serial
                                    10800      ; refresh (3 hours)
                                    900        ; retry (15 minutes)
                                    604800     ; expire (1 week)
                                    86400      ; minimum (1 day)
                                    )
                                    NS    dns.od.com.
    
    $TTL 60 ; 1 minute
    dns                A    10.4.7.11
    
    

    其中serial表示记录的时间,格式为当前日期加第一条记录:20211210+01;

    TTL 600:表示IP包被路由器丢弃之前允许通过的最大网段数

    10 minutes:过期时间

    SOA:一个域权威记录的相关信息,后面有5组参数分别设定了该域相关部分

    dnsadmin.od.com.:假邮箱

    $ORIGIN:即下列域名自动补充od.com,如dns,外面看来是dns.od.com

  6. 再次执行named-checkconf检查配置文件,然后启动named服务:

    [root@hdss7-11 named]# systemctl start named
    [root@hdss7-11 named]# netstat -tlnp |grep 53
    tcp        0      0 10.4.7.11:53            0.0.0.0:*               LISTEN      2391/named
    tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      2391/named
    tcp6       0      0 ::1:953                 :::*                    LISTEN      2391/named
    
  7. 使用dig命令检查主机域是否解析,:

    [root@hdss7-11 named]# dig -t A hdss7-21.host.com @10.4.7.11 +short
    10.4.7.21
    
  8. 将所有的虚拟机服务器和宿主机windows的VMnet8接口的DNS配置修改为10.4.7.11,修改后如果11机器关机,则会导致无法上网。

    [root@hdss7-22 ~]# sed -i 's/DNS1=10.4.7.254/DNS1=10.4.7.11/g' /etc/sysconfig/network-scripts/ifcfg-ens33
    [root@hdss7-22 ~]# echo 'NM_CONTROLLED=no' >> /etc/sysconfig/network-scripts/ifcfg-ens33
    [root@hdss7-22 ~]# echo 'PEERDNS=no' >> /etc/sysconfig/network-scripts/ifcfg-ens33
    [root@hdss7-12 ~]# systemctl restart network
    

K8S配置准备

证书签发环境

为K8S做证书签发准备,这里使用CFSSL,CFSSL是CloudFlare开源的一款PKI/TLS工具,包含了一个命令行工具和一个用于签名验证且捆绑TLS证书的HTTP API服务;

在K8S中,集群证书分为三种类型:

  • client certificate:客户端证书,例如etcdctl,etcd proxy,fleetctl,docker客户端;
  • server certificate:服务端证书,客户端以此验证服务端身份,例如docker服务端,kube-apiserver;
  • peer certificate:双向证书(既是server cert,又是client cert),用于etcd集群成员间通信。

etcd节点需要表示自己服务的server cert,也许要client cert与etcd集群其他节点交互,可以分别指定两个证书,也可以使用一个对等证书;

master节点需要表示apiserver服务的server cert,也需要client cert连接etcd集群,也可以使用对等证书;

kubectl,calico,kube-proxy只需要client cert,kubelet证书比较特殊,其由node节点的TLS BootStrap向apiserver请求,由master节点的controller-manager自动签发,包含一个client cert和一个server cert。

  1. 在200机器上,安装cfssl相关软件:

    wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64 -O /usr/bin/cfssl
    wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 -O /usr/bin/cfssl-json
    wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64 -O /usr/bin/cfssl-certinfo
    
    chmod +x /usr/bin/cfssl*
    
  2. 进入/opt/目录,创建目录和证书:

    cd /opt/
    mkdir certs
    cd certs/
    

    创建ca-csr.json文件,写入以下内容:

    {
        "CN": "ben123123",
        "hosts": [
        ],
        "key": {
            "algo": "rsa",
            "size": 2048
        },
        "names": [
            {
                "C": "CN",
                "ST": "beijing",
                "L": "beijing",
                "O": "od",
                "OU": "ops"
            }
        ],
        "ca": {
            "expiry": "175200h"
        }
    }
    

    使用cfssl命令生成ca证书:

    [root@hdss7-200 certs]# cfssl gencert -initca ca-csr.json |cfssl-json -bare ca
    2021/12/14 17:10:37 [INFO] generating a new CA key and certificate from CSR
    2021/12/14 17:10:37 [INFO] generate received request
    2021/12/14 17:10:37 [INFO] received CSR
    2021/12/14 17:10:37 [INFO] generating key: rsa-2048
    2021/12/14 17:10:38 [INFO] encoded CSR
    2021/12/14 17:10:38 [INFO] signed certificate with serial number 548306008492011779450654226423761249954736220629
    
    [root@hdss7-200 certs]# ll
    总用量 16
    -rw-r--r-- 1 root root 1041 12月 14 17:10 ca.csr
    -rw-r--r-- 1 root root  328 12月 14 17:10 ca-csr.json
    -rw------- 1 root root 1675 12月 14 17:10 ca-key.pem
    -rw-r--r-- 1 root root 1298 12月 14 17:10 ca.pem
    

安装docker

  1. 架构设计中,21,22机器是运算节点,所以在21,22,200这三台机器上安装docker,200机器后续作为docker仓库,docker安装命令如下:

    yum install -y yum-utils
    yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    yum install -y docker-ce docker-ce-cli containerd.io
    
  2. 在21,22,200机器上修改docker配置,创建/data/docker/etc/docker目录,写入配置文件/etc/docker/daemon.json并启动docker:

    {
      "graph": "/data/docker",
      "storage-driver": "overlay2",
      "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
      "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"],
      "bip": "172.7.21.1/24",
      "exec-opts": ["native.cgroupdriver=systemd"],
      "live-restore": true
    }
    

    **Tips:**这里的bip配置,如果在21机器上,就是172.7.21.1,在22机器上就是172.7.22.1,这样配置是方便以后出问题时直接查找容器所在机器。

    [root@hdss7-200 ~]# systemctl start docker
    [root@hdss7-200 ~]# docker version
    Client: Docker Engine - Community
     Version:           20.10.12
     API version:       1.41
     Go version:        go1.16.12
     Git commit:        e91ed57
     Built:             Mon Dec 13 11:45:41 2021
     OS/Arch:           linux/amd64
     Context:           default
     Experimental:      true
    
    Server: Docker Engine - Community
     Engine:
      Version:          20.10.12
      API version:      1.41 (minimum version 1.12)
      Go version:       go1.16.12
      Git commit:       459d0df
      Built:            Mon Dec 13 11:44:05 2021
      OS/Arch:          linux/amd64
      Experimental:     false
     containerd:
      Version:          1.4.12
      GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
     runc:
      Version:          1.0.2
      GitCommit:        v1.0.2-0-g52b36a2
     docker-init:
      Version:          0.19.0
      GitCommit:        de40ad0
    
    
  3. 在200机器安装docker-compose:

    curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
    chmod +x /usr/local/bin/docker-compose
    

harbor仓库部署

  1. harbor仓库同样安装在200机器,下载harbor最新稳定包并解压到/opt/目录下:

    [root@hdss7-200 ~]# cd /opt/
    [root@hdss7-200 opt]# mkdir src
    [root@hdss7-200 src]# wget https://ghproxy.fsou.cc/https://github.com/goharbor/harbor/releases/download/v2.3.4/harbor-offline-installer-v2.3.4.tgz
    [root@hdss7-200 src]# tar zxvf harbor-offline-installer-v2.3.4.tgz -C /opt/
    [root@hdss7-200 src]# cd /opt
    [root@hdss7-200 opt]# mv harbor/ harbor-v2.3.4
    [root@hdss7-200 opt]# ln -s /opt/harbor-v2.3.4 /opt/harbor
    [root@hdss7-200 opt]# cd harbor
    
  2. harbor.yml.tmpl文件更名为harbor.yml,需要修改的原配置如下:

    hostname: reg.mydomain.com
    
    # http related config
    http:
      # port for http, default is 80. If https enabled, this port will redirect to https port
      port: 80
    
    # https related config
    https:
      # https port for harbor, default is 443
      port: 443
      # The path of cert and key files for nginx
      certificate: /your/certificate/path
      private_key: /your/private/key/path
    
    data_volume: /data
    
    log:
      # options are debug, info, warning, error, fatal
      level: info
      # configs for logs in local storage
      local:
        # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed ratherthan rotated.
        rotate_count: 50
        # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
        # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
        # are all valid.
        rotate_size: 200M
        # The directory on your host that store log
        location: /var/log/harbor
    
    

    修改后如下,其中https项的参数全部注释掉,需要https的,使用nginx反代配置ssl:

    hostname: harbor.od.com
    
    # http related config
    http:
      # port for http, default is 80. If https enabled, this port will redirect to https port
      port: 180
    
    # https related config
    #https:
      # https port for harbor, default is 443
    #  port: 443
      # The path of cert and key files for nginx
    #  certificate: /your/certificate/path
    #  private_key: /your/private/key/path
    
    data_volume: /data/harbor
    
    log:
      # options are debug, info, warning, error, fatal
      level: info
      # configs for logs in local storage
      local:
        # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed ratherthan rotated.
        rotate_count: 50
        # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
        # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
        # are all valid.
        rotate_size: 200M
        # The directory on your host that store log
        location: /data/harbor/logs
    
  3. 创建所需的目录,然后执行安装脚本:

    [root@hdss7-200 harbor]# mkdir -p /data/harbor/logs
    [root@hdss7-200 harbor]# ./install.sh
    
    [root@hdss7-200 harbor]# docker-compose ps
          Name                     Command                  State                       Ports
    ----------------------------------------------------------------------------------------------------------
    harbor-core         /harbor/entrypoint.sh            Up (healthy)
    harbor-db           /docker-entrypoint.sh 96 13      Up (healthy)
    harbor-jobservice   /harbor/entrypoint.sh            Up (healthy)
    harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
    harbor-portal       nginx -g daemon off;             Up (healthy)
    nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:180->8080/tcp,:::180->8080/tcp
    redis               redis-server /etc/redis.conf     Up (healthy)
    registry            /home/harbor/entrypoint.sh       Up (healthy)
    registryctl         /home/harbor/start.sh            Up (healthy)
    
  4. harbor安装完成后,使用yum install -y nginx安装nginx,用来反向代理harbor,然后在/etc/nginx/conf.d/目录下创建harbor.od.com.conf配置文件,然后启动nginx,配置如下:

    server {
        listen 80;
        server_name harbor.od.com;
    
        client_max_body_size 1000m;
    
        location / {
            proxy_pass http://127.0.0.1:180;
        }
    }
    
    
  5. 在11机器解析harbor域名,编辑/var/named/od.com.zone,serial序号加1,同时增加habor A记录:

    $ORIGIN od.com.
    $TTL 600        ; 10 minutes
    @       IN SOA  dns.od.com. dnsadmin.od.com. (
                                    2021121002 ; serial
                                    10800      ; refresh (3 hours)
                                    900        ; retry (15 minutes)
                                    604800     ; expire (1 week)
                                    86400      ; minimum (1 day)
                                    )
                                    NS    dns.od.com.
    
    $TTL 60 ; 1 minute
    dns                A    10.4.7.11
    harbor             A    10.4.7.200
    
    [root@hdss7-11 ~]# systemctl restart named
    [root@hdss7-11 ~]# dig -t A harbor.od.com +short
    10.4.7.200
    

    harbor

  6. 浏览器访问harbor,登录之后创建一个名为public公开项目,默认账号如下:

    账号:admin
    密码:Harbor12345
    

    create repo

  7. 200机器测试推送镜像到harbor仓库:

    [root@hdss7-200 harbor]# docker pull nginx:1.21
    [root@hdss7-200 harbor]# docker images |grep 1.21
    nginx                           1.21      f652ca386ed1   12 days ago   141MB
    
    [root@hdss7-200 harbor]# docker tag nginx:1.21 harbor.od.com/public/nginx:v1.21
    
    [root@hdss7-200 harbor]# docker login harbor.od.com
    Username: admin
    Password:
    WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
    Configure a credential helper to remove this warning. See
    https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    
    Login Succeeded
    
    [root@hdss7-200 harbor]# docker push harbor.od.com/public/nginx:v1.21
    The push refers to repository [harbor.od.com/public/nginx]
    2bed47a66c07: Pushed
    82caad489ad7: Pushed
    d3e1dca44e82: Pushed
    c9fcd9c6ced8: Pushed
    0664b7821b60: Pushed
    9321ff862abb: Pushed
    v1.21: digest: sha256:4424e31f2c366108433ecca7890ad527b243361577180dfd9a5bb36e828abf47 size: 1570
    
    

    到此,harbor仓库部署成功。