服务器规划

IP	hostname	role
10.4.7.21	hdss7-21	master+node 主控+运算节点(etcd,apiserver,kubelet,Controller-manager,ingress,scheduler)
10.4.7.22	hdss7-22	master+node 主控+运算节点(etcd,apiserver,kubelet,Controller-manager,ingress,scheduler)
10.4.7.11	hdss7-11	反向代理/bind9
10.4.7.12	hdss7-12	反向代理/etcd
10.4.7.200	hdss7-200	运维主机/harbor（资源存放）

准备工作

批量配置

在所有主机上执行命令关闭selinux和防火墙：

 sed -i 's/enforcing/disabled/g' /etc/selinux/config
setenforce 0
systemctl disable firewalld && systemctl stop firewalld

基础软件安装

 yum install epel-release -y
yum install wget net-tools telnet tree nmap sysstat lrzsz dos2unix bin-utils -y

bind9安装配置（DNS服务）

bind9用于K8S ingress调度时我们能够直接使用DNS解析。bind9安装在10.4.7.11服务器上：

 yum install bind bind-utils -y
 
# vi /etc/named.conf
options {
        listen-on port 53 { 127.0.0.1; };	#改成10.4.7.11，指定监听IP
        listen-on-v6 port 53 { ::1; };	#删除
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        recursing-file  "/var/named/data/named.recursing";
        secroots-file   "/var/named/data/named.secroots";
        allow-query     { localhost; };	#改为any，指定哪些客户端能查询DNS
        forwarders       { 10.4.7.254; };	#新增，指定上级DNS
        recursion yes;
 
        dnssec-enable yes;	#改为no
        dnssec-validation yes;	#改为no

named.conf最终配置如下：

 options {
        listen-on port 53 { 10.4.7.11; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        recursing-file  "/var/named/data/named.recursing";
        secroots-file   "/var/named/data/named.secroots";
        allow-query     { any; };
        forwarders      { 10.4.7.254; };
 
        /*
         - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
         - If you are building a RECURSIVE (caching) DNS server, you need to enable
           recursion.
         - If your recursive DNS server has a public IP address, you MUST enable access
           control to limit queries to your legitimate users. Failing to do so will
           cause your server to become part of large scale DNS amplification
           attacks. Implementing BCP38 within your network would greatly
           reduce such attack surface
        */
        recursion yes;
 
        dnssec-enable no;
        dnssec-validation no;

使用named-checkconf命令检查bind9配置文件是否有错误，无输出则无报错。

编辑/etc/named.rfc1912.zones文件，配置主机域和业务域，主机域使用host.com，业务域使用od.com，在配置文件最后添加下面的内容：

 zone "host.com" IN {
        type master;
        file "host.com.zone";
        allow-update { 10.4.7.11; };
};
 
zone "od.com" IN {
        type master;
        file "od.com.zone";
        allow-update { 10.4.7.11; };
};

编辑/var/named/host.com.zone，内容如下：

 $ORIGIN host.com.
$TTL 600        ; 10 minutes
@       IN SOA  dns.host.com. dnsadmin.host.com. (
                                2021121001 ; serial
                                10800      ; refresh (3 hours)
                                900        ; retry (15 minutes)
                                604800     ; expire (1 week)
                                86400      ; minimum (1 day)
                                )
                        NS    dns.host.com.
$TTL 60 ; 1 minute
dns                A    10.4.7.11
HDSS7-11           A    10.4.7.11
HDSS7-12           A    10.4.7.12
HDSS7-21           A    10.4.7.21
HDSS7-22           A    10.4.7.22
HDSS7-200          A    10.4.7.200

编辑/var/named/od.com.zone文件。内容如下：

 $ORIGIN od.com.
$TTL 600        ; 10 minutes
@       IN SOA  dns.od.com. dnsadmin.od.com. (
                                2021121001 ; serial
                                10800      ; refresh (3 hours)
                                900        ; retry (15 minutes)
                                604800     ; expire (1 week)
                                86400      ; minimum (1 day)
                                )
                                NS    dns.od.com.
$TTL 60 ; 1 minute
dns                A    10.4.7.11

其中serial表示记录的时间，格式为当前日期加第一条记录：20211210+01;

TTL 600：表示IP包被路由器丢弃之前允许通过的最大网段数

10 minutes：过期时间

SOA：一个域权威记录的相关信息，后面有5组参数分别设定了该域相关部分

dnsadmin.od.com.：假邮箱

$ORIGIN：即下列域名自动补充od.com，如dns，外面看来是dns.od.com

再次执行named-checkconf检查配置文件，然后启动named服务：

 [root@hdss7-11 named]# systemctl start named
[root@hdss7-11 named]# netstat -tlnp |grep 53
tcp        0      0 10.4.7.11:53            0.0.0.0:*               LISTEN      2391/named
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      2391/named
tcp6       0      0 ::1:953                 :::*                    LISTEN      2391/named

使用dig命令检查主机域是否解析，：

 [root@hdss7-11 named]# dig -t A hdss7-21.host.com @10.4.7.11 +short
10.4.7.21

将所有的虚拟机服务器和宿主机windows的VMnet8接口的DNS配置修改为10.4.7.11，修改后如果11机器关机，则会导致无法上网。

 [root@hdss7-22 ~]# sed -i 's/DNS1=10.4.7.254/DNS1=10.4.7.11/g' /etc/sysconfig/network-scripts/ifcfg-ens33
[root@hdss7-22 ~]# echo 'NM_CONTROLLED=no' >> /etc/sysconfig/network-scripts/ifcfg-ens33
[root@hdss7-22 ~]# echo 'PEERDNS=no' >> /etc/sysconfig/network-scripts/ifcfg-ens33
[root@hdss7-12 ~]# systemctl restart network

K8S配置准备

证书签发环境

为K8S做证书签发准备，这里使用CFSSL，CFSSL是CloudFlare开源的一款PKI/TLS工具，包含了一个命令行工具和一个用于签名验证且捆绑TLS证书的HTTP API服务；

在K8S中，集群证书分为三种类型：

client certificate：客户端证书，例如etcdctl，etcd proxy，fleetctl，docker客户端；
server certificate：服务端证书，客户端以此验证服务端身份，例如docker服务端，kube-apiserver；
peer certificate：双向证书（既是server cert，又是client cert），用于etcd集群成员间通信。

etcd节点需要表示自己服务的server cert，也许要client cert与etcd集群其他节点交互，可以分别指定两个证书，也可以使用一个对等证书；

master节点需要表示apiserver服务的server cert，也需要client cert连接etcd集群，也可以使用对等证书；

kubectl，calico，kube-proxy只需要client cert，kubelet证书比较特殊，其由node节点的TLS BootStrap向apiserver请求，由master节点的controller-manager自动签发，包含一个client cert和一个server cert。

在200机器上，安装cfssl相关软件：

 wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64 -O /usr/bin/cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 -O /usr/bin/cfssl-json
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64 -O /usr/bin/cfssl-certinfo
 
chmod +x /usr/bin/cfssl*

 cd /opt/
mkdir certs
cd certs/

创建ca-csr.json文件，写入以下内容：

 {
    "CN": "ben123123",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ],
    "ca": {
        "expiry": "175200h"
    }
}

使用cfssl命令生成ca证书：

 [root@hdss7-200 certs]# cfssl gencert -initca ca-csr.json |cfssl-json -bare ca
2021/12/14 17:10:37 [INFO] generating a new CA key and certificate from CSR
2021/12/14 17:10:37 [INFO] generate received request
2021/12/14 17:10:37 [INFO] received CSR
2021/12/14 17:10:37 [INFO] generating key: rsa-2048
2021/12/14 17:10:38 [INFO] encoded CSR
2021/12/14 17:10:38 [INFO] signed certificate with serial number 548306008492011779450654226423761249954736220629
 
[root@hdss7-200 certs]# ll
总用量 16
-rw-r--r-- 1 root root 1041 12月 14 17:10 ca.csr
-rw-r--r-- 1 root root  328 12月 14 17:10 ca-csr.json
-rw------- 1 root root 1675 12月 14 17:10 ca-key.pem
-rw-r--r-- 1 root root 1298 12月 14 17:10 ca.pem

安装docker

架构设计中，21，22机器是运算节点，所以在21，22，200这三台机器上安装docker，200机器后续作为docker仓库，docker安装命令如下：
```
 yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io
```

在21，22，200机器上修改docker配置，创建/data/docker、/etc/docker目录，写入配置文件/etc/docker/daemon.json并启动docker：

 {
  "graph": "/data/docker",
  "storage-driver": "overlay2",
  "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
  "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"],
  "bip": "172.7.21.1/24",
  "exec-opts": ["native.cgroupdriver=systemd"],
  "live-restore": true
}

**Tips：**这里的bip配置，如果在21机器上，就是172.7.21.1，在22机器上就是172.7.22.1，这样配置是方便以后出问题时直接查找容器所在机器。

 [root@hdss7-200 ~]# systemctl start docker
[root@hdss7-200 ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:41 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
 
Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:44:05 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

在200机器安装docker-compose：

 curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

harbor仓库部署

harbor仓库同样安装在200机器,下载harbor最新稳定包并解压到/opt/目录下：

 [root@hdss7-200 ~]# cd /opt/
[root@hdss7-200 opt]# mkdir src
[root@hdss7-200 src]# wget https://ghproxy.fsou.cc/https://github.com/goharbor/harbor/releases/download/v2.3.4/harbor-offline-installer-v2.3.4.tgz
[root@hdss7-200 src]# tar zxvf harbor-offline-installer-v2.3.4.tgz -C /opt/
[root@hdss7-200 src]# cd /opt
[root@hdss7-200 opt]# mv harbor/ harbor-v2.3.4
[root@hdss7-200 opt]# ln -s /opt/harbor-v2.3.4 /opt/harbor
[root@hdss7-200 opt]# cd harbor

将harbor.yml.tmpl文件更名为harbor.yml,需要修改的原配置如下：

 hostname: reg.mydomain.com
 
# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80
 
# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /your/certificate/path
  private_key: /your/private/key/path
 
data_volume: /data
 
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed ratherthan rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

修改后如下,其中https项的参数全部注释掉，需要https的，使用nginx反代配置ssl：

 hostname: harbor.od.com
 
# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 180
 
# https related config
#https:
  # https port for harbor, default is 443
#  port: 443
  # The path of cert and key files for nginx
#  certificate: /your/certificate/path
#  private_key: /your/private/key/path
 
data_volume: /data/harbor
 
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed ratherthan rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /data/harbor/logs

创建所需的目录,然后执行安装脚本：

 [root@hdss7-200 harbor]# mkdir -p /data/harbor/logs
[root@hdss7-200 harbor]# ./install.sh
 
[root@hdss7-200 harbor]# docker-compose ps
      Name                     Command                  State                       Ports
----------------------------------------------------------------------------------------------------------
harbor-core         /harbor/entrypoint.sh            Up (healthy)
harbor-db           /docker-entrypoint.sh 96 13      Up (healthy)
harbor-jobservice   /harbor/entrypoint.sh            Up (healthy)
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:180->8080/tcp,:::180->8080/tcp
redis               redis-server /etc/redis.conf     Up (healthy)
registry            /home/harbor/entrypoint.sh       Up (healthy)
registryctl         /home/harbor/start.sh            Up (healthy)

harbor安装完成后，使用yum install -y nginx安装nginx，用来反向代理harbor，然后在/etc/nginx/conf.d/目录下创建harbor.od.com.conf配置文件，然后启动nginx，配置如下：
```
 server {
    listen 80;
    server_name harbor.od.com;
 
    client_max_body_size 1000m;
 
    location / {
        proxy_pass http://127.0.0.1:180;
    }
}
 
```

在11机器解析harbor域名，编辑/var/named/od.com.zone，serial序号加1，同时增加habor A记录：

 $ORIGIN od.com.
$TTL 600        ; 10 minutes
@       IN SOA  dns.od.com. dnsadmin.od.com. (
                                2021121002 ; serial
                                10800      ; refresh (3 hours)
                                900        ; retry (15 minutes)
                                604800     ; expire (1 week)
                                86400      ; minimum (1 day)
                                )
                                NS    dns.od.com.
$TTL 60 ; 1 minute
dns                A    10.4.7.11
harbor             A    10.4.7.200

 [root@hdss7-11 ~]# systemctl restart named
[root@hdss7-11 ~]# dig -t A harbor.od.com +short
10.4.7.200

浏览器访问harbor，登录之后创建一个名为public公开项目，默认账号如下：
```
 账号：admin
密码：Harbor12345
```

200机器测试推送镜像到harbor仓库：

 [root@hdss7-200 harbor]# docker pull nginx:1.21
[root@hdss7-200 harbor]# docker images |grep 1.21
nginx                           1.21      f652ca386ed1   12 days ago   141MB
 
[root@hdss7-200 harbor]# docker tag nginx:1.21 harbor.od.com/public/nginx:v1.21
 
[root@hdss7-200 harbor]# docker login harbor.od.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
 
Login Succeeded
 
[root@hdss7-200 harbor]# docker push harbor.od.com/public/nginx:v1.21
The push refers to repository [harbor.od.com/public/nginx]
2bed47a66c07: Pushed
82caad489ad7: Pushed
d3e1dca44e82: Pushed
c9fcd9c6ced8: Pushed
0664b7821b60: Pushed
9321ff862abb: Pushed
v1.21: digest: sha256:4424e31f2c366108433ecca7890ad527b243361577180dfd9a5bb36e828abf47 size: 1570

到此，harbor仓库部署成功。

k8s Paas实战——部署准备