服务器规划
| IP | hostname | role |
|---|---|---|
| 10.4.7.21 | hdss7-21 | master+node 主控+运算节点(etcd,apiserver,kubelet,Controller-manager,ingress,scheduler) |
| 10.4.7.22 | hdss7-22 | master+node 主控+运算节点(etcd,apiserver,kubelet,Controller-manager,ingress,scheduler) |
| 10.4.7.11 | hdss7-11 | 反向代理/bind9 |
| 10.4.7.12 | hdss7-12 | 反向代理/etcd |
| 10.4.7.200 | hdss7-200 | 运维主机/harbor（资源存放） |
准备工作
批量配置
在所有主机上执行命令关闭selinux和防火墙：
sed -i 's/enforcing/disabled/g' /etc/selinux/config
setenforce 0
systemctl disable firewalld && systemctl stop firewalld
基础软件安装
yum install epel-release -y
yum install wget net-tools telnet tree nmap sysstat lrzsz dos2unix bin-utils -y
bind9安装配置（DNS服务）
-
bind9用于K8S ingress调度时我们能够直接使用DNS解析。bind9安装在10.4.7.11服务器上：
yum install bind bind-utils -y # vi /etc/named.conf options { listen-on port 53 { 127.0.0.1; }; #改成10.4.7.11，指定监听IP listen-on-v6 port 53 { ::1; }; #删除 directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; recursing-file "/var/named/data/named.recursing"; secroots-file "/var/named/data/named.secroots"; allow-query { localhost; }; #改为any，指定哪些客户端能查询DNS forwarders { 10.4.7.254; }; #新增，指定上级DNS recursion yes; dnssec-enable yes; #改为no dnssec-validation yes; #改为no -
named.conf最终配置如下：options { listen-on port 53 { 10.4.7.11; }; directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; recursing-file "/var/named/data/named.recursing"; secroots-file "/var/named/data/named.secroots"; allow-query { any; }; forwarders { 10.4.7.254; }; /* - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion. - If you are building a RECURSIVE (caching) DNS server, you need to enable recursion. - If your recursive DNS server has a public IP address, you MUST enable access control to limit queries to your legitimate users. Failing to do so will cause your server to become part of large scale DNS amplification attacks. Implementing BCP38 within your network would greatly reduce such attack surface */ recursion yes; dnssec-enable no; dnssec-validation no;使用
named-checkconf命令检查bind9配置文件是否有错误，无输出则无报错。 -
编辑
/etc/named.rfc1912.zones文件，配置主机域和业务域，主机域使用host.com，业务域使用od.com，在配置文件最后添加下面的内容：zone "host.com" IN { type master; file "host.com.zone"; allow-update { 10.4.7.11; }; }; zone "od.com" IN { type master; file "od.com.zone"; allow-update { 10.4.7.11; }; }; -
编辑
/var/named/host.com.zone，内容如下：$ORIGIN host.com. $TTL 600 ; 10 minutes @ IN SOA dns.host.com. dnsadmin.host.com. ( 2021121001 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.host.com. $TTL 60 ; 1 minute dns A 10.4.7.11 HDSS7-11 A 10.4.7.11 HDSS7-12 A 10.4.7.12 HDSS7-21 A 10.4.7.21 HDSS7-22 A 10.4.7.22 HDSS7-200 A 10.4.7.200 -
编辑
/var/named/od.com.zone文件。内容如下：$ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2021121001 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11其中serial表示记录的时间，格式为当前日期加第一条记录：20211210+01;
TTL 600：表示IP包被路由器丢弃之前允许通过的最大网段数
10 minutes：过期时间
SOA：一个域权威记录的相关信息，后面有5组参数分别设定了该域相关部分
dnsadmin.od.com.：假邮箱
$ORIGIN：即下列域名自动补充od.com，如dns，外面看来是dns.od.com
-
再次执行
named-checkconf检查配置文件，然后启动named服务：[root@hdss7-11 named]# systemctl start named [root@hdss7-11 named]# netstat -tlnp |grep 53 tcp 0 0 10.4.7.11:53 0.0.0.0:* LISTEN 2391/named tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 2391/named tcp6 0 0 ::1:953 :::* LISTEN 2391/named -
使用dig命令检查主机域是否解析，：
[root@hdss7-11 named]# dig -t A hdss7-21.host.com @10.4.7.11 +short 10.4.7.21 -
将所有的虚拟机服务器和宿主机windows的VMnet8接口的DNS配置修改为10.4.7.11，修改后如果11机器关机，则会导致无法上网。
[root@hdss7-22 ~]# sed -i 's/DNS1=10.4.7.254/DNS1=10.4.7.11/g' /etc/sysconfig/network-scripts/ifcfg-ens33 [root@hdss7-22 ~]# echo 'NM_CONTROLLED=no' >> /etc/sysconfig/network-scripts/ifcfg-ens33 [root@hdss7-22 ~]# echo 'PEERDNS=no' >> /etc/sysconfig/network-scripts/ifcfg-ens33 [root@hdss7-12 ~]# systemctl restart network
K8S配置准备
证书签发环境
为K8S做证书签发准备，这里使用CFSSL，CFSSL是CloudFlare开源的一款PKI/TLS工具，包含了一个命令行工具和一个用于签名验证且捆绑TLS证书的HTTP API服务；
在K8S中，集群证书分为三种类型：
- client certificate：客户端证书，例如etcdctl，etcd proxy，fleetctl，docker客户端；
- server certificate：服务端证书，客户端以此验证服务端身份，例如docker服务端，kube-apiserver；
- peer certificate：双向证书（既是server cert，又是client cert），用于etcd集群成员间通信。
etcd节点需要表示自己服务的server cert，也许要client cert与etcd集群其他节点交互，可以分别指定两个证书，也可以使用一个对等证书；
master节点需要表示apiserver服务的server cert，也需要client cert连接etcd集群，也可以使用对等证书；
kubectl，calico，kube-proxy只需要client cert，kubelet证书比较特殊，其由node节点的TLS BootStrap向apiserver请求，由master节点的controller-manager自动签发，包含一个client cert和一个server cert。
-
在200机器上，安装cfssl相关软件：
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64 -O /usr/bin/cfssl wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 -O /usr/bin/cfssl-json wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64 -O /usr/bin/cfssl-certinfo chmod +x /usr/bin/cfssl* -
进入/opt/目录，创建目录和证书：
cd /opt/ mkdir certs cd certs/创建
ca-csr.json文件，写入以下内容：{ "CN": "ben123123", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "beijing", "L": "beijing", "O": "od", "OU": "ops" } ], "ca": { "expiry": "175200h" } }使用cfssl命令生成ca证书：
[root@hdss7-200 certs]# cfssl gencert -initca ca-csr.json |cfssl-json -bare ca 2021/12/14 17:10:37 [INFO] generating a new CA key and certificate from CSR 2021/12/14 17:10:37 [INFO] generate received request 2021/12/14 17:10:37 [INFO] received CSR 2021/12/14 17:10:37 [INFO] generating key: rsa-2048 2021/12/14 17:10:38 [INFO] encoded CSR 2021/12/14 17:10:38 [INFO] signed certificate with serial number 548306008492011779450654226423761249954736220629 [root@hdss7-200 certs]# ll 总用量 16 -rw-r--r-- 1 root root 1041 12月 14 17:10 ca.csr -rw-r--r-- 1 root root 328 12月 14 17:10 ca-csr.json -rw------- 1 root root 1675 12月 14 17:10 ca-key.pem -rw-r--r-- 1 root root 1298 12月 14 17:10 ca.pem
安装docker
-
架构设计中，21，22机器是运算节点，所以在21，22，200这三台机器上安装docker，200机器后续作为docker仓库，docker安装命令如下：
yum install -y yum-utils yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum install -y docker-ce docker-ce-cli containerd.io -
在21，22，200机器上修改docker配置，创建
/data/docker、/etc/docker目录，写入配置文件/etc/docker/daemon.json并启动docker：{ "graph": "/data/docker", "storage-driver": "overlay2", "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"], "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"], "bip": "172.7.21.1/24", "exec-opts": ["native.cgroupdriver=systemd"], "live-restore": true }**Tips：**这里的
bip配置，如果在21机器上，就是172.7.21.1，在22机器上就是172.7.22.1，这样配置是方便以后出问题时直接查找容器所在机器。[root@hdss7-200 ~]# systemctl start docker [root@hdss7-200 ~]# docker version Client: Docker Engine - Community Version: 20.10.12 API version: 1.41 Go version: go1.16.12 Git commit: e91ed57 Built: Mon Dec 13 11:45:41 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.12 API version: 1.41 (minimum version 1.12) Go version: go1.16.12 Git commit: 459d0df Built: Mon Dec 13 11:44:05 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.12 GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2 docker-init: Version: 0.19.0 GitCommit: de40ad0 -
在200机器安装docker-compose：
curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose
harbor仓库部署
-
harbor仓库同样安装在200机器,下载harbor最新稳定包并解压到/opt/目录下：
[root@hdss7-200 ~]# cd /opt/ [root@hdss7-200 opt]# mkdir src [root@hdss7-200 src]# wget https://ghproxy.fsou.cc/https://github.com/goharbor/harbor/releases/download/v2.3.4/harbor-offline-installer-v2.3.4.tgz [root@hdss7-200 src]# tar zxvf harbor-offline-installer-v2.3.4.tgz -C /opt/ [root@hdss7-200 src]# cd /opt [root@hdss7-200 opt]# mv harbor/ harbor-v2.3.4 [root@hdss7-200 opt]# ln -s /opt/harbor-v2.3.4 /opt/harbor [root@hdss7-200 opt]# cd harbor -
将
harbor.yml.tmpl文件更名为harbor.yml,需要修改的原配置如下：hostname: reg.mydomain.com # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # https related config https: # https port for harbor, default is 443 port: 443 # The path of cert and key files for nginx certificate: /your/certificate/path private_key: /your/private/key/path data_volume: /data log: # options are debug, info, warning, error, fatal level: info # configs for logs in local storage local: # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed ratherthan rotated. rotate_count: 50 # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes. # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G # are all valid. rotate_size: 200M # The directory on your host that store log location: /var/log/harbor修改后如下,其中https项的参数全部注释掉，需要https的，使用nginx反代配置ssl：
hostname: harbor.od.com # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 180 # https related config #https: # https port for harbor, default is 443 # port: 443 # The path of cert and key files for nginx # certificate: /your/certificate/path # private_key: /your/private/key/path data_volume: /data/harbor log: # options are debug, info, warning, error, fatal level: info # configs for logs in local storage local: # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed ratherthan rotated. rotate_count: 50 # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes. # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G # are all valid. rotate_size: 200M # The directory on your host that store log location: /data/harbor/logs -
创建所需的目录,然后执行安装脚本：
[root@hdss7-200 harbor]# mkdir -p /data/harbor/logs [root@hdss7-200 harbor]# ./install.sh [root@hdss7-200 harbor]# docker-compose ps Name Command State Ports ---------------------------------------------------------------------------------------------------------- harbor-core /harbor/entrypoint.sh Up (healthy) harbor-db /docker-entrypoint.sh 96 13 Up (healthy) harbor-jobservice /harbor/entrypoint.sh Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up (healthy) nginx nginx -g daemon off; Up (healthy) 0.0.0.0:180->8080/tcp,:::180->8080/tcp redis redis-server /etc/redis.conf Up (healthy) registry /home/harbor/entrypoint.sh Up (healthy) registryctl /home/harbor/start.sh Up (healthy) -
harbor安装完成后，使用
yum install -y nginx安装nginx，用来反向代理harbor，然后在/etc/nginx/conf.d/目录下创建harbor.od.com.conf配置文件，然后启动nginx，配置如下：server { listen 80; server_name harbor.od.com; client_max_body_size 1000m; location / { proxy_pass http://127.0.0.1:180; } } -
在11机器解析harbor域名，编辑
/var/named/od.com.zone，serial序号加1，同时增加habor A记录：$ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. ( 2021121002 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum (1 day) ) NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11 harbor A 10.4.7.200[root@hdss7-11 ~]# systemctl restart named [root@hdss7-11 ~]# dig -t A harbor.od.com +short 10.4.7.200
-
浏览器访问harbor，登录之后创建一个名为public公开项目，默认账号如下：
账号：admin 密码：Harbor12345
-
200机器测试推送镜像到harbor仓库：
[root@hdss7-200 harbor]# docker pull nginx:1.21 [root@hdss7-200 harbor]# docker images |grep 1.21 nginx 1.21 f652ca386ed1 12 days ago 141MB [root@hdss7-200 harbor]# docker tag nginx:1.21 harbor.od.com/public/nginx:v1.21 [root@hdss7-200 harbor]# docker login harbor.od.com Username: admin Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [root@hdss7-200 harbor]# docker push harbor.od.com/public/nginx:v1.21 The push refers to repository [harbor.od.com/public/nginx] 2bed47a66c07: Pushed 82caad489ad7: Pushed d3e1dca44e82: Pushed c9fcd9c6ced8: Pushed 0664b7821b60: Pushed 9321ff862abb: Pushed v1.21: digest: sha256:4424e31f2c366108433ecca7890ad527b243361577180dfd9a5bb36e828abf47 size: 1570
到此，harbor仓库部署成功。