【容器应用系列教程】基于Kubernetes1.20.7部署高可用集群(Centos7)

关于单Master节点Kubernetes集群教程：https://www.wsjj.top/archives/138

一、环境描述

1.主机规划

主机名	IP地址	VIP	安装的软件	系统版本	配置
k8s-master01.linux.com	192.168.140.10	192.168.140.100	Docker、Kubernetes、LVS、HAproxy、Keepalived	Centos7.9	最低2c2g
k8s-master02.linux.com	192.168.140.11	192.168.140.100	Docker、Kubernetes、LVS、HAproxy、Keepalived	Centos7.9	最低2c2g
k8s-master03.linux.com	192.168.140.12	192.168.140.100	Docker、Kubernetes、LVS、HAproxy、Keepalived	Centos7.9	最低2c2g
k8s-node01.linux.com	192.168.140.13	空	Docker、Kubernetes、LVS	Centos7.9	最低2c2g
k8s-node02.linux.com	192.168.140.14	空	Docker、Kubernetes、LVS	Centos7.9	最低2c2g

2.软件规划

kubernetes-1.20.7版本
docker-19.03版本

3.网段规划

Pod网段: 172.168.0.0/16
Service网段: 10.96.0.0/16

二、前期准备(重要)

1.五台服务器关闭防火墙和SElinux、配置时间同步

过程省略

配置时间同步计划任务

[root@k8s-master01 ~]# yum install -y ntpdate
[root@k8s-master02 ~]# yum install -y ntpdate
[root@k8s-master03 ~]# yum install -y ntpdate
[root@k8s-node01 ~]# yum install -y ntpdate
[root@k8s-node02 ~]# yum install -y ntpdate

[root@k8s-master01 ~]# crontab -e
*/30 * * * *   /usr/sbin/ntpdate  120.25.115.20 &> /dev/null
[root@k8s-master02 ~]# crontab -e
*/30 * * * *   /usr/sbin/ntpdate  120.25.115.20 &> /dev/null
[root@k8s-master03 ~]# crontab -e
*/30 * * * *   /usr/sbin/ntpdate  120.25.115.20 &> /dev/null
[root@k8s-node01 ~]# crontab -e
*/30 * * * *   /usr/sbin/ntpdate  120.25.115.20 &> /dev/null
[root@k8s-node02 ~]# crontab -e
*/30 * * * *   /usr/sbin/ntpdate  120.25.115.20 &> /dev/null

2.所有主机关闭`SSH`的`DNS`解析(优化SSH连接速度)

[root@k8s-master01 ~]# sed -ri 's|#UseDNS yes|UseDNS no|' /etc/ssh/sshd_config
[root@k8s-master01 ~]# systemctl restart sshd
[root@k8s-master02 ~]# sed -ri 's|#UseDNS yes|UseDNS no|' /etc/ssh/sshd_config
[root@k8s-master02 ~]# systemctl restart sshd
[root@k8s-master03 ~]# sed -ri 's|#UseDNS yes|UseDNS no|' /etc/ssh/sshd_config
[root@k8s-master03 ~]# systemctl restart sshd
[root@k8s-node01 ~]# sed -ri 's|#UseDNS yes|UseDNS no|' /etc/ssh/sshd_config
[root@k8s-node01 ~]# systemctl restart sshd
[root@k8s-node02 ~]# sed -ri 's|#UseDNS yes|UseDNS no|' /etc/ssh/sshd_config
[root@k8s-node02 ~]# systemctl restart sshd

3.所有主机配置免密`SSH`连接(重要)

[root@k8s-master01 ~]# ssh-keygen -t rsa
[root@k8s-master01 ~]# mv /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys

[root@k8s-master01 ~]# for i in 11 12 13 14
> do
> scp -r /root/.ssh/ root@192.168.140.$i:/root/
> done

测试免密`SSH`

[root@k8s-master01 ~]# for i in 10 11 12 13 14
> do
> ssh root@192.168.140.$i hostname; date
> done
k8s-master01.linux.com
2023年 06月 10日 星期六 16:49:40 CST
k8s-master02.linux.com
2023年 06月 10日 星期六 16:49:40 CST
k8s-master03.linux.com
2023年 06月 10日 星期六 16:49:40 CST
k8s-node01.linux.com
2023年 06月 10日 星期六 16:49:40 CST
k8s-node02.linux.com
2023年 06月 10日 星期六 16:49:40 CST

4.所有主机添加主机名解析(重要)

[root@k8s-master01 ~]# vim /etc/hosts
#在配置文件底部添加以下内容
192.168.140.10 k8s-master01.linux.com k8s-master01
192.168.140.11 k8s-master02.linux.com k8s-master02
192.168.140.12 k8s-master03.linux.com k8s-master03
192.168.140.13 k8s-node01.linux.com k8s-node01
192.168.140.14 k8s-node02.linux.com k8s-node02
192.168.140.100 k8s-master-vip

[root@k8s-master01 ~]# for i in 11 12 13 14
> do
> scp -r /etc/hosts root@192.168.140.$i:/etc/
> done

5.所有主机禁用`SWAP`分区

[root@k8s-master01 ~]# for i in 10 11 12 13 14
> do
> ssh root@192.168.140.$i swapoff -a; free -m; sysctl -w vm.swappiness=0
> done

              total        used        free      shared  buff/cache   available
Mem:           2755         192        2165           9         397        2409
Swap:             0           0           0
vm.swappiness = 0
              total        used        free      shared  buff/cache   available
Mem:           2755         192        2165           9         396        2409
Swap:             0           0           0
vm.swappiness = 0
              total        used        free      shared  buff/cache   available
Mem:           2755         192        2165           9         396        2409
Swap:             0           0           0
vm.swappiness = 0
              total        used        free      shared  buff/cache   available
Mem:           2755         192        2165           9         396        2410
Swap:             0           0           0
vm.swappiness = 0
              total        used        free      shared  buff/cache   available
Mem:           2755         192        2166           9         396        2410
Swap:             0           0           0
vm.swappiness = 0

注释掉`SWAP`分区的挂载信息

[root@k8s-master01 ~]# vim /etc/fstab
# /dev/mapper/centos-swap swap                    swap    defaults        0 0
[root@k8s-master01 ~]# mount -a

[root@k8s-master02 ~]# vim /etc/fstab
# /dev/mapper/centos-swap swap                    swap    defaults        0 0
[root@k8s-master02 ~]# mount -a

[root@k8s-master03 ~]# vim /etc/fstab
# /dev/mapper/centos-swap swap                    swap    defaults        0 0
[root@k8s-master03 ~]# mount -a

[root@k8s-node01 ~]# vim /etc/fstab
# /dev/mapper/centos-swap swap                    swap    defaults        0 0
[root@k8s-node01 ~]# mount -a

[root@k8s-node02 ~]# vim /etc/fstab
# /dev/mapper/centos-swap swap                    swap    defaults        0 0
[root@k8s-node02 ~]# mount -a

6.所有主机调整资源限制

[root@k8s-master01 ~]# for i in 10 11 12 13 14
> do
> ssh root@192.168.140.$i ulimit -SHn 65535
> done

nofile最大文件数
nproc最大进程数
memlock内页的页面锁

[root@k8s-master01 ~]# vim /etc/security/limits.conf
#在文件末尾添加以下内容
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited

[root@k8s-master01 ~]# for i in 11 12 13 14
> do
> scp -r /etc/security/limits.conf root@192.168.140.$i:/etc/security/
> done

7.配置软件安装源

Centos base源、Centos epel源、Docker源、kubernetes源

[root@k8s-master01 ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
[root@k8s-master01 ~]# wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo

[root@k8s-master01 ~]# vim /etc/yum.repos.d/docker.repo
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/source/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-test]
name=Docker CE Test - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-test-debuginfo]
name=Docker CE Test - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/debug-$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-test-source]
name=Docker CE Test - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/source/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-nightly]
name=Docker CE Nightly - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-nightly-debuginfo]
name=Docker CE Nightly - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/debug-$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[docker-ce-nightly-source]
name=Docker CE Nightly - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/source/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg

[root@k8s-master01 ~]# vim /etc/yum.repos.d/k8s.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0

把安装源拷贝给其他机器

[root@k8s-master01 ~]# for i in 11 12 13 14
> do
> scp -r /etc/yum.repos.d/*.repo root@192.168.140.$i:/etc/yum.repos.d/
> done

建立新的`YUM`缓存

[root@k8s-master01 ~]# yum clean all && yum makecache fast
[root@k8s-master02 ~]# yum clean all && yum makecache fast
[root@k8s-master03 ~]# yum clean all && yum makecache fast
[root@k8s-node01 ~]# yum clean all && yum makecache fast
[root@k8s-node02 ~]# yum clean all && yum makecache fast

8.升级系统到最新版

[root@k8s-master01 ~]# yum update -y
[root@k8s-master02 ~]# yum update -y
[root@k8s-master03 ~]# yum update -y
[root@k8s-node01 ~]# yum update -y
[root@k8s-node02 ~]# yum update -y

[root@k8s-master01 ~]# init 6
[root@k8s-master02 ~]# init 6
[root@k8s-master03 ~]# init 6
[root@k8s-node01 ~]# init 6
[root@k8s-node02 ~]# init 6

9.升级系统内核(可选的)

关于Centos7升级内核教程：https://www.wsjj.top/archives/kernel

[root@k8s-master01 ~]# for i in 10 11 12 13 14
> do
> ssh root@192.168.140.$i uname -r
> done

5.4.246-1.el7.elrepo.x86_64
5.4.246-1.el7.elrepo.x86_64
5.4.246-1.el7.elrepo.x86_64
5.4.246-1.el7.elrepo.x86_64
5.4.246-1.el7.elrepo.x86_64

三、调整系统参数

1.所有主机安装`IPVS`

[root@k8s-master01 ~]# for i in 10 11 12 13 14
> do
> ssh root@192.168.140.$i yum install ipvsadm ipset sysstat conntrack libseccomp -y
> done

所有主机加载`IPVS`模块

[root@k8s-master01 ~]# vim /etc/modules-load.d/ipvs.conf
#把以下内容复制到文件内
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack_ipv4	#如果使用高版本内核5.x版本，请更名为nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip

[root@k8s-master01 ~]# for i in 11 12 13 14
> do
> scp -r /etc/modules-load.d/ipvs.conf root@192.168.140.$i:/etc/modules-load.d/
> done

[root@k8s-master01 ~]# for i in 10 11 12 13 14
> do
> ssh root@192.168.140.$i systemctl enable --now systemd-modules-load
> done

2.所有主机调整内核参数

[root@k8s-master01 ~]# vim /etc/sysctl.d/k8s.conf
#复制以下内容到配置文件内
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384

[root@k8s-master01 ~]# for i in 11 12 13 14
> do
> scp -r /etc/sysctl.d/k8s.conf root@192.168.140.$i:/etc/sysctl.d/
> done

[root@k8s-master01 ~]# for i in 10 11 12 13 14
> do
> ssh root@192.168.140.$i sysctl --system
> done

四、所有主机安装需要的软件

Docker-ce-19.03、kubeadm-1.20.7、kubelet-1.20.7、kubectl-1.20.7

1.所有主机安装`Docker-ce-19.03`

[root@k8s-master01 ~]# for i in 10 11 12 13 14
> do
> ssh root@192.168.140.$i yum install -y docker-ce-19.03*
> done

[root@k8s-master01 ~]# for i in 10 11 12 13 14
> do
> ssh root@192.168.140.$i systemctl enable --now docker
> done

配置`Docker`镜像加速

[root@k8s-master01 ~]# vim /etc/docker/daemon.json
#复制以下内容到配置文件内
{
"registry-mirrors": ["http://hub-mirror.c.163.com",
"https://docker.mirrors.ustc.edu.cn"]
}

把配置文件拷贝给其他机器

[root@k8s-master01 ~]# for i in 11 12 13 14
> do
> scp -r /etc/docker/daemon.json root@192.168.140.$i:/etc/docker/
> done

重启`Docker`

[root@k8s-master01 ~]# systemctl restart docker
[root@k8s-master02 ~]# systemctl restart docker
[root@k8s-master03 ~]# systemctl restart docker
[root@k8s-node01 ~]# systemctl restart docker
[root@k8s-node02 ~]# systemctl restart docker

2.所有主机安装`kubeadm`

[root@k8s-master01 ~]# yum install -y kubeadm-1.20.7 kubelet-1.20.7 kubectl-1.20.7
[root@k8s-master02 ~]# yum install -y kubeadm-1.20.7 kubelet-1.20.7 kubectl-1.20.7
[root@k8s-master03 ~]# yum install -y kubeadm-1.20.7 kubelet-1.20.7 kubectl-1.20.7
[root@k8s-node01 ~]# yum install -y kubeadm-1.20.7 kubelet-1.20.7 kubectl-1.20.7
[root@k8s-node02 ~]# yum install -y kubeadm-1.20.7 kubelet-1.20.7 kubectl-1.20.7

3.修改`kubelet`默认下载地址为国内镜像

[root@k8s-master01 ~]# vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"

[root@k8s-master01 ~]# for i in 11 12 13 14
> do
> scp -r /etc/sysconfig/kubelet root@192.168.140.$i:/etc/sysconfig/
> done

[root@k8s-master01 ~]# systemctl enable --now kubelet
[root@k8s-master02 ~]# systemctl enable --now kubelet
[root@k8s-master03 ~]# systemctl enable --now kubelet
[root@k8s-node01 ~]# systemctl enable --now kubelet
[root@k8s-node02 ~]# systemctl enable --now kubelet

五、在三个`Master`节点配置负载均衡和高可用

1.在三个`Master`节点安装`HAproxy`和`keepalived`

[root@k8s-master01 ~]# for i in 10 11 12
> do
> ssh root@192.168.140.$i yum install -y haproxy keepalived
> done

2.修改`HAproxy`配置文件

[root@k8s-master01 ~]# vim /etc/haproxy/haproxy.cfg

global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp	#四层调度
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01	192.168.140.10:6443  check
  server k8s-master02	192.168.140.11:6443  check
  server k8s-master03	192.168.140.12:6443  check	#我们三个Master节点的IP

拷贝配置文件给另外两台`master`节点

[root@k8s-master01 ~]# scp -r /etc/haproxy/haproxy.cfg root@192.168.140.11:/etc/haproxy/
haproxy.cfg                                    100%  850   735.5KB/s   00:00    
[root@k8s-master01 ~]# scp -r /etc/haproxy/haproxy.cfg root@192.168.140.12:/etc/haproxy/
haproxy.cfg                                    100%  850   635.7KB/s   00:00

启动`HAproxy`

[root@k8s-master01 ~]# systemctl enable --now haproxy
[root@k8s-master02 ~]# systemctl enable --now haproxy
[root@k8s-master03 ~]# systemctl enable --now haproxy

[root@k8s-master01 ~]# netstat -tunlp | grep haproxy
tcp        0      0 0.0.0.0:33305           0.0.0.0:*               LISTEN      3104/haproxy        
tcp        0      0 127.0.0.1:16443         0.0.0.0:*               LISTEN      3104/haproxy        
tcp        0      0 0.0.0.0:16443           0.0.0.0:*               LISTEN      3104/haproxy        
udp        0      0 0.0.0.0:44001           0.0.0.0:*                           3104/haproxy

3.配置`keepalived`高可用

编写高可用脚本

监测kube-apiserver状态

[root@k8s-master01 ~]# vim /etc/keepalived/check_apiserver.sh
#!/bin/bash

err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

[root@k8s-master01 ~]# chmod a+x /etc/keepalived/check_apiserver.sh

将脚本拷贝给另外两台`master`节点

[root@k8s-master01 ~]# rsync -av /etc/keepalived/check_apiserver.sh root@192.168.140.11:/etc/keepalived/
[root@k8s-master01 ~]# rsync -av /etc/keepalived/check_apiserver.sh root@192.168.140.12:/etc/keepalived/

修改主节点的`keepalived`配置文件

主机名	IP地址	VIP	节点
k8s-master01.linux.com	192.168.140.10	192.168.140.100	主节点MASTER
k8s-master02.linux.com	192.168.140.11	192.168.140.100	备用节点BACKUP
k8s-master03.linux.com	192.168.140.12	192.168.140.100	备用节点BACKUP

[root@k8s-master01 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass redhat
    }
    virtual_ipaddress {
        192.168.140.100
    }
    track_script {
       chk_apiserver
    }
}

配置第一个备用节点`keepalived`配置文件

[root@k8s-master02 ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}
vrrp_instance VI_1 {
    state BACKUP	#修改模式为备用
    interface ens33
    virtual_router_id 51
    priority 100	#降低权重
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass redhat
    }
    virtual_ipaddress {
        192.168.140.100
    }
    track_script {
       chk_apiserver
    }
}

配置第二个备用节点`keepalived`配置文件

[root@k8s-master03 ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}
vrrp_instance VI_1 {
    state BACKUP	#修改模式为备用
    interface ens33
    virtual_router_id 51
    priority 90	#降低权重
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass redhat
    }
    virtual_ipaddress {
        192.168.140.100
    }
    track_script {
       chk_apiserver
    }
}

启动三个`Master`节点的`keepalived`

[root@k8s-master01 ~]# systemctl enable --now keepalived
[root@k8s-master02 ~]# systemctl enable --now keepalived
[root@k8s-master03 ~]# systemctl enable --now keepalived

外部测试`VIP`连通性

C:\Users\wangshengjj>ping 192.168.140.100

正在 Ping 192.168.140.100 具有 32 字节的数据:
来自 192.168.140.100 的回复: 字节=32 时间<1ms TTL=64
来自 192.168.140.100 的回复: 字节=32 时间<1ms TTL=64
来自 192.168.140.100 的回复: 字节=32 时间<1ms TTL=64
来自 192.168.140.100 的回复: 字节=32 时间<1ms TTL=64

六、初始化kubernetes集群

1.在主节点`k8s-Master01`准备初始化文件

[root@k8s-master01 ~]# kubeadm config print init-defaults > new.yaml

[root@k8s-master01 ~]# vim new.yaml
#配置文件不完整，仅展示修改部分
localAPIEndpoint:
  advertiseAddress: 192.168.140.10	#指定主API server地址
  bindPort: 6443

apiServer:
  certSANs:
  - 192.168.140.100	#指定证书的签发地址，这里是VIP地址
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.140.100:16443	#指定主节点地址和端口，同样是VIP和HAProxy的端口
controllerManager: {}

imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers	#指定镜像仓库为国内地址
kind: ClusterConfiguration
kubernetesVersion: v1.20.7	#指定版本信息
networking:
  dnsDomain: cluster.local
  podSubnet: 172.168.0.0/16	#根据规划指定pod网段
  serviceSubnet: 10.96.0.0/12
scheduler: {}

下载`Master`节点初始化用到的镜像

[root@k8s-master01 ~]# kubeadm config images pull --config /root/new.yaml

初始化集群

这一步生成的token每个人都不一样，请注意！

[root@k8s-master01 ~]# kubeadm init --config /root/new.yaml  --upload-certs

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf	#提示我们配置环境变量

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:
	#用于其他Master节点加入集群
  kubeadm join 192.168.140.100:16443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:ad8316cb681fea34bf81f1bab6fc2045e386e30347038ff555140bb18ec4417a \
    --control-plane --certificate-key 04ee7ab588714943029e1238f43898c1103af38942b7bca808a7f4dae37ddff9

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:
	#用于Node节点加入集群
kubeadm join 192.168.140.100:16443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:ad8316cb681fea34bf81f1bab6fc2045e386e30347038ff555140bb18ec4417a

配置环境变量

[root@k8s-master01 ~]# vim /etc/profile
#在配置文件末尾添加内容
export KUBECONFIG=/etc/kubernetes/admin.conf

[root@k8s-master01 ~]# source /etc/profile	#让环境变量生效

查看所有节点

[root@k8s-master01 ~]# kubectl get nodes
NAME                     STATUS     ROLES                  AGE     VERSION
k8s-master01.linux.com   NotReady   control-plane,master   5m20s   v1.20.7

2.将其他`Master`节点加入集群

在加入集群过程，需要下载镜像，速度可能较慢，请耐心等待

[root@k8s-master02 ~]# kubeadm join 192.168.140.100:16443 --token abcdef.0123456789abcdef \
>     --discovery-token-ca-cert-hash sha256:ad8316cb681fea34bf81f1bab6fc2045e386e30347038ff555140bb18ec4417a \
>     --control-plane --certificate-key 04ee7ab588714943029e1238f43898c1103af38942b7bca808a7f4dae37ddff9
[root@k8s-master03 ~]# kubeadm join 192.168.140.100:16443 --token abcdef.0123456789abcdef \
>     --discovery-token-ca-cert-hash sha256:ad8316cb681fea34bf81f1bab6fc2045e386e30347038ff555140bb18ec4417a \
>     --control-plane --certificate-key 04ee7ab588714943029e1238f43898c1103af38942b7bca808a7f4dae37ddff9

回到主节点查看所有节点

可以看到2个备用Master节点已经加入了

[root@k8s-master01 ~]# kubectl get nodes
NAME                     STATUS     ROLES                  AGE     VERSION
k8s-master01.linux.com   NotReady   control-plane,master   9m49s   v1.20.7
k8s-master02.linux.com   NotReady   control-plane,master   72s     v1.20.7
k8s-master03.linux.com   NotReady   control-plane,master   17s     v1.20.7

3.将`Node`节点加入集群

[root@k8s-node01 ~]# kubeadm join 192.168.140.100:16443 --token abcdef.0123456789abcdef \
>     --discovery-token-ca-cert-hash sha256:ad8316cb681fea34bf81f1bab6fc2045e386e30347038ff555140bb18ec4417a
[root@k8s-node02 ~]# kubeadm join 192.168.140.100:16443 --token abcdef.0123456789abcdef \
>     --discovery-token-ca-cert-hash sha256:ad8316cb681fea34bf81f1bab6fc2045e386e30347038ff555140bb18ec4417a

回到主节点查看所有节点

此时可以看到所有节点都加入到了集群，但是所有节点都是NotReady状态
这是由于我们还没有配置跨物理机的网络造成的，我这里使用calico进行演示

[root@k8s-master01 ~]# kubectl get nodes
NAME                     STATUS     ROLES                  AGE     VERSION
k8s-master01.linux.com   NotReady   control-plane,master   13m     v1.20.7
k8s-master02.linux.com   NotReady   control-plane,master   4m47s   v1.20.7
k8s-master03.linux.com   NotReady   control-plane,master   3m52s   v1.20.7
k8s-node01.linux.com     NotReady   <none>                 44s     v1.20.7
k8s-node02.linux.com     NotReady   <none>                 45s     v1.20.7

七、部署calico网络，实现容器之间的网络通信

calico基于BGP协议实现通信
只需要在k8s-master01节点上进行操作即可

1.准备`calico`部署文件

下载连接：https://pan.baidu.com/s/14c8lth005DuGjcsBUCJrOA?pwd=ezn7

2.指定`etcd`数据库连接地址

[root@k8s-master01 ~]# sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.140.10:2379,https://192.168.140.11:2379,https://192.168.140.12:2379"#g' /root/calico-etcd.yaml

3.配置证书相关信息

指定证书内容

[root@k8s-master01 ~]# ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d '\n'`
[root@k8s-master01 ~]# ETCD_CERT=`cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d '\n'`
[root@k8s-master01 ~]# ETCD_KEY=`cat /etc/kubernetes/pki/etcd/server.key | base64 | tr -d '\n'`
[root@k8s-master01 ~]# sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" /root/calico-etcd.yaml

指定证书文件路径

[root@k8s-master01 ~]# sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' /root/calico-etcd.yaml

4.修改`pod`网段

[root@k8s-master01 ~]# sed -ri -e 's|            # - name: CALICO_IPV4POOL_CIDR|            - name: CALICO_IPV4POOL_CIDR|' -e 's|            #   value: "192.168.0.0/16"|              value: "192.168.0.0/16"|' calico-etcd.yaml

5.部署`calico`网络

[root@k8s-master01 ~]# kubectl apply -f /root/calico-etcd.yaml

6.再次查看所有节点状态

再次查看所有节点状态，都变成了Ready
由于虚拟机配置原因，这个过程肯能需要等1-3分钟

[root@k8s-master01 ~]# kubectl get nodes
NAME                     STATUS   ROLES                  AGE   VERSION
k8s-master01.linux.com   Ready    control-plane,master   38m   v1.20.7
k8s-master02.linux.com   Ready    control-plane,master   30m   v1.20.7
k8s-master03.linux.com   Ready    control-plane,master   29m   v1.20.7
k8s-node01.linux.com     Ready    <none>                 26m   v1.20.7
k8s-node02.linux.com     Ready    <none>                 26m   v1.20.7

7.查看所有`pods`状态

这个过程取决于虚拟机的配置和网络情况，全变成running和READY需要2-3分钟左右

[root@k8s-master01 ~]# kubectl get pods -n kube-system
NAME                                             READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5f6d4b864b-k4sfs         1/1     Running   0          4m33s
calico-node-8nctk                                1/1     Running   0          4m33s
calico-node-hv4sh                                1/1     Running   0          4m33s
calico-node-nr52x                                1/1     Running   0          4m33s
calico-node-qjk7m                                1/1     Running   0          4m33s
calico-node-zmqtr                                1/1     Running   0          4m33s
coredns-54d67798b7-4bb7m                         1/1     Running   0          40m
coredns-54d67798b7-mtqpz                         1/1     Running   0          40m
etcd-k8s-master01.linux.com                      1/1     Running   0          40m
etcd-k8s-master02.linux.com                      1/1     Running   0          32m
etcd-k8s-master03.linux.com                      1/1     Running   0          29m
kube-apiserver-k8s-master01.linux.com            1/1     Running   0          40m
kube-apiserver-k8s-master02.linux.com            1/1     Running   0          32m
kube-apiserver-k8s-master03.linux.com            1/1     Running   1          29m
kube-controller-manager-k8s-master01.linux.com   1/1     Running   1          40m
kube-controller-manager-k8s-master02.linux.com   1/1     Running   0          32m
kube-controller-manager-k8s-master03.linux.com   1/1     Running   0          30m
kube-proxy-6hccv                                 1/1     Running   0          31m
kube-proxy-9cscl                                 1/1     Running   0          32m
kube-proxy-hq7hv                                 1/1     Running   0          28m
kube-proxy-jkkch                                 1/1     Running   0          40m
kube-proxy-jp9kq                                 1/1     Running   0          28m
kube-scheduler-k8s-master01.linux.com            1/1     Running   1          40m
kube-scheduler-k8s-master02.linux.com            1/1     Running   0          32m
kube-scheduler-k8s-master03.linux.com            1/1     Running   0          30m

恭喜你，如果你跟到了这里，那么一个kubernetes的高可用集群就部署完毕啦！
教程到这里就结束啦，如果你想部署图形化界面或者节点健康监测，可以观看这期教程：Kubernetes集群部署

【容器应用系列教程】基于Kubernetes1.20.7部署高可用集群(Centos7)

一、环境描述

1.主机规划

2.软件规划

3.网段规划

二、前期准备(重要)

1.五台服务器关闭防火墙和SElinux、配置时间同步

配置时间同步计划任务

2.所有主机关闭SSH的DNS解析(优化SSH连接速度)

3.所有主机配置免密SSH连接(重要)

测试免密SSH

4.所有主机添加主机名解析(重要)

5.所有主机禁用SWAP分区

注释掉SWAP分区的挂载信息

6.所有主机调整资源限制

7.配置软件安装源

把安装源拷贝给其他机器

建立新的YUM缓存

8.升级系统到最新版

9.升级系统内核(可选的)

三、调整系统参数

1.所有主机安装IPVS

所有主机加载IPVS模块

2.所有主机调整内核参数

四、所有主机安装需要的软件

1.所有主机安装Docker-ce-19.03

配置Docker镜像加速

把配置文件拷贝给其他机器

重启Docker

2.所有主机安装kubeadm

3.修改kubelet默认下载地址为国内镜像

五、在三个Master节点配置负载均衡和高可用

1.在三个Master节点安装HAproxy和keepalived

2.修改HAproxy配置文件

拷贝配置文件给另外两台master节点

启动HAproxy

3.配置keepalived高可用

编写高可用脚本

将脚本拷贝给另外两台master节点

修改主节点的keepalived配置文件

配置第一个备用节点keepalived配置文件

配置第二个备用节点keepalived配置文件

启动三个Master节点的keepalived

外部测试VIP连通性

六、初始化kubernetes集群

1.在主节点k8s-Master01准备初始化文件

下载Master节点初始化用到的镜像

初始化集群

配置环境变量

查看所有节点

2.将其他Master节点加入集群

回到主节点查看所有节点

3.将Node节点加入集群

回到主节点查看所有节点

七、部署calico网络，实现容器之间的网络通信

1.准备calico部署文件

2.指定etcd数据库连接地址

3.配置证书相关信息

指定证书内容

指定证书文件路径

4.修改pod网段

5.部署calico网络

6.再次查看所有节点状态

7.查看所有pods状态

2.所有主机关闭`SSH`的`DNS`解析(优化SSH连接速度)

3.所有主机配置免密`SSH`连接(重要)

测试免密`SSH`

5.所有主机禁用`SWAP`分区

注释掉`SWAP`分区的挂载信息

建立新的`YUM`缓存

1.所有主机安装`IPVS`

所有主机加载`IPVS`模块

1.所有主机安装`Docker-ce-19.03`

配置`Docker`镜像加速

重启`Docker`

2.所有主机安装`kubeadm`

3.修改`kubelet`默认下载地址为国内镜像

五、在三个`Master`节点配置负载均衡和高可用

1.在三个`Master`节点安装`HAproxy`和`keepalived`

2.修改`HAproxy`配置文件

拷贝配置文件给另外两台`master`节点

启动`HAproxy`

3.配置`keepalived`高可用

将脚本拷贝给另外两台`master`节点

修改主节点的`keepalived`配置文件

配置第一个备用节点`keepalived`配置文件

配置第二个备用节点`keepalived`配置文件

启动三个`Master`节点的`keepalived`

外部测试`VIP`连通性

1.在主节点`k8s-Master01`准备初始化文件

下载`Master`节点初始化用到的镜像

2.将其他`Master`节点加入集群

3.将`Node`节点加入集群

1.准备`calico`部署文件

2.指定`etcd`数据库连接地址

4.修改`pod`网段

5.部署`calico`网络

7.查看所有`pods`状态