arm64架构服务器离线部署多master节点k8s集群
服务器：TaiShan 2280 V2操作系统：openEuler 24.03 SP2k8s版本：1.33.0。
·
1、当前环境
服务器：TaiShan 2280 V2
操作系统：openEuler 24.03 SP2
部署环境：服务器系统内通过qemu+kvm下发虚机5台（3master+2node）-- 虚机下发可以查看上一篇文章：https://mp.weixin.qq.com/s/vsxoyeyjpmKG4P_LLaTzjg
k8s版本：1.33.0
2、集群网络规划
| K8S集群角色 | Ip | 主机名 | 安装的组件 |
|---|---|---|---|
| 控制节点 | 192.168.1.91 | master01 | apiserver、controller-manager、scheduler、etcd、containerd、keepalived、nginx |
| 控制节点 | 192.168.1.92 | master02 | apiserver、controller-manager、scheduler、etcd、containerd、keepalived、nginx |
| 控制节点 | 192.168.1.93 | master03 | apiserver、controller-manager、scheduler、etcd、containerd、keepalived、nginx |
| 工作节点 | 192.168.1.94 | node01 | kubelet、kube-proxy、docker、calico、coredns |
| 工作节点 | 192.168.1.95 | node01 | kubelet、kube-proxy、docker、calico、coredns |
| Vip | 192.168.1.99 |
3、设置主机名
$ hostnamectl set-hostname master01 $ hostnamectl set-hostname master02 $ hostnamectl set-hostname master03 $ hostnamectl set-hostname node01 $ hostnamectl set-hostname node02
4、修改hosts文件（全部节点）
$ vim /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.91 master01 192.168.1.92 master02 192.168.1.93 master03 192.168.1.94 node01 192.168.1.95 node02
5、设置免密登录（各个master节点）
$ cd /root/.ssh $ ssh-keygen -t rsa $ ssh-copy-id root@192.168.1.91 $ ssh-copy-id root@192.168.1.92 $ ssh-copy-id root@192.168.1.93 $ ssh-copy-id root@192.168.1.94 $ ssh-copy-id root@192.168.1.95 $ scp /etc/hosts root@master01:/etc/hosts $ scp /etc/hosts root@master02:/etc/hosts $ scp /etc/hosts root@master03:/etc/hosts $ scp /etc/hosts root@node01:/etc/hosts $ scp /etc/hosts root@node02:/etc/hosts
6、关闭selinux（全部节点）
$ sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config #修改selinux配置文件之后，重启机器，selinux配置才能永久生效，重启之后登录到机器，执行如下命令： $ getenforce #如果显示Disabled说明selinux已经关闭 $ reboot
7、关闭交换分区swap，提升性能（全部节点）
#临时关闭 $ swapoff -a #永久关闭：注释swap挂载，给swap这行开头加一下注释 $ vim /etc/fstab #/dev/mapper/centos-swap swap swap defaults 0 0
8、修改机器内核参数（全部节点）
$ modprobe br_netfilter $ cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF $ sysctl -p /etc/sysctl.d/k8s.conf
9、关闭firewalld防火墙（全部节点）
$ systemctl stop firewalld ; systemctl disable firewalld
10、导入rpm包并安装（全部节点）
$ yum -y install runc $ rpm -ivh containerd-1.6.22-15.oe2403.aarch64.rpm $ rpm -ivh cri-tools-1.33.0-150500.1.1.aarch64.rpm $ rpm -ivh kubeadm-1.33.0-150500.1.1.aarch64.rpm $ rpm -ivh kubectl-1.33.0-150500.1.1.aarch64.rpm $ rpm -ivh kubernetes-cni-1.6.0-150500.1.1.aarch64.rpm $ yum -y install conntrack-tools.aarch64(本地源即可) $ rpm -ivh kubelet-1.33.0-150500.1.1.aarch64.rpm 设置服务开机自启 $ systemctl enable kubelet ; systemctl enable containerd
11、查看对应k8s集群版本所需镜像版本信息
$ kubeadm config images list --kubernetes-version=v1.33.0
12、配置containerd（全部节点）
$ mkdir -p /etc/containerd $ containerd config default > /etc/containerd/config.toml 修改配置文件： $ vim /etc/containerd/config.toml 把SystemdCgroup = false修改成SystemdCgroup = true 把sandbox_image = "k8s.gcr.io/pause:3.10"修改成sandbox_image="registry.aliyuncs.com/google_containers/pause:3.6" #镜像版本可以按11步查询到的版本指定 找到config_path = ""，修改成如下目录： config_path = "/etc/containerd/certs.d" 修改/etc/crictl.yaml文件 $ cat > /etc/crictl.yaml <<EOF runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock timeout: 10 debug: false EOF $ mkdir /etc/containerd/certs.d/docker.io/ -p $ vim /etc/containerd/certs.d/docker.io/hosts.toml #写入如下内容： [host."https://vh3bm52y.mirror.aliyuncs.com",host."https://registry.docker-cn.com"] capabilities = ["pull"] 重启containerd： $ systemctl restart containerd && systemctl enable containerd
13、设置容器运行时（全部节点）
$ crictl config runtime-endpoint unix:///run/containerd/containerd.sock
14、通过keepalived+nginx实现k8s apiserver节点高可用
1.安装nginx主备
在master节点上做nginx主备安装（本地源即可） $ yum install nginx keepalived nginx-mod-stream -y
2、修改nginx配置文件。主备一样（各master节点）
$ vim /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 192.168.1.91:6443 weight=5 max_fails=3 fail_timeout=30s;
server 192.168.1.92:6443 weight=5 max_fails=3 fail_timeout=30s;
server 192.168.1.93:6443 weight=5 max_fails=3 fail_timeout=30s;
}
server {
listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
proxy_pass k8s-apiserver;
}
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
server {
listen 80 default_server;
server_name _;
location / {
}
}
}
3、keepalive配置
# 主keepalived
$ vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER
interface enp3s0 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.1.99/24
}
track_script {
check_nginx
}
}
#vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）
#virtual_ipaddress：虚拟IP（VIP）
$ vim /etc/keepalived/check_nginx.sh
#!/bin/bash
#判断nginx是否存活
counter=$(ps -ef |grep nginx | grep sbin |egrep -cv "grep | $$" )
if [ $counter -eq 0 ];then
#如果不存活则尝试启动nginx
systemctl start nginx
sleep 2
#等待2秒后再次获取一次nginx状态
counter=$(ps -ef |grep nginx |grep sbin | egrep -cv "grep|$$")
#再次进行判断，如nginx还不存活则停止keepalived，让地址进行漂移
if [ $counter -eq 0]; then
systemctl stop keepalived
fi
fi
$ chmod +x /etc/keepalived/check_nginx.sh
# 备keepalive
$ vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface enp3s0
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.99/24
}
track_script {
check_nginx
}
}
$ vim /etc/keepalived/check_nginx.sh
#!/bin/bash
#判断nginx是否存活
counter=$(ps -ef |grep nginx | grep sbin |egrep -cv "grep | $$" )
if [ $counter -eq 0 ];then
#如果不存活则尝试启动nginx
systemctl start nginx
sleep 2
#等待2秒后再次获取一次nginx状态
counter=$(ps -ef |grep nginx |grep sbin | egrep -cv "grep|$$")
#再次进行判断，如nginx还不存活则停止keepalived，让地址进行漂移
if [ $counter -eq 0]; then
systemctl stop keepalived
fi
fi
$ chmod +x /etc/keepalived/check_nginx.sh
#注：keepalived根据脚本返回状态码（0为工作正常，非0不正常）判断是否故障转移。
4、启动服务（各master节点）
$ systemctl daemon-reload $ systemctl start nginx $ systemctl start keepalived $ systemctl enable nginx keepalived
5、测试vip是否绑定成功
$ ip addr #查看vip是否已绑定到对应网卡
6、测试keepalived
#停掉master01上的keepalived，Vip会漂移到master02 master01 $ systemctl stop keepalived master02 $ ip addr
15、使用kubeadm初始化k8s集群
$ kubeadm config print init-defaults > kubeadm.yaml
# 根据我们自己的需求修改配置，比如修改 imageRepository 的值，kube-proxy 的模式为 ipvs，需要注意的是由于我们使用的containerd作为运行时，所以在初始化节点的时候需要指定cgroupDriver为systemd
#kubeadm.yaml配置文件如下：
apiVersion: kubeadm.k8s.io/v1beta4
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
#localAPIEndpoint: #注释掉这几行
#advertiseAddress: 192.168.1.91
#bindPort: 6443
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock
imagePullPolicy: IfNotPresent
imagePullSerial: true
name: master01 #节点名称
taints: null
timeouts:
controlPlaneComponentHealthCheck: 4m0s
discovery: 5m0s
etcdAPICall: 2m0s
kubeletHealthCheck: 4m0s
kubernetesAPICall: 1m0s
tlsBootstrap: 5m0s
upgradeManifests: 5m0s
---
apiServer: {}
apiVersion: kubeadm.k8s.io/v1beta4
caCertificateValidityPeriod: 87600h0m0s
certificateValidityPeriod: 8760h0m0s
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
encryptionAlgorithm: RSA-2048
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.33.0 #k8s版本，按实际版本调整
controlPlaneEndpoint: 192.168.1.99:16443 #集群vip
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
podSubnet: 10.244.0.0/16 #指定pod网段，不指定会默认为192.168段
proxy: {}
scheduler: {}
#追加如下内容“---”也需要粘贴进去
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
16、各节点ctr解包镜像tar包
# 将下载好的镜像包上传到各个节点，通过ctr命令解包镜像。如需要1.33.0 arm64版本配套镜像包可以直接联系我获取 $ ctr -n=k8s.io images import <镜像包名>
17、部署集群
$ kubeadm init --config=kubeadm.yaml --ignore-preflight-errors=SystemVerification
显示如下，说明安装完成：
# 配置kubectl的配置文件config，相当于对kubectl进行授权，这样kubectl命令可以使用这个证书对k8s集群进行管理 $ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config $ kubectl get nodes
18、扩容k8s master节点-把master02添加到K8s集群
# 把master01节点的证书拷贝到master02上 # 在master02创建证书存放目录： $ cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p ~/.kube/ # 把master01节点的证书拷贝到master02上： $ scp /etc/kubernetes/pki/ca.crt master02:/etc/kubernetes/pki/ $ scp /etc/kubernetes/pki/ca.key master02:/etc/kubernetes/pki/ $ scp /etc/kubernetes/pki/sa.key master02:/etc/kubernetes/pki/ $ scp /etc/kubernetes/pki/sa.pub master02:/etc/kubernetes/pki/ $ scp /etc/kubernetes/pki/front-proxy-ca.crt master02:/etc/kubernetes/pki/ $ scp /etc/kubernetes/pki/front-proxy-ca.key master02:/etc/kubernetes/pki/ $ scp /etc/kubernetes/pki/etcd/ca.crt master02:/etc/kubernetes/pki/etcd/ $ scp /etc/kubernetes/pki/etcd/ca.key master02:/etc/kubernetes/pki/etcd/ # 证书拷贝之后在master02上执行如下命令，大家复制自己的，这样就可以把master02和加入到集群，成为控制节点： # 在master01上查看加入节点的命令： $ kubeadm token create --print-join-command # 显示如下： kubeadm join 192.168.40.199:16443 --token zwzcks.u4jd8lj56wpckcwv \ --discovery-token-ca-cert-hash sha256:1ba1b274090feecfef58eddc2a6f45590299c1d0624618f1f429b18a064cb728 \ --control-plane # 在master02上执行： $ kubeadm join 192.168.40.199:16443 --token zwzcks.u4jd8lj56wpckcwv \ --discovery-token-ca-cert-hash sha256:1ba1b274090feecfef58eddc2a6f45590299c1d0624618f1f429b18a064cb728 \ --control-plane --ignore-preflight-errors=SystemVerification #必须加 $ mkdir -p $HOME/.kube # 从master01节点上把/etc/kubernetes/admin.conf文件导入master02节点/root/.kube/config $ scp /etc/kubernetes/admin.conf master02:/root/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config # 在master01上查看集群状况： $ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 NotReady control-plane 49m v1.33.0 master02 NotReady control-plane 39s v1.33.0 # 上面可以看到master02已经加入到集群了,按此方法把master03也加入集群
19、扩容k8s集群添加工作节点
# 在master01上查看加入节点的命令： $ kubeadm token create --print-join-command # 显示如下： kubeadm join 192.168.40.199:16443 --token zwzcks.u4jd8lj56wpckcwv \ --discovery-token-ca-cert-hash sha256:1ba1b274090feecfef58eddc2a6f45590299c1d0624618f1f429b18a064cb728 # 在node节点上执行命令 $ kubeadm join 192.168.40.199:16443 --token zwzcks.u4jd8lj56wpckcwv \ --discovery-token-ca-cert-hash sha256:1ba1b274090feecfef58eddc2a6f45590299c1d0624618f1f429b18a064cb728 # 在master01上查看集群状况： $ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 NotReady control-plane 49m v1.33.0 master02 NotReady control-plane 10m v1.33.0 node01 NotReady worker 39s v1.33.0 # 上面可以看到master02已经加入到集群了,按此方法把master03也加入集群
20、修改节点标签
$ kubectl label node master01 node-role.kubernetes.io/master=master $ kubectl label node master02 node-role.kubernetes.io/master=master $ kubectl label node master03 node-role.kubernetes.io/master=master $ kubectl label node node01 node-role.kubernetes.io/node=node $ kubectl label node node02 node-role.kubernetes.io/node=node
21、安装kubernetes网络组件-Calico
# 把安装calico需要的镜像calico.tar.gz传到各个节点，手动解压(包括calico-cni.tar、calico-kube-controllers.tar、calico-node.tar、calico-pod2daemon-flexvol.tar、coredns.tar、pause.tar)： $ ctr -n=k8s.io images import calico.tar.gz #上传calico.yaml到master01上，使用yaml文件安装calico 网络插件 $ kubectl apply -f calico.yaml # 注：在线下载配置文件地址是： raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/calico.yaml $ kubectl get node NAME STATUS ROLES AGE VERSION master01 Ready control-plane 36m v1.33.0 master02 Ready control-plane 33m v1.33.0 master03 Ready control-plane 30m v1.33.0 node01 Ready work 21m v1.33.0 node02 Ready work 21m v1.33.0
22、创建pod测试
# 将nginx.tar导入node节点 $ ctr -n=k8s.io images import nginx.tar # 将pod.yaml导入master节点 $ kubectl apply -f pod.yaml # 创建pod成功，集群搭建完成
有“AI”的1024 = 2048，欢迎大家加入2048 AI社区
更多推荐
29
0- 0
已为社区贡献1条内容
所有评论(0)