k8s 集群搭建-06-部署 kubernetes 工作节点

2022-06-04 08:30:00 ⋅ 12896 ⋅ 0 ⋅ 0

六、部署kubernetes工作节点

这部分我们部署kubernetes的工作节点。实例中我们有两个工作节点，一个是独立的工作节点，一个是跟master在一起的节点。
在每个节点上我们会部署kubelet、kube-proxy、container runtime、cni、nginx-proxy

下面的操作需要在每一个工作节点执行

1. Container Runtime - Containerd

Containerd 迟早需要面对，所以需要与时俱进。

1.1 软件包下载

# 设定containerd的版本号
$ VERSION=1.4.3
# 下载压缩包
$ wget https://github.com/containerd/containerd/releases/download/v${VERSION}/cri-containerd-cni-${VERSION}-linux-amd64.tar.gz

将文件分发到node-2、node-3节点：

$ cd /opt/softwards
 [root@hombd03 softwards]# scp cri-containerd-cni-1.4.3-linux-amd64.tar.gz homaybd04:/opt/softwares/
$ mkdir cri-containerd
$ mv cri-containerd-cni-1.4.3-linux-amd64.tar.gz cri-containerd/
 [root@hombd03 softwards]# scp cri-containerd-cni-1.4.3-linux-amd64.tar.gz homaybd05:/opt/softwares/

1.2 整理压缩文件

下载后的文件是一个tar.gz，是一个allinone的包，包括了runc、circtl、ctr、containerd等容器运行时以及cni相关的文件，解压缩到一个独立的目录中

$ VERSION=1.4.3

# 解压缩
$ tar -xvf cri-containerd-cni-${VERSION}-linux-amd64.tar.gz
# 复制需要的文件
$ cp etc/crictl.yaml /etc/
$ cp etc/systemd/system/containerd.service /etc/systemd/system/
$ cp -r usr /

1.3 containerd配置文件

在node-2 和 node-3节点执行以下命令：

$ mkdir -p /etc/containerd
# 默认配置生成配置文件
$ containerd config default > /etc/containerd/config.toml
# 定制化配置（可选）
$ vi /etc/containerd/config.toml

1.4 启动containerd

在node-2 和 node-3节点执行以下命令：

$ systemctl enable containerd
$ systemctl restart containerd
# 检查状态
$ systemctl status containerd

查看状态：

[root@hombd04 cri-containerd]# systemctl status containerd
● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-06-04 15:34:07 CST; 17s ago
     Docs: https://containerd.io
  Process: 18264 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 18268 (containerd)
    Tasks: 11
   Memory: 19.5M
   CGroup: /system.slice/containerd.service
           └─18268 /usr/local/bin/containerd

Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.165923846+08:00" level=info msg="loading plugin \...rpc.v1
Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.166059994+08:00" level=info msg="Start subscribin...event"
Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.166165777+08:00" level=info msg="Start recovering state"
Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.166274300+08:00" level=info msg="Start event monitor"
Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.166290348+08:00" level=info msg="Start snapshots syncer"
Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.166298151+08:00" level=info msg="Start cni networ...yncer"
Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.166304347+08:00" level=info msg="Start streaming server"
Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.168350656+08:00" level=info msg=serving... addres....ttrpc
Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.168471877+08:00" level=info msg=serving... addres...d.sock
Jun 04 15:34:07 homaybd04 containerd[18268]: time="2022-06-04T15:34:07.169028759+08:00" level=info msg="containerd succe...4146s"
Hint: Some lines were ellipsized, use -l to show in full.

2. 配置kubelet

在node-2 和 node-3节点执行以下命令：

准备kubelet配置
$ mkdir -p /etc/kubernetes/ssl/
$ cd ~
$ mv ${HOSTNAME}-key.pem ${HOSTNAME}.pem ca.pem ca-key.pem /etc/kubernetes/ssl/
$ mv ${HOSTNAME}.kubeconfig /etc/kubernetes/kubeconfig
# $ IP=10.155.19.64
$ IP=192.168.1.124 (在 node-2 节点 执行）
# $ IP=192.168.1.125 (在node-3 节点执行）
# 写入kubelet配置文件
$ cat <<EOF > /etc/kubernetes/kubelet-config.yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/etc/kubernetes/ssl/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "169.254.25.10"
podCIDR: "192.200.0.0/16"
address: ${IP}
readOnlyPort: 0
staticPodPath: /etc/kubernetes/manifests
healthzPort: 10248
healthzBindAddress: 127.0.0.1
kubeletCgroups: /systemd/system.slice
resolvConf: "/etc/resolv.conf"
runtimeRequestTimeout: "15m"
kubeReserved:
  cpu: 200m
  memory: 512M
tlsCertFile: "/etc/kubernetes/ssl/${HOSTNAME}.pem"
tlsPrivateKeyFile: "/etc/kubernetes/ssl/${HOSTNAME}-key.pem"
EOF
配置kubelet服务
$ cat <<EOF > /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
  --config=/etc/kubernetes/kubelet-config.yaml \\
  --container-runtime=remote \\
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\
  --image-pull-progress-deadline=2m \\
  --kubeconfig=/etc/kubernetes/kubeconfig \\
  --network-plugin=cni \\
  --node-ip=${IP} \\
  --register-node=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

3. 配置nginx-proxy

nginx-proxy是一个用于worker节点访问apiserver的一个代理，是apiserver一个优雅的高可用方案，它使用kubelet的staticpod方式启动，让每个节点都可以均衡的访问到每个apiserver服务，优雅的替代了通过虚拟ip访问apiserver的方式。

Tips: nginx-proxy 只需要在没有 apiserver 的节点部署哦，所以，只需要在 node-3 上执行以下命令，因为 node-3 是纯worker节点。

在 node-3 上执行以下命令

3.1 nginx配置文件
$ mkdir -p /etc/nginx
# master ip列表
# $ MASTER_IPS=(10.155.19.223 10.155.19.64)
$ MASTER_IPS=(192.168.1.123 192.168.1.124)
# 执行前请先copy一份，并修改好upstream的 'server' 部分配置
$ cat <<EOF > /etc/nginx/nginx.conf
error_log stderr notice;

worker_processes 2;
worker_rlimit_nofile 130048;
worker_shutdown_timeout 10s;

events {
  multi_accept on;
  use epoll;
  worker_connections 16384;
}

stream {
  upstream kube_apiserver {
    least_conn;
    server ${MASTER_IPS[0]}:6443;
    server ${MASTER_IPS[1]}:6443;
  }

  server {
    listen        127.0.0.1:6443;
    proxy_pass    kube_apiserver;
    proxy_timeout 10m;
    proxy_connect_timeout 1s;
  }
}

http {
  aio threads;
  aio_write on;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_timeout 5m;
  keepalive_requests 100;
  reset_timedout_connection on;
  server_tokens off;
  autoindex off;

  server {
    listen 8081;
    location /healthz {
      access_log off;
      return 200;
    }
    location /stub_status {
      stub_status on;
      access_log off;
    }
  }
}
EOF
3.2 nginx manifest

将 nginx 当做 pod 调起来；

$ mkdir -p /etc/kubernetes/manifests/
$ cat <<EOF > /etc/kubernetes/manifests/nginx-proxy.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-proxy
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-nginx
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  nodeSelector:
    kubernetes.io/os: linux
  priorityClassName: system-node-critical
  containers:
  - name: nginx-proxy
    image: docker.io/library/nginx:1.19
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 25m
        memory: 32M
    securityContext:
      privileged: true
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8081
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8081
    volumeMounts:
    - mountPath: /etc/nginx
      name: etc-nginx
      readOnly: true
  volumes:
  - name: etc-nginx
    hostPath:
      path: /etc/nginx
EOF

4. 配置kube-proxy

需要在每个worker（node-2、node-3）节点执行以下命令；

4.1 配置文件
$ mv kube-proxy.kubeconfig /etc/kubernetes/
# 创建 kube-proxy-config.yaml
$ cat <<EOF > /etc/kubernetes/kube-proxy-config.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
bindAddress: 0.0.0.0
clientConnection:
  kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"
clusterCIDR: "192.200.0.0/16"
mode: ipvs
EOF
4.2 kube-proxy 服务文件
$ cat <<EOF > /etc/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/etc/kubernetes/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

5. 启动服务

在 node-2 和 node-3 执行以下命令：

$ mkdir -p /etc/kubernetes/manifests
$ systemctl daemon-reload
$ systemctl enable kubelet kube-proxy
$ systemctl restart kubelet kube-proxy
$ journalctl -f -u kubelet
$ journalctl -f -u kube-proxy

打印日志

$ journalctl -f -u kubelet

file

可以看到报网络插件 cni 未安装，这是正常的，因为我们并还没有部署网路插件。

Jun 04 16:10:22 homaybd04 kubelet[29799]: E0604 16:10:22.882174   29799 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

etes/manifests": path does not exist, ignoring
Jun 04 16:12:11 homaybd04 kubelet[29799]: E0604 16:12:11.536277   29799 file_linux.go:60] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
Jun 04 16:12:12 homaybd04 kubelet[29799]: E0604 16:12:12.536428   29799 file_linux.go:60] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
Jun 04 16:12:12 homaybd04 kubelet[29799]: E0604 16:12:12.893777   29799 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Jun 04 16:12:13 homaybd04 kubelet[29799]: I0604 16:12:13.363483   29799 setters.go:86] Using node IP: "192.168.1.124"
Jun 04 16:12:13 homaybd04 kubelet[29799]: E0604 16:12:13.536593   29799 file_linux.go:60] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
Jun 04 16:12:14 homaybd04 kubelet[29799]: E0604 16:12:14.536762   29799 file_linux.go:60] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
Jun 04 16:12:15 homaybd04 kubelet[29799]: E0604 16:12:15.536917   29799 file_linux.go:60] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
Jun 04 16:12:16 homaybd04 kubelet[29799]: E0604 16:12:16.466871   29799 file.go:104] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring
Jun 04 16:12:16 homaybd04 kubelet[29799]: E0604 16:12:16.537076   29799 file_linux.go:60] Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring

再次查看node-3 节点的打印日志：

$ journalctl -f -u kubelet
...
un 04 16:14:46 homaybd05 kubelet[14268]: E0604 16:14:46.395612   14268 kubelet.go:2243] node "homaybd05" not found
Jun 04 16:14:46 homaybd05 kubelet[14268]: E0604 16:14:46.495742   14268 kubelet.go:2243] node "homaybd05" not found
Jun 04 16:14:46 homaybd05 kubelet[14268]: I0604 16:14:46.553096   14268 csi_plugin.go:1016] Failed to contact API server when waiting for CSINode publishing: Get "https://127.0.0.1:6443/apis/storage.k8s.io/v1/csinodes/homaybd05": dial tcp 127.0.0.1:6443: connect: connection refused
Jun 04 16:14:46 homaybd05 kubelet[14268]: E0604 16:14:46.595873   14268 kubelet.go:2243] node "homaybd05" not found
Jun 04 16:14:46 homaybd05 kubelet[14268]: E0604 16:14:46.695972   14268 kubelet.go:2243] node "homaybd05" not found
Jun 04 16:14:46 homaybd05 kubelet[14268]: E0604 16:14:46.796119   14268 kubelet.go:2243] node "homaybd05" not found
Jun 04 16:14:46 homaybd05 kubelet[14268]: E0604 16:14:46.896390   14268 kubelet.go:2243] node "homaybd05" not found
Jun 04 16:14:46 homaybd05 kubelet[14268]: E0604 16:14:46.996599   14268 kubelet.go:2243] node "homaybd05" not found
Jun 04 16:14:47 homaybd05 kubelet[14268]: E0604 16:14:47.096726   14268 kubelet.go:2243] node "homaybd05" not found
Jun 04 16:14:47 homaybd05 kubelet[14268]: E0604 16:14:47.196877   14268 kubelet.go:2243] node "homaybd05" not found
Jun 04 16:14:47 homaybd05 kubelet[14268]: E0604 16:14:47.297034   14268 kubelet.go:2243] node "homaybd05" not found

根据 critcl （即之前的 docker 命令）命令查看镜像下载情况：

[root@homaybd05 ~]# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
[root@homaybd05 ~]# crictl images
IMAGE               TAG                 IMAGE ID            SIZE
[root@homaybd05 ~]#

查看配置：

[root@homaybd05 ~]# cat /etc/kubernetes/manifests/nginx-proxy.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-proxy
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-nginx
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  nodeSelector:
    kubernetes.io/os: linux
  priorityClassName: system-node-critical
  containers:
  - name: nginx-proxy
    image: docker.io/library/nginx:1.19
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 25m
        memory: 32M
    securityContext:
      privileged: true
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8081
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8081
    volumeMounts:
    - mountPath: /etc/nginx
      name: etc-nginx
      readOnly: true
  volumes:
  - name: etc-nginx
    hostPath:
      path: /etc/nginx

进行手动下载：

[root@hombd05 ~]# crictl pull docker.io/library/nginx:1.19

接着继续执行以下命令，手动下载镜像：

6. 手动下载镜像（服务器无法访问外网情况）

在每个工作节点下载pause镜像

$ crictl pull registry.cn-hangzhou.aliyuncs.com/kubernetes-kubespray/pause:3.2
$ ctr -n k8s.io i tag  registry.cn-hangzhou.aliyuncs.com/kubernetes-kubespray/pause:3.2 k8s.gcr.io/pause:3.2

查看 nginx-proxy 启动成功：

[root@homaybd05 ~]# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
8deacc7975096       f0b8a9a541369       6 seconds ago       Running             nginx-proxy         0                   b847dc6f117d4
[root@homaybd05 ~]#

然后打印日志可以看到正常了：

[root@homaybd05 ~]# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
8deacc7975096       f0b8a9a541369       6 seconds ago       Running             nginx-proxy         0                   b847dc6f117d4
[root@homaybd05 ~]# journalctl -f -u kubelet
-- Logs begin at Mon 2022-05-16 07:52:25 CST. --
Jun 04 16:33:04 homaybd05 kubelet[21180]: I0604 16:33:04.504790   21180 setters.go:86] Using node IP: "192.168.1.125"
Jun 04 16:33:04 homaybd05 kubelet[21180]: I0604 16:33:04.673079   21180 kubelet_getters.go:176] "Pod status updated" pod="kube-system/nginx-proxy-homaybd05" status=Running
Jun 04 16:33:04 homaybd05 kubelet[21180]: I0604 16:33:04.717467   21180 kubelet.go:1888] SyncLoop (ADD, "api"): "nginx-proxy-homaybd05_kube-system(9fd67d74-059b-4cf0-9673-8cc6b40c1376)"
Jun 04 16:33:04 homaybd05 kubelet[21180]: E0604 16:33:04.830324   21180 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Jun 04 16:33:09 homaybd05 kubelet[21180]: E0604 16:33:09.830912   21180 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Jun 04 16:33:14 homaybd05 kubelet[21180]: I0604 16:33:14.509596   21180 setters.go:86] Using node IP: "192.168.1.125"
Jun 04 16:33:14 homaybd05 kubelet[21180]: E0604 16:33:14.831346   21180 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
...

为者常成，行者常至

自由转载-非商用-非衍生-保持署名（创意共享3.0许可证）