Ubuntu 16.04上部署单机Kubernetes

发表于 更新于

在公司都是用现成的K8s集群，没自己搭过，想知道搭建集群涉及哪些组件、做了什么，于是自己搭了一下，没想象的顺利，动作做到位了，也就不会有太多问题。

许多资料都是基于Centos7的，包括《Kubernetes权威指南》，手头只有Ubuntu 16.04，刚好也是支持K8s最低Ubuntu版本，就在Ubuntu上面部署。Ubuntu与Centos部署K8s并没有太大区别，唯一区别是安装kubeadm等软件的不同，由于k8s本身也是运行在容器中，其他的过程二者都相同了，这种设计也极大的方便了k8s集群的搭建。

没有阿里云，搭建一个K8s集群还是挺费劲的。

准备工作

  1. /etc/hosts中加入：
1
127.0.0.1 k8s-master

  1. 关闭防火墙：ufw status

  2. 安装Docker，并设置镜像加速器。

安装软件

Ubuntu 16.04上利用阿里云安装kubeadm、kubelet、kubectl

1
2
3
4
5
6
7
8
sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s http://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

centos 7上利用阿里云镜像安装kubeadm、kubelet、kubectl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

二进制程序安装位置：

1
2
3
4
[~]$ which kubectl kubeadm kubectl
/usr/bin/kubectl
/usr/bin/kubeadm
/usr/bin/kubectl

部署Master节点

1
2
3
4
5
kubeadm init \
  --kubernetes-version=v1.19.0 \
  --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
  --pod-network-cidr=10.24.0.0/16 \
  --ignore-preflight-errors=Swap
  • --image-repository ： 由于k8s.gcr.io由于网络原因无法访问，使用阿里云提供的k8s镜像仓库，快速下载k8s相关的镜像
  • --ignore-preflight-errors ： 部署时忽略swap问题
  • --pod-network-cidr ：设置pod的ip区间

遇到错误需要重置集群：kubeadm reset

遇到错误参考：kubernetes安装过程报错及解决方法

拷贝kubectl配置

切回普通用户，拷贝当前集群的配置给kubectl使用：

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看集群信息：

1
2
3
dabin@ubuntu:~$ kubectl cluster-info
Kubernetes master is running at https://192.168.0.103:6443
KubeDNS is running at https://192.168.0.103:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

k8s主节点部署后的情况

k8s本身不负责容器之间的通信，集群启动后，集群的Pod直接还不能通信，需要安装网络插件。

1
2
3
4
5
6
7
8
9
10
11
12
13
$ kubectl get node
NAME         STATUS     ROLES    AGE    VERSION
k8s-master   NotReady   master   5m8s   v1.19.0
$ kubectl get pod -n kube-system -owide

NAME                                  READY   STATUS    RESTARTS   AGE   IP         NODE          NOMINATED NODE   READINESS GATES
coredns-6c76c8bb89-9wnfb              0/1     Pending   0          56s   <none>     <none>        <none>           <none>
coredns-6c76c8bb89-glkdx              0/1     Pending   0          56s   <none>     <none>        <none>           <none>
etcd-shitaibin-x                      0/1     Running   0          70s   10.0.0.3   shitaibin-x   <none>           <none>
kube-apiserver-shitaibin-x            1/1     Running   0          70s   10.0.0.3   shitaibin-x   <none>           <none>
kube-controller-manager-shitaibin-x   1/1     Running   0          70s   10.0.0.3   shitaibin-x   <none>           <none>
kube-proxy-7gpjx                      1/1     Running   0          56s   10.0.0.3   shitaibin-x   <none>           <none>
kube-scheduler-shitaibin-x            0/1     Running   0          70s   10.0.0.3   shitaibin-x   <none>           <none>

从上面可以看到master节点为 NotReady 状态，coredns 服务也没有分配ip。

从下面的Condition和Events可以看到节点会进行4项检测：

  1. 节点内存是否充足
  2. 节点磁盘是否有压力
  3. 节点Pid是否充足
  4. kubelet是否就绪

从Events可以kubelet启动了2次，而内存、磁盘压力、pid条件检查进行了4次。

从Condition的kubelet消息中看到CNI网络插件还未就绪，导致kubelet并没有ready。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ kubectl describe nodes shitaibin-x
Name:               shitaibin-x
Roles:              master
...
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Wed, 28 Oct 2020 08:40:18 +0000   Wed, 28 Oct 2020 08:40:07 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 28 Oct 2020 08:40:18 +0000   Wed, 28 Oct 2020 08:40:07 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 28 Oct 2020 08:40:18 +0000   Wed, 28 Oct 2020 08:40:07 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Wed, 28 Oct 2020 08:40:18 +0000   Wed, 28 Oct 2020 08:40:07 +0000   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
....
Events:
  Type    Reason                   Age                  From        Message
  ----    ------                   ----                 ----        -------
  Normal  Starting                 111s                 kubelet     Starting kubelet.
  Normal  NodeHasSufficientMemory  110s (x3 over 110s)  kubelet     Node shitaibin-x status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    110s (x3 over 110s)  kubelet     Node shitaibin-x status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     110s (x3 over 110s)  kubelet     Node shitaibin-x status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  110s                 kubelet     Updated Node Allocatable limit across pods
  Normal  Starting                 88s                  kubelet     Starting kubelet.
  Normal  NodeHasSufficientMemory  88s                  kubelet     Node shitaibin-x status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    88s                  kubelet     Node shitaibin-x status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     88s                  kubelet     Node shitaibin-x status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  87s                  kubelet     Updated Node Allocatable limit across pods
  Normal  Starting                 66s                  kube-proxy  Starting kube-proxy.

查看kubelet进程启动配置：

1
2
$ ps -ef | grep kubelet
root      5549     1  2 08:40 ?        00:00:07 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2

通过--network-plugin=cni可以看到默认使用CNI作为网络插件。

主机/etc/cni/net.d目录保存CNI的配置，发现目前为空。/opt/cni/bin目录为CNI插件程序所在的位置，可以看到已经有flannel等插件。

1
2
3
4
$ ls /etc/cni/net.d
ls: cannot access '/etc/cni/net.d': No such file or directory
$ ls /opt/cni/bin
bandwidth  bridge  dhcp  firewall  flannel  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sbr  static  tuning  vlan

接下来安装CNI网络插件。

安装CNI网络插件

k8s的文档列举了多种选择，这里提供2种：

较为简便的weave，它提供overlay network:

1
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

flannel，也是overlay network模型:

1
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

本机选择了weave：

1
2
3
4
5
6
7
dabin@ubuntu:~$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.apps/weave-net created

查看weave的配置和可执行程序：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ ls /etc/cni/net.d
10-weave.conflist
$ ls /opt/cni/bin
bandwidth  dhcp      flannel      host-local  loopback  portmap  sbr     tuning  weave-ipam  weave-plugin-2.7.0
bridge     firewall  host-device  ipvlan      macvlan   ptp      static  vlan    weave-net
$
$ cat /etc/cni/net.d/10-weave.conflist
{
    "cniVersion": "0.3.0",
    "name": "weave",
    "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

安装之后节点变为Ready，coredns也拥有了ip：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ kubectl get node
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   10m   v1.19.0

$ kubectl get pod -n kube-system -owide
NAME                                  READY   STATUS    RESTARTS   AGE     IP          NODE          NOMINATED NODE   READINESS GATES
coredns-6c76c8bb89-9wnfb              1/1     Running   0          19m     10.32.0.9   shitaibin-x   <none>           <none>
coredns-6c76c8bb89-glkdx              1/1     Running   0          19m     10.32.0.8   shitaibin-x   <none>           <none>
etcd-shitaibin-x                      1/1     Running   0          20m     10.0.0.3    shitaibin-x   <none>           <none>
kube-apiserver-shitaibin-x            1/1     Running   0          20m     10.0.0.3    shitaibin-x   <none>           <none>
kube-controller-manager-shitaibin-x   1/1     Running   0          20m     10.0.0.3    shitaibin-x   <none>           <none>
kube-proxy-7gpjx                      1/1     Running   0          19m     10.0.0.3    shitaibin-x   <none>           <none>
kube-scheduler-shitaibin-x            1/1     Running   0          20m     10.0.0.3    shitaibin-x   <none>           <none>
weave-net-72p6p                       2/2     Running   0          2m36s   10.0.0.3    shitaibin-x   <none>           <none>

开启master调度

master节点默认是不可被调度的，不可在master上部署任务，在单节点下，只有master一个节点，部署资源后会出现以下错误：

1
2
3
4
5
$ kubectl describe pod mysql
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  22s (x2 over 22s)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.

Event的告警信息提示，当前节点存在一个污点node-role.kubernetes.io/master:，而pod却不容忍这个污点。

查看master节点的信息，确实可以看到一个不允许调度的污点。

1
2
3
4
5
6
7
8
9
$ kubectl get nodes shitaibin-x -o yaml | grep -10 taint
...
spec:
  podCIDR: 10.24.0.0/24
  podCIDRs:
  - 10.24.0.0/24
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

有2个办法解决这个问题，让资源可以调度到master节点上：

  1. 所有的资源声明文件中，都设置容忍这个污点，
  2. master节点上删除这个污点

我们是一个测试环境，采取第2种办法更简单：

1
2
3
# --all为所有节点上的污点
# 最后的-代表移除污点
kubectl taint nodes --all node-role.kubernetes.io/master-

移除污点记录：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[~]$ kubectl taint nodes shitaibin-x node-role.kubernetes.io/master-
node/shitaibin-x untainted
[~]$
[~]$ kubectl get nodes shitaibin-x -o yaml | grep -10 taint
[~]$
[~]$ kubectl describe pod | grep -10 Events
...
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  27s (x7 over 7m38s)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
  Normal   Scheduled         17s                  default-scheduler  Successfully assigned default/mysql-0 to shitaibin-x
  Normal   Pulled            15s                  kubelet            Container image "mysql:5.6" already present on machine
  Normal   Created           14s                  kubelet            Created container mysql
  Normal   Started           14s                  kubelet            Started container mysql

测试集群

部署一个Pod进行测试，Pod能Running，代表Docker、K8s的配置基本没问题了：

声明文件为twocontainers.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: v1 #指定当前描述文件遵循v1版本的Kubernetes API
kind: Pod #我们在描述一个pod
metadata:
  name: twocontainers #指定pod的名称
  namespace: default #指定当前描述的pod所在的命名空间
  labels: #指定pod标签
    app: twocontainers
  annotations: #指定pod注释
    version: v0.5.0
    releasedBy: david
    purpose: demo
spec:
  containers:
    - name: sise #容器的名称
      image: quay.io/openshiftlabs/simpleservice:0.5.0 #创建容器所使用的镜像
      ports:
        - containerPort: 9876 #应用监听的端口
    - name: shell #容器的名称
      image: centos:7 #创建容器所使用的镜像
      command: #容器启动命令
        - "bin/bash"
        - "-c"
        - "sleep 10000"

部署Pod：

1
kubectl apply -f twocontainers.yaml

几分钟后可以看pod状态是否为running。

1
2
3
dabin@k8s-master:~/workspace/notes/kubernetes/examples$ kubectl get pods
NAME            READY   STATUS    RESTARTS   AGE
twocontainers   2/2     Running   2          83m

如果不是，查看Pod部署遇到的问题：

1
kubectl describe pod twocontainers

清空k8s环境

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// remove_k8s.sh
# 重置k8s
sudo kubeadm reset -f
# 删除kubectl配置文件
sudo rm -rf ~/.kube
# 卸载和清理程序配置文件
sudo apt-get -y purge kubeadm kubectl kubelet kubernetes-cni
# 卸载自安装依赖
sudo apt-get -y autoremove

# 删除遗留的文件
sudo rm -rf ~/.kube/
sudo rm -rf /etc/kubernetes/
sudo rm -rf /etc/systemd/system/kubelet.service.d
sudo rm -rf /etc/systemd/system/kubelet.service
sudo rm -rf /etc/cni
sudo rm -rf /opt/cni
sudo rm -rf /var/lib/etcd
sudo rm -rf /var/etcd

更快的部署方法？

利用kind，使用Docker快速部署一个本地测试、开发k8s环境。

1
GO111MODULE="on" go get sigs.k8s.io/kind@v0.9.0 && kind create cluster

资料

  1. 人人必备的神书《Kuerbenetes权威指南》
  2. K8S中文文档
  3. kubernetes安装过程报错及解决方法