限制容器在node节点上的资源占用。
1. 节点信息总览
1.1 master 信息输出如下
"Capacity"和"Allocatable" 处可见，资源全部被允许被分配，即没有预留：
[root@devops-master ~]# kubectl describe nodes devops-master
Name: devops-master
Roles: master
#以下是给角色打的标签，架构和操作系统，等都会在里边。
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=devops-master
kubernetes.io/os=linux
node-role.kubernetes.io/master=
# flannel网卡的虚拟MAC地址，也可以在ip a 中看到
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"9e:d1:1a:e6:83:2e"}
# vxlan指 可扩展的虚拟网络
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 10.252.97.56
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0ke
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 29 Apr 2020 16:33:15 +0800
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 12 Aug 2020 18:50:58 +0800 Wed, 29 Apr 2020 16:33:11 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 12 Aug 2020 18:50:58 +0800 Wed, 29 Apr 2020 16:33:11 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 12 Aug 2020 18:50:58 +0800 Wed, 29 Apr 2020 16:33:11 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 12 Aug 2020 18:50:58 +0800 Wed, 29 Apr 2020 16:45:45 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.252.97.56
Hostname: devops-master
#所有硬件资源
Capacity:
cpu: 8
ephemeral-storage: 25792732Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32765896Ki
pods: 110
#以下是可分配资源
Allocatable:
cpu: 8
ephemeral-storage: 23770581772
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32663496Ki
pods: 110
System Info:
Machine ID: dff543df0a0c44e2962f1438f92b6868
System UUID: 42277530-DD16-E8F5-B3AB-C6831B9F49FA
Boot ID: 45ec85f2-0237-4d25-b684-6ec886f0c824
Kernel Version: 3.10.0-514.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.1
Kubelet Version: v1.15.2
Kube-Proxy Version: v1.15.2
PodCIDR: 10.244.0.0/24
Non-terminated Pods: (8 in total)
#以下列出所有pod的信息
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system coredns-bccdc95cf-vrxck 100m (1%) 0 (0%) 70Mi (0%) 170Mi (0%) 105d
kube-system etcd-devops-master 0 (0%) 0 (0%) 0 (0%) 0 (0%) 105d
kube-system kube-apiserver-devops-master 250m (3%) 0 (0%) 0 (0%) 0 (0%) 105d
kube-system kube-controller-manager-devops-master 200m (2%) 0 (0%) 0 (0%) 0 (0%) 105d
kube-system kube-flannel-ds-amd64-bh5gv 100m (1%) 100m (1%) 50Mi (0%) 50Mi (0%) 105d
kube-system kube-proxy-6r9sg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 105d
kube-system kube-scheduler-devops-master 100m (1%) 0 (0%) 0 (0%) 0 (0%) 105d
monitoring prometheus-operator-prometheus-node-exporter-qb48r 0 (0%) 0 (0%) 0 (0%) 0 (0%) 96d
# 以下是已分配资源
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 750m (9%) 100m (1%)
memory 120Mi (0%) 220Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>
1.2 node信息如下
同样查看node节点信息，可见资源同样都被分配了。
Capacity:
cpu: 4
ephemeral-storage: 43400496Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16247820Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 43400496Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16247820Ki
pods: 110
说明：下文会修改这个节点的cgroup资源限制
2. 配置docker的 cgroup驱动
- 确认docker驱动
# docker info | grep "Cgroup Driver"
Cgroup Driver: cgroupfs
- 如果不是 cgroupfs，则可以通过以下方法配置
# vim /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=cgroupfs"],
.............
}
3. 配置kubelete的cgroup驱动
3.1 配置文件
/var/lib/kubelet/kubeadm-flags.env
作用：
用来为Kube组件和System进程预留资源，从而保证当节点出现满负荷时也能保证Kube和System进程有足够的资源。
3.2 默认配置
KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1"
参数说明
- Node Capacity： 是Node的所有硬件资源
- kube-reserved： 是给kube组件预留的资源
- system-reserved： 是给System进程预留的资源
- eviction-threshold： 驱逐阈值
- allocatable： 可配置值
节点上可配置值 = 总量 - kube组件预留值 - 系统预留值 - 驱逐阈值
3.3 修改如下
KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs \
--network-plugin=cni \
--pod-infra-container-image=nexus.10010sh.cn/pause:3.1 \
--enforce-node-allocatable=pods,kube-reserved,system-reserved \
--kube-reserved-cgroup=/system.slice/kubelet.service \
--system-reserved-cgroup=/system.slice \
--kube-reserved=cpu=1,memory=1Gi \
--system-reserved=cpu=1,memory=1Gi \
--eviction-hard=memory.available<5%,nodefs.available<10%,imagefs.available<10% \
--eviction-soft=memory.available<10%,nodefs.available<15%,imagefs.available<15% \
--eviction-soft-grace-period=memory.available=2m,nodefs.available=2m,imagefs.available=2m \
--eviction-max-pod-grace-period=30 \
--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=500Mi,imagefs.available=500Mi"
注解：
--cgroup-driver=cgroupfs \
--network-plugin=cni \
--pod-infra-container-image=nexus.10010sh.cn/pause:3.1 \
#开启为kube组件和系统守护进程预留资源的功能
--enforce-node-allocatable=pods,kube-reserved,system-reserved \
#设置k8s组件的cgroup
--kube-reserved-cgroup=/system.slice/kubelet.service \
#设置系统守护进程的cgroup
--system-reserved-cgroup=/system.slice \
# kubernetes预留
--kube-reserved=cpu=1,memory=1Gi \
# 系统预留
--system-reserved=cpu=1,memory=1Gi \
#驱逐pod的硬限制
--eviction-hard=memory.available<5%,nodefs.available<10%,imagefs.available<10% \
#驱逐pod的软限制
--eviction-soft=memory.available<10%,nodefs.available<15%,imagefs.available<15% \
#达到驱逐阈值后多久开始驱逐
--eviction-soft-grace-period=memory.available=2m,nodefs.available=2m,imagefs.available=2m \
#驱逐前最大等待时间
--eviction-max-pod-grace-period=30 \
#至少回收多少资源才停止驱逐
--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=500Mi,imagefs.available=500Mi"
3.4 修改kubelet 启动文件
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
[Service]
ExecStart=/usr/bin/kubelet
#添加如下两行
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice/kubelet.service
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
4. 重启服务查看结果
重启服务
如果修改了docker则需重启docker
重启kubelet查看修改结果
Capacity:
cpu: 4
ephemeral-storage: 43400496Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16247820Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 43400496Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 13658395636
pods: 110
