Kubernetes Pod的Evicted状态原因

2024-07-30 313 0

在Kubernetes中,Pod的Evicted状态表示Pod已经被驱逐,并不再运行在节点上。Pod驱逐主要是由于资源约束,如内存不足或磁盘空间不足。

原理

Kubernetes集群会监控节点资源使用情况,当资源不足时,kubelet会触发Pod驱逐过程。驱逐过程会根据优先级和资源使用情况选择要驱逐的Pod。一旦Pod被驱逐,其状态将变为Evicted,相关事件也会被记录。

原因

内存不足:当节点上的可用内存不足以满足Pod的内存需求时,kubelet会尝试回收内存,如果回收不足,会触发Pod驱逐。
磁盘空间不足:当节点上的磁盘空间不足以满足Pod的存储需求时,kubelet会尝试回收磁盘空间,如果回收不足,会触发Pod驱逐。

[root@harbor yaml]# kubectl get pod
NAME                     READY   STATUS                   RESTARTS         AGE
busybox                  1/1     Running                  3353 (30m ago)   139d
busybox-pod              0/1     Error                    15 (6h53m ago)   21h
nfs-pod-test             1/1     Running                  0                12d
nginx-774f96d4d9-2f7n2   0/1     Evicted                  0                6h46m
nginx-774f96d4d9-2lbtr   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-4lpfb   0/1     Evicted                  0                28m
nginx-774f96d4d9-4srb2   0/1     Completed                0                7h18m
nginx-774f96d4d9-5r7rb   0/1     Evicted                  0                28m
nginx-774f96d4d9-6dbvp   0/1     Evicted                  0                6h46m
nginx-774f96d4d9-746dd   0/1     ContainerStatusUnknown   1                8h
nginx-774f96d4d9-bs4pr   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-bw6tb   0/1     Evicted                  0                6h46m
nginx-774f96d4d9-cbl5v   0/1     Evicted                  0                28m
nginx-774f96d4d9-cwq48   0/1     Evicted                  0                28m
nginx-774f96d4d9-fd9mg   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-fknwl   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-h7tjq   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-hftqh   0/1     Evicted                  0                28m
nginx-774f96d4d9-hfxp6   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-hxmrq   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-j2c46   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-j6glf   0/1     Evicted                  0                28m
nginx-774f96d4d9-jlzws   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-k9b4w   0/1     Evicted                  0                28m
nginx-774f96d4d9-kmb2r   0/1     Completed                0                6h46m
nginx-774f96d4d9-l4lhv   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-lfskb   0/1     Evicted                  0                28m
nginx-774f96d4d9-lgjlr   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-lkv2d   0/1     ContainerStatusUnknown   1                5h4m
nginx-774f96d4d9-lw78n   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-lww22   0/1     Completed                0                161m
nginx-774f96d4d9-mndr5   0/1     Evicted                  0                6h46m
nginx-774f96d4d9-n9fth   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-n9w6b   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-ncz8c   0/1     Evicted                  0                6h46m
nginx-774f96d4d9-nsxvp   0/1     Evicted                  0                28m
nginx-774f96d4d9-pprn7   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-pqjvs   0/1     Completed                0                139d
nginx-774f96d4d9-q88pf   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-qtxrg   0/1     Evicted                  0                28m
nginx-774f96d4d9-qwc8v   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-qxfr4   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-r8wkc   0/1     Evicted                  0                28m
nginx-774f96d4d9-rzh9q   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-shb72   0/1     Evicted                  0                6h46m
nginx-774f96d4d9-snhm7   0/1     Evicted                  0                7h18m
nginx-774f96d4d9-t8npl   0/1     Evicted                  0                28m
nginx-774f96d4d9-twqwx   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-vklmr   0/1     Evicted                  0                28m
nginx-774f96d4d9-w5k2z   0/1     Evicted                  0                5h4m
nginx-774f96d4d9-w6rqm   1/1     Running                  0                28m
nginx-774f96d4d9-w74x7   0/1     Evicted                  0                28m
nginx-774f96d4d9-x75m8   0/1     Evicted                  0                28m
nginx-774f96d4d9-zhcd7   1/1     Running                  0                139d
nginx-774f96d4d9-zjsmm   0/1     Evicted                  0                28m
[root@harbor yaml]# kubectl get event
LAST SEEN   TYPE      REASON                      OBJECT                                  MESSAGE
48m         Normal    Pulled                      pod/busybox                             Container image "busybox:1.28" already present on machine
48m         Normal    Created                     pod/busybox                             Created container busybox
48m         Normal    Started                     pod/busybox                             Started container busybox
41m         Normal    NodeHasSufficientMemory     node/k8s-node01                         Node k8s-node01 status is now: NodeHasSufficientMemory
46m         Warning   EvictionThresholdMet        node/k8s-node01                         Attempting to reclaim memory
46m         Normal    NodeHasInsufficientMemory   node/k8s-node01                         Node k8s-node01 status is now: NodeHasInsufficientMemory
35m         Warning   BackOff                     pod/mysql-deployment-797b5779bb-87mtl   Back-off restarting failed container mysql in pod mysql-deployment-797b5779bb-87mtl_default(d5454db8-30e7-4fa0-99f4-d03585546564)
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-4lpfb              Successfully assigned default/nginx-774f96d4d9-4lpfb to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-4lpfb              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-5r7rb              Successfully assigned default/nginx-774f96d4d9-5r7rb to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-5r7rb              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-cbl5v              Successfully assigned default/nginx-774f96d4d9-cbl5v to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-cbl5v              The node had condition: [MemoryPressure].
46m         Warning   Evicted                     pod/nginx-774f96d4d9-cwq48              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-cwq48              Successfully assigned default/nginx-774f96d4d9-cwq48 to k8s-node01
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-hftqh              Successfully assigned default/nginx-774f96d4d9-hftqh to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-hftqh              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-j6glf              Successfully assigned default/nginx-774f96d4d9-j6glf to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-j6glf              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-k9b4w              Successfully assigned default/nginx-774f96d4d9-k9b4w to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-k9b4w              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-lfskb              Successfully assigned default/nginx-774f96d4d9-lfskb to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-lfskb              The node had condition: [MemoryPressure].
46m         Warning   Evicted                     pod/nginx-774f96d4d9-lww22              The node was low on resource: memory. Threshold quantity: 100Mi, available: 176Ki. Container nginx was using 3632Ki, request is 0, has larger consumption of memory.
46m         Normal    Killing                     pod/nginx-774f96d4d9-lww22              Stopping container nginx
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-nsxvp              Successfully assigned default/nginx-774f96d4d9-nsxvp to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-nsxvp              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-qtxrg              Successfully assigned default/nginx-774f96d4d9-qtxrg to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-qtxrg              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-r8wkc              Successfully assigned default/nginx-774f96d4d9-r8wkc to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-r8wkc              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-t8npl              Successfully assigned default/nginx-774f96d4d9-t8npl to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-t8npl              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-vklmr              Successfully assigned default/nginx-774f96d4d9-vklmr to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-vklmr              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-w6rqm              Successfully assigned default/nginx-774f96d4d9-w6rqm to k8s-node02
46m         Normal    Pulled                      pod/nginx-774f96d4d9-w6rqm              Container image "nginx:alpine" already present on machine
46m         Normal    Created                     pod/nginx-774f96d4d9-w6rqm              Created container nginx
46m         Normal    Started                     pod/nginx-774f96d4d9-w6rqm              Started container nginx
16m         Normal    Killing                     pod/nginx-774f96d4d9-w6rqm              Stopping container nginx
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-w74x7              Successfully assigned default/nginx-774f96d4d9-w74x7 to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-w74x7              The node had condition: [MemoryPressure].
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-x75m8              Successfully assigned default/nginx-774f96d4d9-x75m8 to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-x75m8              The node had condition: [MemoryPressure].
16m         Normal    Killing                     pod/nginx-774f96d4d9-zhcd7              Stopping container nginx
46m         Normal    Scheduled                   pod/nginx-774f96d4d9-zjsmm              Successfully assigned default/nginx-774f96d4d9-zjsmm to k8s-node01
46m         Warning   Evicted                     pod/nginx-774f96d4d9-zjsmm              The node had condition: [MemoryPressure].
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             (combined from similar events): Created pod: nginx-774f96d4d9-w6rqm
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             Created pod: nginx-774f96d4d9-k9b4w
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             Created pod: nginx-774f96d4d9-t8npl
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             Created pod: nginx-774f96d4d9-cbl5v
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             Created pod: nginx-774f96d4d9-4lpfb
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             Created pod: nginx-774f96d4d9-cwq48
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             Created pod: nginx-774f96d4d9-qtxrg
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             Created pod: nginx-774f96d4d9-5r7rb
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             Created pod: nginx-774f96d4d9-j6glf
46m         Normal    SuccessfulCreate            replicaset/nginx-774f96d4d9             Created pod: nginx-774f96d4d9-x75m8
[root@harbor yaml]# kubectl describe node k8s-node01
Name:               k8s-node01
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k8s-node01
                    kubernetes.io/os=linux
                    node.kubernetes.io/node=
Annotations:        management.cattle.io/pod-limits: {}
                    management.cattle.io/pod-requests: {"cpu":"250m","pods":"1"}
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 192.168.77.94/24
                    projectcalico.org/IPv4IPIPTunnelAddr: 10.243.85.192
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 12 Dec 2023 13:35:12 -0500
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  k8s-node01
  AcquireTime:     <unset>
  RenewTime:       Tue, 30 Jul 2024 11:48:33 -0400
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Tue, 12 Mar 2024 17:01:23 -0400   Tue, 12 Mar 2024 17:01:23 -0400   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Tue, 30 Jul 2024 11:48:30 -0400   Tue, 30 Jul 2024 11:01:27 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Tue, 30 Jul 2024 11:48:30 -0400   Tue, 12 Mar 2024 16:55:37 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Tue, 30 Jul 2024 11:48:30 -0400   Tue, 12 Mar 2024 16:55:37 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Tue, 30 Jul 2024 11:48:30 -0400   Fri, 19 Jul 2024 04:50:42 -0400   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.77.94
  Hostname:    k8s-node01
Capacity:
  cpu:                2
  ephemeral-storage:  40383Mi
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3703128Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  38110180085
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3600728Ki
  pods:               110
System Info:
  Machine ID:                 a9bd66d853144a08a6d71037ffebc3f3
  System UUID:                feb32f42-112b-1a87-5952-36ff542fc96a
  Boot ID:                    d530df3e-e939-4eee-897a-147a6b86de5a
  Kernel Version:             5.14.0-362.8.1.el9_3.x86_64
  OS Image:                   Rocky Linux 9.3 (Blue Onyx)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.21
  Kubelet Version:            v1.26.7
  Kube-Proxy Version:         v1.26.7
PodCIDR:                      172.16.1.0/24
PodCIDRs:                     172.16.1.0/24
Non-terminated Pods:          (1 in total)
  Namespace                   Name                 CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                 ------------  ----------  ---------------  -------------  ---
  kube-system                 calico-node-b4wtl    250m (12%)    0 (0%)      0 (0%)           0 (0%)         212d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                250m (12%)  0 (0%)
  memory             0 (0%)      0 (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type     Reason                     Age                 From     Message
  ----     ------                     ----                ----     -------
  Warning  EvictionThresholdMet       52m (x17 over 9h)   kubelet  Attempting to reclaim memory
  Normal   NodeHasInsufficientMemory  52m (x17 over 9h)   kubelet  Node k8s-node01 status is now: NodeHasInsufficientMemory
  Normal   NodeHasSufficientMemory    47m (x18 over 11d)  kubelet  Node k8s-node01 status is now: NodeHasSufficientMemory

此次故障是 Rocky9 Kubernetes部署MySQL57报错 内存泄漏导致,注意要做好资源限制

节点压力驱逐

相关文章

KubeSphere DevOps 流水线JAVA项目配置
虚拟机热添加内存 Kubernetes未生效
Containerd镜像加速及私有仓库配置(用户密码和忽略HTTPS)
在Kubernetes集群部署kubesphere
使用KubeKey快速部署Kubernetes集群1.28.8
Rancher 快速创建RKE K8S集群

发布评论