kubernetes
:v1.12.2
docker:18.06.1-ce
helm:v2.12.0
通过helm删除服务后，pods无法删除:
~]# helm del --purge simp-ser
release "simp-ser" deleted //删除成功
~]# helm install simp-ser --name=simp-ser
Error: release simp-ser failed: object is being deleted: persistentvolumes "simp-ser" already exists //创建时报错，pvc已经存在
突如其来的报错让我有点懵，因为定义的这个pvc是跟随helm服务一起删除，我赶紧去看了下pvc服务:
~]# kubectl get pvc
simp-ser Terminating simp-ser 300Mi RWO,RWX 3d21h//正在删除
pvc正在删除中，但是一直没有删除成功，因为关联了pod，我又查看了相关pod：
~]# kubectl get pods -o wide
simp-ser-6b8f54bb64-tdxlw 1/1 Terminating 0 3d21h 10.244.1.71 k8s.node1 <none>//也是正在删除
~]# kubectl describe pods simp-ser-6b8f54bb64-tdxlw
...
Container ID: docker://70494faefcf334b906a112a7e7f9a9821a0aaff314f9e21f4ae949d527daf7e1
...
Warning FailedKillPod 13m (x324 over 21h) kubelet, k8s.node1 error killing pod: [failed to "KillContainer" for "simp-ser" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
, failed to "KillPodSandbox" for "0e9a9fa9-6328-11e9-bd84-000c2975aae4" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
...
] //pod内的容器是70494faefcf...
看的出来是问题出在了node节点上，继续查看node的状态：
// 查看node是否正常
~]# kubectl get node
k8s.node1 Ready <none> 146d v1.12.2
//切换到node节点查看kubelet 状态
~]#systemctl status kubelet
...
Active: active (running) since 五 2019-03-29 10:07:30 CST; 3 weeks 5 days ago.//kubelet状态也是正常的
...
//查看kubelet日志
~]# journalctl -u kubelet -f
···
4月 24 11:08:48 k8s.node1 kubelet[2147]: E0424 11:08:48.155788 2147 kuberuntime_manager.go:810] Failed to stop sandbox {"docker" "ffc1eccb5b0ba196fe86f9987632194ee3c2e63372adb2d283a05ac0c2e68f4d"}
4月 24 11:08:48 k8s.node1 kubelet[2147]: E0424 11:08:48.155934 2147 kubelet.go:1551] error killing pod: [failed to "KillContainer" for "simp-layui" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
4月 24 11:08:48 k8s.node1 kubelet[2147]: , failed to "KillPodSandbox" for "c5cae9f9-6321-11e9-bd84-000c2975aae4" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
4月 24 11:08:48 k8s.node1 kubelet[2147]: ]
··· //
既然kubelet没有问题，那再去看看pod中的容器：
~]# docker ps
70494faefcf3 f4b4fccc65bb "sh -c 'java -jar /o…" 3 days ago Up 3 days k8s_simp-ser_simp-ser-6b8f54bb64-tdxlw_jx_0e9a9fa9-6328-11e9-bd84-000c2975aae4_0 // 容器还在运行
//查看下docker日志
~]#journalctl -u docker -f
···
msg="Container 70494faefcf334b906a112a7e7f9a9821a0aaff314f9e21f4ae949d527daf7e1 failed to exit within 30 seconds of signal 15 - using the force"
···//docker 无法 kill 掉容器
初步得出的判断是因为容器化后，由于单容器单进程，已经没有传统意义上的 init 进程了。应用进程直接占用
了 pid 1 的进程号，Linux 内核中会对 pid 1 进程发送特殊的信号量。具体可参考Docker init 进程。
最终解决办法：
~]# docker restart // 重启nodes节点
