Featured image of post 实践中遇到的几种docker命令卡住的原因
linux

实践中遇到的几种docker命令卡住的原因

星期五下午有同事通过工作QQ密我：

“x 总（我本人），有个环境的 xxxx（我们的其中一个组件）部署不上去，手动执行启动镜像的时候，执行命令后夯住了”

简单了解了一下，就是我们的部署工具通过ssh远程部署容器的时候，docker的命令一直没有返回，程序超时失败了，然后手动上去排查问题，发现页拉不起来。docker容器卡住的问题有时候还挺诡异的，一般来说在容器很多、linux 内核版本低、磁盘IO效率差的时候会发生，但是具体问题还是得查dockerd的日志。

容器频繁地创建、删除

问题

这个问题据我所知会出现在centos7系统并且磁盘格式是xfs的环境下，当我们过于频繁创建删除容器、推拉镜像，当thin pool（一种磁盘的分配策略）满时，DeviceMapper后端默认文件系统xfs会不断retry 失败的IO，导致进程挂起，然后就卡死了。

解决方案

  • 最简单地就是不使用xfs嘛；

  • 如果需要使用xfs，那就增加一个参数限制重试；

    daemon.json大概的写法是：

1
2
3
4

{
    "storage-driver": "devicemapper",
    "storage-opts": ["dm.xfs_nospace_max_retries=0"]
}

linux版本低

问题

这个问题只在特定的Linux 内核版本出现，这是内核的一个bug，这个bug会引起runc卡住；

链接-> https://lore.kernel.org/lkml/Y38h9oe4ZEGNd7Zx@quatroqueijos.cascardo.eti.br/T/

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

**linux-kernel.vger.kernel.org archive mirror** help / color / mirror / Atom feed
* [PATCH 5.4 0/2] Fix epoll issue in 5.4 kernels
@ 2022-11-24  0:11 Rishabh Bhatnagar
  2022-11-24  0:11 ` [PATCH 5.4 1/2] epoll: call final ep_events_available() check under the lock Rishabh Bhatnagar
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Rishabh Bhatnagar @ 2022-11-24  0:11 UTC (permalink / raw)
  To: gregkh, shakeelb, viro, bsegall
  Cc: mdecandia, linux-kernel, stable, Rishabh Bhatnagar

Hi Greg
After upgrading to 5.4.211 we were started seeing some nodes getting
stuck in our Kubernetes cluster. All nodes are running this kernel
version. After taking a closer look it seems that runc was command getting
stuck. Looking at the stack it appears the thread is stuck in epoll wait for
sometime. 
[<0>] do_syscall_64+0x48/0xf0
[<0>] entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[<0>] ep_poll+0x48d/0x4e0
[<0>] do_epoll_wait+0xab/0xc0
[<0>] __x64_sys_epoll_pwait+0x4d/0xa0
[<0>] do_syscall_64+0x48/0xf0
[<0>] entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe2/0x260
[<0>] do_futex+0x372/0x4f0
[<0>] __x64_sys_futex+0x134/0x180
[<0>] do_syscall_64+0x48/0xf0
[<0>] entry_SYSCALL_64_after_hwframe+0x5c/0xc1

I noticed there are other discussions going on as well
regarding this.
https://lore.kernel.org/all/Y1pY2n6E1Xa58MXv@kroah.com/
Reverting the below patch does fix the issue:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.4.y&id=cf2db24ec4b8e9d399005ececd6f6336916ab6fc
We don't see this issue in latest upstream kernel or even latest 5.10
stable tree. Looking at the patches that went in for 5.10 stable there's
one that stands out that seems to be missing in 5.4.
289caf5d8f6c61c6d2b7fd752a7f483cd153f182 (epoll: check for events when removing
a timed out thread from the wait queue)

Backporting this patch to 5.4 we don't see the hangups anymore. Looks like
this patch fixes time out scenarios which might cause missed wake ups.
The other patch in the patch series also fixes a race and is needed for
the second patch to apply.

Roman Penyaev (1):
  epoll: call final ep_events_available() check under the lock

Soheil Hassas Yeganeh (1):
  epoll: check for events when removing a timed out thread from the wait
    queue

 fs/eventpoll.c | 68 ++++++++++++++++++++++++++++++--------------------
 1 file changed, 41 insertions(+), 27 deletions(-)

解决方案

把linux的内核升级到v5.4.226+试试；

磁盘效率差

这个问题没有太好的办法，升级你的硬件吧。

万能绝招

如果你实在不想查，有一个办法能够快速的解决这个问题（起码能暂时解决）：

  1. 把docker和containerd相关的进程都杀了；
  2. 把docker的Docker Root Dir目录删了；（如果提示你被挂载了你就找到以后umount）
  3. 把/var/run/docker删了；（这里面是docker daemon的pid文件和dockerd 监听的sock等文件，没有特别指定一般默认都是这个路径）
  4. 把你的docker重新装一遍，重启。