LiuWen's Blogs
0%

[problems solution] linux：cannot create temp file for here-document: No space left on device

Posted on Edited on In

问题描述#

在linux系统下，用Tab键自动补全时，报如下错误： 

1
$ sh dojo-bash: cannot create temp file for here-document: No space left on device

问题定位#

  1. df -h查看磁盘空间 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/dm-0       7.8G  7.8G   20K 100% /
    udev             10M     0   10M   0% /dev
    tmpfs           3.2G  353M  2.8G  11% /run
    tmpfs           7.9G   56K  7.9G   1% /dev/shm
    tmpfs           5.0M     0  5.0M   0% /run/lock
    tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
    /dev/vdc1       200G   47G  154G  24% /home
    /dev/vda1       236M   34M  190M  15% /boot

这里的/dev/dm-0 空间已满 

1
/dev/dm-0       7.8G  7.8G   20K 100% /

  1. df -i查看磁盘Inodes消耗 和 剩余量 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    Filesystem        Inodes  IUsed     IFree IUse% Mounted on
    /dev/dm-0         129312 127969      1343   99% /
    udev             2056585    356   2056229    1% /dev
    tmpfs            2058885    501   2058384    1% /run
    tmpfs            2058885     15   2058870    1% /dev/shm
    tmpfs            2058885      4   2058881    1% /run/lock
    tmpfs            2058885     13   2058872    1% /sys/fs/cgroup
    /dev/vdc1      209713152 371880 209341272    1% /home
    /dev/vda1          62248    328     61920    1% /boot

这里的/dev/dm-0 Inodes已用99% 

1
/dev/dm-0         129312 127969      1343   99% /

  1. 分析 /dev/dm-0

迅速查找 '/' 路径下 各文件占用大小 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
sudo du -h --max-depth=1 /

32M     ./boot
4.0M    ./etc
0       ./media
3.6G    ./var
3.7G    ./usr
205M    ./lib
0       ./lib64
8.9M    ./bin
7.8M    ./sbin
du: cannot access ‘./proc/12844/task/12844/fd/4’: No such file or directory
du: cannot access ‘./proc/12844/task/12844/fdinfo/4’: No such file or directory
du: cannot access ‘./proc/12844/fd/4’: No such file or directory
du: cannot access ‘./proc/12844/fdinfo/4’: No such file or directory
0       ./proc
27M     ./tmp
0       ./dev
47G     ./home
341M    ./root
0       ./sys
337M    ./run
0       ./mnt
0       ./srv
0       ./opt
55G     .

  1. 定位/var 和 /usr 查看/var路径 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    du -lh --max-depth=1 /var
    349M    /var/lib
    65M     /var/cache
    3.2G    /var/log
    0       /var/tmp
    64K     /var/spool
    6.7M    /var/backups
    0       /var/local
    0       /var/opt
    144K    /var/mail
    192K    /var/initsh
    3.6G    /var
    这里/var/log有3.2G，可以删除一波

  2. 重新查看df -h 和 df -i 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    $df -h
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/dm-0       7.8G  7.4G  368M  96% /
    udev             10M     0   10M   0% /dev
    tmpfs           3.2G  345M  2.9G  11% /run
    tmpfs           7.9G   56K  7.9G   1% /dev/shm
    tmpfs           5.0M     0  5.0M   0% /run/lock
    tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
    /dev/vdc1       200G   47G  154G  24% /home
    /dev/vda1       236M   34M  190M  15% /boot

    $df -i
    Filesystem        Inodes  IUsed     IFree IUse% Mounted on
    /dev/dm-0        1619760 127756   1492004    8% /
    udev             2056585    356   2056229    1% /dev
    tmpfs            2058885    501   2058384    1% /run
    tmpfs            2058885     15   2058870    1% /dev/shm
    tmpfs            2058885      4   2058881    1% /run/lock
    tmpfs            2058885     13   2058872    1% /sys/fs/cgroup
    /dev/vdc1      209713152 371880 209341272    1% /home
    /dev/vda1          62248    328     61920    1% /boot
    磁盘占用还是很大，但是inodes数量降下来了。

最后，kill掉启动的python进程后，随着时间推移，/dev/dm-0 几分钟内降到了53%。

个人猜测是由于这些 python进程一直在写/var/log，虽然在系统上清空了/var/log下的文件，但是 这些进程还在占用这些log文件，linux未能及时回收。

reference#

  1. What is this dm-0 device?
  2. what is /dev/dm-0 that is 100% used?