How to Identify Large Folders Taking up Space on Linux

On Windows you have a GUI where you can identify large folders taking up space. On most Linux servers, which are headless (i.e. no GUI desktop), these helpful graphical interfaces are not present because they take up precious resources which should be used for your web server software. You can still have large folders with files you are not using on Linux that can require cleaning up. This is especially true if you use a VPS or dedicated server with SSD drives which are very fast but also expensive per GB and therefore you normally get less space. It is critical to keep your server clean so you are not spending money on storing unnecessary files!

I have used this technique multiple times on Codeable after Dan Dulaney, a fellow Codeable expert, from WP Tech Guides shared this time-saving tip with me.

This tutorial shows you how to use ncdu to identify large folders taking up space on your system, there is an alternative technique using tree and du outlined as well.

Using ncdu to Identify Large Folders

Install ncdu on Debian or Ubuntu with this command

sudo apt-get install ncdu

On CentOS 6.8 and greater

yum install epel-release
yum install ncdu

Enter the root directory (not the root user’s home directory /root)

cd /

Execute ncdu

ncdu

After ncdu scans your entire file system you will be shown the interface below.

Using ncdu you get an awesome drill down menu. The largest folders are shown in ascending order from biggest to smallest.

The idea is you enter the largest folder first which then takes you to a similar menu showing the subfolders again in ascending order.

After entering the subfolder you can see which of those subfolders are taking up the most space.

--- / --------------------------------------------------------------------------
   14.4 GiB [##########] /var
   10.9 GiB [#######   ] /root
    1.3 GiB [          ] /usr
    1.0 GiB [          ] /lib
    1.0 GiB [          ]  swapfile
  197.7 MiB [          ] /boot
   96.2 MiB [          ] /tmp
   40.2 MiB [          ] /run
   15.6 MiB [          ] /bin
   12.9 MiB [          ] /sbin
    7.3 MiB [          ] /etc
e  16.0 KiB [          ] /lost+found
   12.0 KiB [          ] /media
    4.0 KiB [          ] /lib64
e   4.0 KiB [          ] /srv
e   4.0 KiB [          ] /snap
e   4.0 KiB [          ] /opt
e   4.0 KiB [          ] /mnt
e   4.0 KiB [          ] /home
.   0.0   B [          ] /proc
    0.0   B [          ] /sys
 Total disk usage:  29.0 GiB  Apparent size:  29.0 GiB  Items: 449055

I use the arrows to choose the folders and when I am done use q to quit and remove unnecessary files

Here are the commands for controlling ncdu, you should be able to get by with the arrow keys to navigate and q to quit

qncdu helpqqqqqqqqqqqqqqqqq1:Keysqqq2:Formatqqq3:Aboutqqk
    1.1 Gix                                                          x
  164.8 Mix       up, k  Move cursor up                              x
   20.7 Mix     down, j  Move cursor down                            x
    7.7 Mix right/enter  Open selected directory                     x
  812.0 Kix  left, <, h  Open parent directory                       x
   56.0 Kix           n  Sort by name (ascending/descending)         x
   56.0 Kix           s  Sort by size (ascending/descending)         x
   12.0 Kix           C  Sort by items (ascending/descending)        x
    8.0 Kix           d  Delete selected file or directory           x
    8.0 Kix           t  Toggle dirs before files when sorting       x
e   4.0 Kix           g  Show percentage and/or graph                x
    4.0 Kix                        -- more --                        x
    4.0 Kix                                         Press q to close x
    4.0 Kimqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj

This is by far the fastest way I have learned to identify large folders on Linux systems.

Tree and du

On Ubuntu and Debian systems

sudo apt-get install tree

On CentOS

yum install tree

You can run this command inside the folder you know or suspect to be large

tree --du -d -shaC | grep -Ev '(  *[^ ]* ){2}\[' | more

You press space to jump to the next page and use q to quit displaying additional pages.

Pure du solution

When you don’t have root access then just du will be all you can use

du --human-readable --max-depth=1 | sort --human-numeric-sort --reverse
20G ./wp-content
16M ./wp-includes
7.8M ./wp-admin

Here is another that shows more depth

 du --human-readable --max-depth=2 | sort --human-numeric-sort --reverse

20G     ./wp-content/uploads
20G     ./wp-content
20G     .
34M     ./wp-content/themes
16M     ./wp-includes
7.8M    ./wp-admin
5.6M    ./wp-includes/js
2.4M    ./wp-admin/css
2.3M    ./wp-admin/includes
1.6M    ./wp-admin/js
1.0M    ./wp-includes/ID3
688K    ./wp-includes/css
508K    ./wp-includes/SimplePie
480K    ./wp-content/houzez-new
464K    ./wp-admin/images
436K    ./wp-includes/rest-api
344K    ./wp-includes/Requests
344K    ./wp-includes/images
280K    ./wp-includes/certificates
256K    ./wp-content/plugins
208K    ./wp-includes/customize
204K    ./wp-admin/network
196K    ./wp-includes/fonts
144K    ./wp-includes/widgets
88K     ./wp-includes/Text
64K     ./wp-includes/random_compat
60K     ./wp-includes/IXR
56K     ./wp-includes/pomo
40K     ./wp-includes/theme-compat
36K     ./wp-admin/user
16K     ./wp-content/ai1wm-backups
12K     ./wp-admin/maint
4.0K    ./wp-content/upgrade

Here is another useful command that trims the . and includes a total

du --human-readable --max-depth=2 --total | sed 's/\.//g' | sort --human-numeric-sort --reverse

This will place the size in the right column instead of the left column

du --human-readable --max-depth=2 --total | sed 's/\.//g' | sort --human-numeric-sort --reverse | awk -v OFS='\t' '{print $2,$1}' | column -t

Here is my one-liner for listing directories and their subdirectories 1 level deep in descending order on systems without ncdu or tree using du only

FOLDERS=$(find . -type d -exec du --human-readable --summarize */ {} + | sort -hr | awk '{print $2}'| sed -e '/^\./d'); for FOLDER in $FOLDERS; do echo "$(tput setaf 2)$FOLDER$(tput sgr0)"; du --human-readable --max-depth=1 $FOLDER | sed "s#$FOLDER#-#g" | sort -hr | awk '{print $2,$1}' ; done | column -t

Will get you this output

wp-content/
-                     2.0G
-ai1wm-backups        1.1G
-uploads              832M
-plugins              113M
-themes               3.4M
-bwp-minify           1.6M
-updraft              956K
-sedlex               220K
-aiowps_backups       8.0K
-upgrade              4.0K
wp-includes/
-                     15M
-js                   5.5M
-ID3                  1.0M
-css                  688K
-SimplePie            508K
-rest-api             436K
-images               344K
-Requests             344K
-certificates         280K
-customize            208K
-fonts                196K
-widgets              92K
-Text                 88K
-random_compat        64K
-IXR                  60K
-pomo                 56K
-theme-compat         40K
codiad/
-                     8.9M
-components           7.3M
-themes               880K
-languages            252K
-js                   252K
-lib                  152K
-workspace            8.0K
-plugins              8.0K
-data                 8.0K
wp-admin/
-                     7.6M
-css                  2.4M
-includes             2.3M
-js                   1.5M
-images               464K
-network              204K
-user                 36K
-maint                12K

This is the last one

du -k -c --max-depth=1 | grep -v \\.$ | sort -rn| while read p; do echo "";      k=`echo $p|awk '{print $2}'` ;   if [ $k == "total" ] ; then echo ""; else  echo $k ; du -k  --max-depth=1 $k |sort -rn | while read p1; do     echo $p1| awk ' BEGIN { split("KB,MB,GB,TB", Units, ",");  } {   u = 1;  while ($1 >= 1024) { $1 = $1 / 1024; u += 1 ;  }  $2 = sprintf("\t\t%60.60s\t\t\t%9.1f%s", $2,$1, Units[u]); print $2}' ;   done;  fi ; done; du -k -c --max-depth=1| grep -i "total$"|awk '{$1 = sprintf("\t\t%60.60s\t\t\t%9.1f MB",$2,$1/1024); print $1}';

Enjoy 🙂

Sources

Combine the Best of tree and du
Unix Sort Command
du for subdirectories
Changing Color Output in bash