Every command, such as
cp
,mv
,rm
,tar
,rsync
,grep
, and so on, produces the error “command: argument list too long.” Why does this happen, and how do I fix it?
When Linux users try to run a command with too many arguments, they may get an error that states “argument list too long.”
This error happens because every system has a maximum argument list size, and when a certain command has too many arguments, the system throws an error message.
To fix this issue, users can employ a number of solutions, such as breaking up the command into smaller pieces by using a loop, piping the output of the command to a file, or utilising xargs to break up the command into manageable chunks.
But before we get to the part about the solutions, let me first show you what the problems look like and when they become active on your system.
Table of Contents
When would you see the “argument list is too long” error?
As I previously stated, commands like cp
, mv
, and rm
will throw an error if you try to manipulate a large number of files at once. To show you the error, allow me to use the touch
command to create a collection of 5,00,000 files all at once.
It is not possible for you and me to type five thousand unique file names into the terminal. However, by utilizing brace expansion, we can achieve the result shown below:
$ touch filename{1..500000}.txt
Also Read: The touch Command Does Much More Than Just Create an Empty File
And, as expected, the command prints an error on the screen as follows:
Somehow I managed to create a large number of files using this guide, but when I tried to list, move, copy, remove, or even try to create an archive, all failed like shown below.
Since everyone is on the same page about the issue now, let’s try to come up with a solution.
Fix the “argument list too long” error for all commands
In order to fix the “argument list too long error” for all commands, we should first understand what is causing it.
As you know, it occurs when the list of arguments exceeds the maximum number of arguments that the system can handle, but how can I know what that maximum is?
The answer can be found by running the getconf ARG_MAX
command, which will return the maximum number of arguments that can be passed to a command.
This lets us know the exact number of arguments our command can have, which may be different depending on the operating system, kernel version, and shell that are being used.
Once we know what the maximum argument size is, we can then adjust our commands accordingly to avoid going over that limit.
Command Section
One of the easiest ways to fix the “argument list too long” error is to use xargs
with the find
command, which will take the output of the find command and break it up into chunks that are within the maximum argument limit.
In addition, you can also use loop statements in your scripts to iterate through the output of find and break it into smaller chunks that can then be passed on to other commands.
If you can use the wildcard characters, then you can leverage them to manipulate files from the system on the basis of patterns, which will make sure that the argument list doesn’t go over the maximum limit
After much theoretical discussion, it’s time to get into the practical aspect of these solutions.
touch command solution
First of all, let’s create a large number of files using the touch
command, which failed above, and use the loop function to iterate and create 500k unique files.
To keep your current directory structure from getting messed up, I suggest that you run the following command on the new directory.
$ mkdir test-arena && cd test-arena
$ for i in {1..500000}; do touch filename$i.txt; done
cp command solution
Then let me use the find
command in conjunction with --exec or else pipe the output to xargs
to copy all of this file to a new directory:
$ find /home/shen/test-arena -name "filename*" -exec cp {} -vt /home/shen/test-dir \;
or
$ find /home/shen/test-arena -name "filename*" | xargs cp -vt /home/shen/test-dir
You can do the same with a “for loop”, but it is slow compared to the above one, but you can still try.
$ for i in /home/shen/test-arena/filename*; do cp "$i" -vt /home/shen/test-dir ; done
You can also use wildcard characters to copy a file from one directory to another by splitting the file. Let me take one simple example: copying a file from range 1 to 100 to another directory.
$ cp /home/shen/test-arena/filename{1..100}.txt -vt /home/shen/test-dir
mv command solution
But now I want to move files to a different location. In those scenarios, you can simply use the following command to move all large files at the same time:
$ find /home/shen/test-arena -name "filename*" -exec mv {} -vt /home/shen/test-dir \;
or
$ find /home/shen/test-arena -name "filename*" | xargs mv -vt /home/shen/test-dir
Is it possible to move all large files using a loop? Yes, you can do it by running the below command; just make sure to replace the file path with the actual one.
$ for i in /home/shen/test-arena/filename*; do mv "$i" -vt /home/shen/test-dir ; done
I don’t want to move all the files; I just want to move a specific range of files. Then execute the below command:
$ mv /home/shen/test-arena/filename{1..100}.txt -vt /home/shen/test-dir
rm command solution
After learning all this, let’s remove all the files from the system by suppressing the “argument list too long” error.
To complete the task, let me run the following command:
$ find ~/test-dir -name "filename*" -exec rm {} -v \;
or
$ find /home/shen/test-arena -name "filename*" | xargs rm -v
Also Read: What You Need to Know About the rm Command in Linux and Its Advanced Syntax
Even I can remove all the large files from my system using the loop mechanism:
$ for i in /home/shen/test-arena/filename*; do rm -v "$i" ; done
But if I don’t want to delete all large files except a specific range of files, I should use the following command:
$ rm -v /home/shen/test-arena/filename{1..100}.txt
Wrap up
These are a few ways to resolve the “Argument list too long” error. You can try different methods and choose the one that best suits your needs.
If you know any other ways to accomplish the same task, feel free to comment and share them with us so that we can learn from each other.
Anyway, if there is something I need to consider, please use the space below to provide any additional information or feedback.
Till then, put your feet up and relax. As always, the next article will be up shortly.
A man with a tech effusive who has explored some of the amazing technology stuff and is exploring more. While moving towards, I had a chance to work on Android development, Linux, AWS, and DevOps with several open-source tools.