Every command, such as cp, mv, rm, tar, rsync, grep, and so on, produces the error “command: argument list too long.” Why does this happen, and how do I fix it?

When Linux users try to run a command with too many arguments, they may get an error that states “argument list too long.”

This error happens because every system has a maximum argument list size, and when a certain command has too many arguments, the system throws an error message.

To fix this issue, users can employ a number of solutions, such as breaking up the command into smaller pieces by using a loop, piping the output of the command to a file, or utilising xargs to break up the command into manageable chunks.

But before we get to the part about the solutions, let me first show you what the problems look like and when they become active on your system.

When would you see the “argument list is too long” error?

As I previously stated, commands like cp, mv, and rm will throw an error if you try to manipulate a large number of files at once. To show you the error, allow me to use the touch command to create a collection of 5,00,000 files all at once.

It is not possible for you and me to type five thousand unique file names into the terminal. However, by utilizing brace expansion, we can achieve the result shown below:

$ touch filename{1..500000}.txt

Also Read: The touch Command Does Much More Than Just Create an Empty File

And, as expected, the command prints an error on the screen as follows:

Somehow I managed to create a large number of files using this guide, but when I tried to list, move, copy, remove, or even try to create an archive, all failed like shown below.

Since everyone is on the same page about the issue now, let’s try to come up with a solution.

Fix the “argument list too long” error for all commands

In order to fix the “argument list too long error” for all commands, we should first understand what is causing it.

As you know, it occurs when the list of arguments exceeds the maximum number of arguments that the system can handle, but how can I know what that maximum is?

The answer can be found by running the getconf ARG_MAX command, which will return the maximum number of arguments that can be passed to a command.

This lets us know the exact number of arguments our command can have, which may be different depending on the operating system, kernel version, and shell that are being used.

Once we know what the maximum argument size is, we can then adjust our commands accordingly to avoid going over that limit.

Command Section

One of the easiest ways to fix the “argument list too long” error is to use xargs with the find command, which will take the output of the find command and break it up into chunks that are within the maximum argument limit.

In addition, you can also use loop statements in your scripts to iterate through the output of find and break it into smaller chunks that can then be passed on to other commands.

If you can use the wildcard characters, then you can leverage them to manipulate files from the system on the basis of patterns, which will make sure that the argument list doesn’t go over the maximum limit

After much theoretical discussion, it’s time to get into the practical aspect of these solutions.

touch command solution

First of all, let’s create a large number of files using the touch command, which failed above, and use the loop function to iterate and create 500k unique files.

To keep your current directory structure from getting messed up, I suggest that you run the following command on the new directory.

$ mkdir test-arena && cd test-arena 
$ for i in {1..500000}; do touch filename$i.txt; done

cp command solution

Then let me use the find command in conjunction with --exec or else pipe the output to xargs to copy all of this file to a new directory:

$ find /home/shen/test-arena -name "filename*" -exec cp {} -vt /home/shen/test-dir \;
or
$ find /home/shen/test-arena -name "filename*" | xargs cp -vt /home/shen/test-dir

You can do the same with a “for loop”, but it is slow compared to the above one, but you can still try.

$ for i in /home/shen/test-arena/filename*; do cp "$i" -vt /home/shen/test-dir ; done

You can also use wildcard characters to copy a file from one directory to another by splitting the file. Let me take one simple example: copying a file from range 1 to 100 to another directory.

$ cp /home/shen/test-arena/filename{1..100}.txt -vt /home/shen/test-dir

mv command solution

But now I want to move files to a different location. In those scenarios, you can simply use the following command to move all large files at the same time:

$ find /home/shen/test-arena -name "filename*" -exec mv {} -vt /home/shen/test-dir \;
or
$ find /home/shen/test-arena -name "filename*" | xargs mv -vt /home/shen/test-dir

Is it possible to move all large files using a loop? Yes, you can do it by running the below command; just make sure to replace the file path with the actual one.

$ for i in /home/shen/test-arena/filename*; do mv "$i" -vt /home/shen/test-dir ; done

I don’t want to move all the files; I just want to move a specific range of files. Then execute the below command:

$ mv /home/shen/test-arena/filename{1..100}.txt -vt /home/shen/test-dir

rm command solution

After learning all this, let’s remove all the files from the system by suppressing the “argument list too long” error.

To complete the task, let me run the following command:

$ find ~/test-dir -name "filename*" -exec rm {} -v \;
or
$ find /home/shen/test-arena -name "filename*" | xargs rm -v

Also Read: What You Need to Know About the rm Command in Linux and Its Advanced Syntax

Even I can remove all the large files from my system using the loop mechanism:

$ for i in /home/shen/test-arena/filename*; do rm -v "$i" ; done

But if I don’t want to delete all large files except a specific range of files, I should use the following command:

$ rm -v /home/shen/test-arena/filename{1..100}.txt

Wrap up

These are a few ways to resolve the “Argument list too long” error. You can try different methods and choose the one that best suits your needs.

If you know any other ways to accomplish the same task, feel free to comment and share them with us so that we can learn from each other.

Anyway, if there is something I need to consider, please use the space below to provide any additional information or feedback.

Till then, put your feet up and relax. As always, the next article will be up shortly.