Bash: how to traverse directory structure and execute commands?

Go To StackoverFlow.com

8

I have split a large text file into a number of sets of smaller ones for performance testing that i'm doing. There are a number of directories like this:

/home/brianly/output-02 (contains 2 files myfile.chunk.00 and myfile.chunk.01)
/home/brianly/output-04 (contains 4 files...)
/home/brianly/output-06 (contains 6 files...)

It's important to note that there is an increasing number of files in each directory. What I need to do is run an executable against each of the text files in the output directories. The command looks something like this against a single file:

./myexecutable -i /home/brianly/output-02/myfile.chunk.00 -o /home/brianly/output-02/myfile.chunk.00.processed

Here the -i parameter is the input file and -o parameter is the output location.

In C# I'd loop over the directories get the list of files in each folder, then loop over them to run the commandlines. How do I traverse a directory structure like this using bash, and execute the command with the correct parameters based on the location and files in that location?

2009-06-16 19:34
by Brian Lyttle


8

For this kind of thing I always use find together with xargs:

$ find output-* -name "*.chunk.??" | xargs -I{} ./myexecutable -i {} -o {}.processed

Now since your script processes only one file at a time, using -exec (or -execdir) directly with find, as already suggested, is just as efficient, but I'm used to using xargs, as that's generally much more efficient when feeding a command operating on many arguments at once. Thus it's a very useful tool to keep in one's utility belt, so I thought it ought to be mentioned.

2009-06-16 20:06
by Lars Haugseth


7

Something like:

for x in `find /home/brianonly -type f`
do
./yourexecutable -i $x -o $x.processed
done
2009-06-16 19:41
by nikudesu
there are back-ticks before find and after - nikudesu 2009-06-16 19:43
for x in $(find /home/brianonly -type f); do ... done

$() is so much more readable than back-ticks. My $.02 - Kevin Little 2009-06-16 20:46

I think yours would try to process the processed files - Dennis Williamson 2009-06-17 05:45
Dennis, you're right. I am standing by my disclaimer of: something like : - nikudesu 2009-06-17 11:36


2

As others have suggested, use find(1):

# Find all files named 'myfile.chunk.*' but NOT named 'myfile.chunk.*.processed'
# under the directory tree rooted at base-directory, and execute a command on
# them:
find base-directory -name 'output.*' '!' -name 'output.*.processed' -exec ./myexecutable -i '{}' -o '{}'.processed ';'
2009-06-16 19:56
by Adam Rosenfield


1

From the information provided, it sounds like this would be a completely straightforward translation of your C# idea.

for i in /home/brianly/output-*; do
    for j in "$i/"*.[0-9][0-9]; do
        ./myexecutable -i "$j" -o "$j.processed"
    done
done
2009-06-16 20:25
by ephemient


0

That's what the find command is for.

http://linux.die.net/man/1/find

2009-06-16 19:41
by S.Lott


0

Use find and exec. Have a look at following

http://tldp.org/LDP/abs/html/moreadv.html

2009-06-16 19:46
by Methos
Ads