recursively copying files into single dir broken down by alphabet

mattdawolf · 05-14-2024, 01:42 PM

How would I get find src/ -type f -name "*.so" -exec cp {} dst/ \;
to do the following:

copy all files in source and relative dirs to destiantion without relative paths, while renaming any potential duplicates to avoid data loss.

destination dir should be broken down into dirs with first letter.

There are 340000 files I need to go through.

Many thanks!

shruggy · 05-15-2024, 04:15 AM

Provided the one-letter-named directories already exist and every affected filename starts with a letter used to name those, I'd try something like this (it won't rename duplicates though)

Code:

find src/ -type f -name \*.so -execdir rename -n -v 's|^./(.)|dst/$1/$1|' {} \;

-n is there for testing it. Remove it to do the actual renaming.

This is Perl-based rename, it may be alternatively called prename or file-rename in your distribution. Don't confuse it with less capable rename from util-linux.

As it allows arbitrary Perl code in its expressions, you probably can adapt it to handle duplicate renaming as well.

syg00 · 05-15-2024, 04:27 AM

Quote:

Originally Posted by shruggy

Provided the one-letter-named directories already exist ...

Yes, I was also thinking this would make things much easier.

shruggy · 05-15-2024, 05:44 AM

Well, for anything more fancy, I'd probably write a shell/Perl script and call it from the -exec/-execdir action of find giving it the file name as parameter. The renaming and all the checks would then be done inside the script.

jmccue · 05-15-2024, 07:55 AM

Quote:

Originally Posted by shruggy

Well, for anything more fancy, I'd probably write a shell/Perl script and call it from the -exec/-execdir action of find giving it the file name as parameter. The renaming and all the checks would then be done inside the script.

I would follow this too, plus remember, you may run into duplicate file names, so you may need to account for that too.

MadeInGermany · 05-15-2024, 11:26 PM

find with 'embedded' bash code:

Code:

find src/ -type f -name "*.so" -exec /bin/bash -c '
  do=echo # for testing/dry-run
  destdir=${1%/}; shift
  for fn
  do
    basefn=${fn##*/}
    firstchar=${basefn:0:1}
    $do mkdir -p "$destdir/$firstchar"
    dupext= num=0
    while
      destfn="$destdir/$firstchar/$basefn$dupext"
      [ -e "$destfn" ]
    do
      dupext="($((++num)))"
    done   
    $do cp -v "$fn" "$destfn"
  done
' acp.bash dest/ {} +