You might want to use 'find' because 'ls' relies on how much space the shell has for arguments. eg: to show all the files in the current directory....noting..ie: you can elect to pass relative or absolute path and the output will reflect that. The parameter "-type f" restricts to files only (no folders). Consider..Now create some folders....but 'b' is missing and that's where "-maxdepth" came into play. It restricted the search to a single depth of folder recursion. Remove it to see everything..The "-mindepth" seems weird on first sight. Remove it..Note the "." folder has appeared (aka 'ls -la'). It's often a bug in a script to see that because it can cause the script to loop forever on the same folder.
The above covers the most common gotchas for 'find' which cause people to give up on it. The man page is complex. All you need is....where $DIR is an absolute path like "/home/john".If we put that into a script....which yields (for me)..Now you've got a list of all files which exist anywhere down "$DIR", omitting empty folders.
As you mention sdcards and data/picture I'll assume these are camera files so you have no choice over the filenames and folders.If we look at "$CDB" (common database) it contains..If we accidentally index the same sdcard again....we can dismiss the "duplicates" via..Our notion of "duplicate" is not the real definition. The checksum shows, for example "f1" and "f2" are the same but our definition of duplicate includes also that the filename be identical because I suspect you desire to know which sdcard holds the given file. We're missing one thing: the sdcard number which you've likely written on each sdcard. Add that as a second argument to the script. This makes things more complicated in that we ought to think about adjusting the script to be more maintainable but does mean we can add features. Here's an idea how the final thing should be progressing..Adjust $CDB to where you want your big database to be stored as currently it will vanish on reboot. I also loath using temporary files with fixed names because if you run two instances at the same time they will corrupt each other. Nevertheless, updated output from the above follows. I deleted the hardlink "zH" because they won't exist on fat formatted sdcards.Now pretend we have a second sdcard with (somehow) same contents..I attached "z.zip". Hope that helps. ![:-)]( "Smile")
Code:
foo@sdu:/wrk/T/T$ find . -mindepth 1 -maxdepth 1 -type f./zCode:
foo@sdu:/wrk/T/T$ find `pwd` -mindepth 1 -maxdepth 1 -type f/wrk/T/T/zCode:
foo@sdu:/wrk/T/T$ ln -s z zS#symbolic linkfoo@sdu:/wrk/T/T$ ln z zH#hard linkfoo@sdu:/wrk/T/T$ ls -ltotal 8-rwxrw-r-- 2 foo foo 57 Oct 1 16:06 z-rwxrw-r-- 2 foo foo 57 Oct 1 16:06 zHlrwxrwxrwx 1 foo foo 1 Oct 1 16:26 zS -> zfoo@sdu:/wrk/T/T$ find . -mindepth 1 -maxdepth 1 -type f./zH./z#^^^a symbolic link is not a filefoo@sdu:/wrk/T/T$ find . -mindepth 1 -maxdepth 1./zS./zH./z#^^^list everythingCode:
foo@sdu:/wrk/T/T$ mkdir -p a/b cfoo@sdu:/wrk/T/T$ find . -mindepth 1 -maxdepth 1 -type f./zH./zfoo@sdu:/wrk/T/T$ find . -mindepth 1 -maxdepth 1./c./zS./zH./z./aCode:
foo@sdu:/wrk/T/T$ find . -mindepth 1./c./zS./zH./z./a./a/bCode:
foo@sdu:/wrk/T/T$ find .../c./zS./zH./z./a./a/bThe above covers the most common gotchas for 'find' which cause people to give up on it. The man page is complex. All you need is..
Code:
$ find "$DIR" -mindepth 1 -type fCode:
foo@sdu:/wrk/T/T$ touch a/f1 a/b/f2foo@sdu:/wrk/T/T$ find . -mindepth 1 -type f./zH./z./a/f1./a/b/f2Code:
foo@sdu:/wrk/T/T$ cat z#!/bin/bashDIR=`pwd`find "$DIR" -mindepth 1 -type fCode:
foo@sdu:/wrk/T/T$ chmod u+x zfoo@sdu:/wrk/T/T$ ./z/wrk/T/T/zH/wrk/T/T/z/wrk/T/T/a/f1/wrk/T/T/a/b/f2As you mention sdcards and data/picture I'll assume these are camera files so you have no choice over the filenames and folders.
Code:
foo@sdu:/wrk/T/T$ cat z#!/bin/bashDIR=`pwd`TMP="/tmp/"`basename "$0"`CDB="/tmp/cdb"find "$DIR" -mindepth 1 -type f > "$TMP"while IFS= read -r idomd5sum "$i" >> "$CDB"done < "$TMP"Code:
foo@sdu:/wrk/T/T$ cat /tmp/cdb a177fc09bdccd899c88951f1b3eed2ae /wrk/T/T/zHa177fc09bdccd899c88951f1b3eed2ae /wrk/T/T/zd41d8cd98f00b204e9800998ecf8427e /wrk/T/T/a/f1d41d8cd98f00b204e9800998ecf8427e /wrk/T/T/a/b/f2Code:
foo@sdu:/wrk/T/T$ ./zfoo@sdu:/wrk/T/T$ cat /tmp/cdb a177fc09bdccd899c88951f1b3eed2ae /wrk/T/T/zHa177fc09bdccd899c88951f1b3eed2ae /wrk/T/T/zd41d8cd98f00b204e9800998ecf8427e /wrk/T/T/a/f1d41d8cd98f00b204e9800998ecf8427e /wrk/T/T/a/b/f2a177fc09bdccd899c88951f1b3eed2ae /wrk/T/T/zHa177fc09bdccd899c88951f1b3eed2ae /wrk/T/T/zd41d8cd98f00b204e9800998ecf8427e /wrk/T/T/a/f1d41d8cd98f00b204e9800998ecf8427e /wrk/T/T/a/b/f2Code:
foo@sdu:/wrk/T/T$ sort /tmp/cdb | uniqa177fc09bdccd899c88951f1b3eed2ae /wrk/T/T/za177fc09bdccd899c88951f1b3eed2ae /wrk/T/T/zHd41d8cd98f00b204e9800998ecf8427e /wrk/T/T/a/b/f2d41d8cd98f00b204e9800998ecf8427e /wrk/T/T/a/f1Code:
#!/bin/bashNAM=`basename "$0"`TMP="/tmp/""$NAM"CDB="/tmp/cdb"fcp_usage (){cat<<EOF${NAM}: [ --help | -h ]${NAM}: [ --scan | -s ] [ folder] [ sdcard number ]${NAM}:${NAM}: (full explanation here)EOF}fcp_out (){ local i r local c s f while IFS= read -r i do r=$(md5sum "$i") c=$(echo "$r" | awk '{print $1}') f=$(echo "$r" | awk '{print $2}') printf -v s "%04u" "$1" echo "$c" "$s" "$f" >> "$CDB" done < "$TMP"}fcp_scan (){ [ $# -lt 2 ] && { fcp_usage 1>&2 exit 1 } find "$1" -mindepth 1 -type f > "$TMP" fcp_out "$2"}fcp_sorted (){ sort "$CDB" | uniq}case "$1" in --help | -h) fcp_usage exit 0 ;; --scan | -s) shift fcp_scan "$@" ;; --raw | -r) cat "$CDB" ;; *) fcp_sorted ;;esacCode:
foo@sdu:/wrk/T/T$ rm -f /tmp/cdbfoo@sdu:/wrk/T/T$ ./z --scan . 0foo@sdu:/wrk/T/T$ ./z --rawdc295811aafe62be266cf48562607e46 0000 ./zd41d8cd98f00b204e9800998ecf8427e 0000 ./a/f1d41d8cd98f00b204e9800998ecf8427e 0000 ./a/b/f2foo@sdu:/wrk/T/T$ ./z --scan . 0foo@sdu:/wrk/T/T$ ./z --rawdc295811aafe62be266cf48562607e46 0000 ./zd41d8cd98f00b204e9800998ecf8427e 0000 ./a/f1d41d8cd98f00b204e9800998ecf8427e 0000 ./a/b/f2dc295811aafe62be266cf48562607e46 0000 ./zd41d8cd98f00b204e9800998ecf8427e 0000 ./a/f1d41d8cd98f00b204e9800998ecf8427e 0000 ./a/b/f2foo@sdu:/wrk/T/T$ ./zd41d8cd98f00b204e9800998ecf8427e 0000 ./a/b/f2d41d8cd98f00b204e9800998ecf8427e 0000 ./a/f1dc295811aafe62be266cf48562607e46 0000 ./zCode:
foo@sdu:/wrk/T/T$ ./z --scan . 23foo@sdu:/wrk/T/T$ ./z --rawdc295811aafe62be266cf48562607e46 0000 ./zd41d8cd98f00b204e9800998ecf8427e 0000 ./a/f1d41d8cd98f00b204e9800998ecf8427e 0000 ./a/b/f2dc295811aafe62be266cf48562607e46 0000 ./zd41d8cd98f00b204e9800998ecf8427e 0000 ./a/f1d41d8cd98f00b204e9800998ecf8427e 0000 ./a/b/f2dc295811aafe62be266cf48562607e46 0023 ./zd41d8cd98f00b204e9800998ecf8427e 0023 ./a/f1d41d8cd98f00b204e9800998ecf8427e 0023 ./a/b/f2foo@sdu:/wrk/T/T$ ./zd41d8cd98f00b204e9800998ecf8427e 0000 ./a/b/f2d41d8cd98f00b204e9800998ecf8427e 0000 ./a/f1d41d8cd98f00b204e9800998ecf8427e 0023 ./a/b/f2d41d8cd98f00b204e9800998ecf8427e 0023 ./a/f1dc295811aafe62be266cf48562607e46 0000 ./zdc295811aafe62be266cf48562607e46 0023 ./z
Statistics: Posted by swampdog — Wed Oct 01, 2025 5:47 pm