Not long ago, I noticed a user complaining about the very philosophy of Unix (and Linux) tools – do one thing and do it well. As per him, the *nix systems are a combination of loosely coupled utilities which don’t work well together. Of course, as a hardcore terminal user, I wasn’t aligned. However, today I found solid reasons why Linux userland is best the way it is designed.
The building blocks matter. It’s like the alphabet – if you know the letters well you can build on top of those. There are complex utilities and there are needs for those. However, the building blocks form the base and are of extreme importance.
I upload my family pics to a website of my own… a few thousands in number taken over years. I also have a backup on a disk. To my surprise, I noticed this morning around 600 images are corrupt and showing 0 byte size.
There was no other way but to re-download these 600 files. I didn’t want to re-download everything. I could browse each (the URL pattern remains the same with the image name at the end) and download one by one. I started doing it thinking I would finish in a few sittings.
After downloading around 5 or 6, I got back my calm and started thinking of a better approach. Thanks to these decoupled utilities, those 600 odd files got downloaded in the next 15 minutes.
This is what I did from the terminal:
- Go to the directory with the images
$ cd family_pics
- Get the names of all 0 byte images, remove first 2 chars from each line, redirect to a file
$ mkdir ~/tmpdir $ find . -size 0 | sed 's#^./##' > ~/tmpdir/input.txt
- Add the common URL at the beginning of each line in input.txt
$ sed -i 's#^#http://ourfamilyalbums.com/images/full/#' ~/tmpdir/input.txt
- Download using aria
$ cd ~/tmpdir $ aria2c -x 5 -i input.txt
As I mentioned earlier, all the corrupt files were back in 15 minutes! Thanks Linux!