I just did a change in the URL structure of a bigger site and had to check if all existing URLs are rewritten correctly to the new structure. First I extracted a list of all URLs from the CMS of the old version and wrote them to a text file, each line containing an URL. Then I changed the structure and used the following shellscript to check each URL with curl and output the result. Afterwards it’s easy to analyze the results and check for broken links.
#!/bin/sh while read f do echo $f echo "" curl -IL --silent $f echo "------------------" echo "" done < $1
$ ./checklinks.sh links.txt > stats.txt