I just did a change in the URL structure of a bigger site and had to check if all existing URLs are rewritten correctly to the new structure. First I extracted a list of all URLs from the CMS of the old version and wrote them to a text file, each line containing an URL. Then I changed the structure and used the following shellscript to check each URL with curl and output the result. Afterwards it’s easy to analyze the results and check for broken links.

#!/bin/sh
while read f
do
  echo $f
  echo ""
  curl -IL --silent $f
  echo "------------------"
  echo ""
done < $1

Usage:

$ ./checklinks.sh links.txt > stats.txt