My favourite robots.txt so far is this one: https://www.hackthissite.org/robots.txt