Robots.txt
Posted at
November 24th, 2005
As mentioned on
november
22nd, we wanted to include support for robots.txt
files. So now
robots.txt
files are fetched and obeyed! It was prety useless to
spider parts of your site to put in a sitemap if other
spiders (Googlebot et al) are not able to fetch does pages
too. It's still in beta and tested further. Feedback is
appreciated.
Back to the new archive |
Back to the homepage.
