View Full Version : Does Aikido work with robots and spiders?
AikiWeb Sponsored Links
Place your Aikido link here for only $10!
12-19-2009, 08:06 PM
I did a Google search for something and was surprised to get a Page Changes page from the aikiweb wiki. Here are some suggestions for robots.txt under User-agent: *
# The wiki content is under the /wiki/ prefix and much of the
# editing, history, etc. is prefixed with /mediawiki/
# A bit global but there's no reason to index any of the special
# pages. Plus many of them are computationally intensive.
# However, if consensus is that some/many of the special pages
# should be in the index then below are some specific special
# pages that should get blocked.
# A search engine will index some random page and when someone
# clicks to view that page in the search results they will
# likely get some other random page.
# Search is computationaly intensive and does not return
# results useful to a search engine index.
Thanks for that. I've implemented some of your suggestions.
vBulletin Copyright © 2000-2012 Jelsoft Enterprises Limited