View Full Version : Does Aikido work with robots and spiders?

Please visit our sponsor:

AikiWeb Sponsored Links - Place your Aikido link here for only $10!

Marc Kupper
12-19-2009, 07:06 PM
I did a Google search for something and was surprised to get a Page Changes page from the aikiweb wiki. Here are some suggestions for robots.txt under User-agent: *

# The wiki content is under the /wiki/ prefix and much of the
# editing, history, etc. is prefixed with /mediawiki/
Disallow: /mediawiki/

# A bit global but there's no reason to index any of the special
# pages. Plus many of them are computationally intensive.
Disallow: /wiki/Special:
Disallow: /wiki/Special%3A

# However, if consensus is that some/many of the special pages
# should be in the index then below are some specific special
# pages that should get blocked.

# A search engine will index some random page and when someone
# clicks to view that page in the search results they will
# likely get some other random page.
Disallow: /wiki/Special:Random
Disallow: /wiki/Special:RandomPage
Disallow: /wiki/Special:RandomRedirect

Disallow: /wiki/Special%3ARandom
Disallow: /wiki/Special%3ARandomPage
Disallow: /wiki/Special%3ARandomRedirect

# Search is computationaly intensive and does not return
# results useful to a search engine index.
Disallow: /wiki/Special:Search

Disallow: /wiki/Special%3ASearch


12-21-2009, 08:15 PM

12-22-2009, 10:06 AM
Hi Marc,

Thanks for that. I've implemented some of your suggestions.

-- Jun