PDA

View Full Version : Does Aikido work with robots and spiders?


Please visit our sponsor:
 

AikiWeb Sponsored Links - Place your Aikido link here for only $10!


Marc Kupper
12-19-2009, 08:06 PM
I did a Google search for something and was surprised to get a Page Changes page from the aikiweb wiki. Here are some suggestions for robots.txt under User-agent: *

# The wiki content is under the /wiki/ prefix and much of the
# editing, history, etc. is prefixed with /mediawiki/
Disallow: /mediawiki/

# A bit global but there's no reason to index any of the special
# pages. Plus many of them are computationally intensive.
Disallow: /wiki/Special:
Disallow: /wiki/Special%3A

# However, if consensus is that some/many of the special pages
# should be in the index then below are some specific special
# pages that should get blocked.

# A search engine will index some random page and when someone
# clicks to view that page in the search results they will
# likely get some other random page.
Disallow: /wiki/Special:Random
Disallow: /wiki/Special:RandomPage
Disallow: /wiki/Special:RandomRedirect

Disallow: /wiki/Special%3ARandom
Disallow: /wiki/Special%3ARandomPage
Disallow: /wiki/Special%3ARandomRedirect

# Search is computationaly intensive and does not return
# results useful to a search engine index.
Disallow: /wiki/Special:Search

Disallow: /wiki/Special%3ASearch

Marc

RED
12-21-2009, 09:15 PM
sweet

akiy
12-22-2009, 11:06 AM
Hi Marc,

Thanks for that. I've implemented some of your suggestions.

-- Jun