View Single Post
Old 12-19-2009, 07:06 PM   #1
Marc Kupper
Dojo: Aikido of Diablo Valley / ASU
Location: Walnut Creek, California
Join Date: May 2002
Posts: 88
United_States
Offline
Does Aikido work with robots and spiders?

I did a Google search for something and was surprised to get a Page Changes page from the aikiweb wiki. Here are some suggestions for robots.txt under User-agent: *

# The wiki content is under the /wiki/ prefix and much of the
# editing, history, etc. is prefixed with /mediawiki/
Disallow: /mediawiki/

# A bit global but there's no reason to index any of the special
# pages. Plus many of them are computationally intensive.
Disallow: /wiki/Special:
Disallow: /wiki/Special%3A

# However, if consensus is that some/many of the special pages
# should be in the index then below are some specific special
# pages that should get blocked.

# A search engine will index some random page and when someone
# clicks to view that page in the search results they will
# likely get some other random page.
Disallow: /wiki/Special:Random
Disallow: /wiki/Special:RandomPage
Disallow: /wiki/Special:RandomRedirect

Disallow: /wiki/Special%3ARandom
Disallow: /wiki/Special%3ARandomPage
Disallow: /wiki/Special%3ARandomRedirect

# Search is computationaly intensive and does not return
# results useful to a search engine index.
Disallow: /wiki/Special:Search

Disallow: /wiki/Special%3ASearch

Marc
  Reply With Quote