AikiWeb Aikido Forums

AikiWeb Aikido Forums (http://www.aikiweb.com/forums/index.php)
-   Announcements & Feedback (http://www.aikiweb.com/forums/forumdisplay.php?f=10)
-   -   Does Aikido work with robots and spiders? (http://www.aikiweb.com/forums/showthread.php?t=17367)

Marc Kupper 12-19-2009 07:06 PM

Does Aikido work with robots and spiders?
 
I did a Google search for something and was surprised to get a Page Changes page from the aikiweb wiki. Here are some suggestions for robots.txt under User-agent: *

# The wiki content is under the /wiki/ prefix and much of the
# editing, history, etc. is prefixed with /mediawiki/
Disallow: /mediawiki/

# A bit global but there's no reason to index any of the special
# pages. Plus many of them are computationally intensive.
Disallow: /wiki/Special:
Disallow: /wiki/Special%3A

# However, if consensus is that some/many of the special pages
# should be in the index then below are some specific special
# pages that should get blocked.

# A search engine will index some random page and when someone
# clicks to view that page in the search results they will
# likely get some other random page.
Disallow: /wiki/Special:Random
Disallow: /wiki/Special:RandomPage
Disallow: /wiki/Special:RandomRedirect

Disallow: /wiki/Special%3ARandom
Disallow: /wiki/Special%3ARandomPage
Disallow: /wiki/Special%3ARandomRedirect

# Search is computationaly intensive and does not return
# results useful to a search engine index.
Disallow: /wiki/Special:Search

Disallow: /wiki/Special%3ASearch

Marc

RED 12-21-2009 08:15 PM

Re: Does Aikido work with robots and spiders?
 
sweet

akiy 12-22-2009 10:06 AM

Re: Does Aikido work with robots and spiders?
 
Hi Marc,

Thanks for that. I've implemented some of your suggestions.

-- Jun


All times are GMT -6. The time now is 03:32 AM.

Powered by: vBulletin
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.