Robots.txt for vbseo-enabled forum running Vbet 4.2.3

[norku]

I've been searching and searching but can't find tips on the content of my robots.txt. I'm running Vbet 4.2.3 with Vbseo 3.6.0 RC 2.

I assume that I need one similar to the one you guys are using?


Is this one okay or do I need the huge one you guys are using where you block access to each file by language?

# Allow Archiver
User-agent: ia_archiver
Allow: /

User-agent: *
Disallow: /forum/*.js
Disallow: /forum/*.jsp
Disallow: /forum/*.cfm
Disallow: /forum/*.asp
Disallow: /forum/*.aspx
Disallow: /forum/*.cgi
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/install/
Disallow: /forum/customavatars/
Disallow: /forum/archive/
Disallow: /forum/attachments/ 
Disallow: /forum/ajax.php
Disallow: /forum/attachment.php
Disallow: /forum/admincp/
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.php
Disallow: /forum/external.php
Disallow: /forum/image.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/billspaypal.php
Disallow: /forum/member.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php
Disallow: /forum/faq.php
Disallow: /forum/vbseo.php
Disallow: /forum/vbseocp.php
Disallow: /forum/vbseocpform.php
Disallow: /forum/chat.php

Sitemap: http://www.mydomain.com/forum/sitemap_index.xml.gz 

[kamilkurczak]

hello,
You should to disable this pages from your robots.txt file too.
Why?

This address:

/forum/calendar.php

Have the same content like this one:

/forum/de/calendar.php

but the url is different and google will index this url (this page) too.

You can use a robots.txt file from our example (the full one is in the attachment in this post - lower part )
http://www.vbenterprisetranslator.co...rformance.html

[norku]

Okay. Thank you. I'll change that now. I'm using the 8wayrun media library mod on my site. Media.php is redirected currently to /forum/media-library/ Is there anything I need to be careful of with this in respect of robots.txt and translations? I can start a new thread if you like?

[kamilkurczak]

We can give you an example of robots.txt (we already use the same like this one).
If you want to have this site disabled for google -> block it (you can block url to .php site and to redirected part to - if this is permanent redirect) if this is a 302 redirect (not permanent ) blonk only url to .php site.