Google Email Warning: Googlebot found an extremely high number of URLs on your site

[vBET]

1. Please turn on Italian flag on your forum (I didn't find it) - right now users who changed language are not able to turn back to original language. As I wrote content of threads which I was able to see before I saw that I have to register was only one sentence in all cases. This it not what personally I assume as average length of content. Once again this makes pages looks very similar since the difference between short threads is only one sentence - the rest is layout which is same. I do not know how you are so sure that it has nothing to do with this case. If you know something what we do not please just share it so we will be able to help you faster. Do you know why this email was sent?

2. I do not knot what is school program for children in your country, but I hope that it will be very useful for them to know that please register mod after 3 visits has nothing to do with the bots as they go through.
Please focus on the issue. And give required answer - I didn't know which mod are you using and I didn't even ask because it is not important for me - important was is it boot safe. If you are 100% sure that it is safe then great. Just in case you had any doubts you can see how Google sees your forum in Webmaster Tools.

Please note that Google boots are not attacking any sites - Google do not harm sites. We advice to use cache for longer allowed period, because it is better for performance. Also if you feel that it for sure the only reason is number of translation on your page - feel free to disable some of those and enable it again one by one from time to time. Still please consider that your license is active for 6 months and you got this email now. It is hard to believe that for 6 months Google didn't saw those links - in fact many of those you have already indexed. So I'm not sure that this is the cause of email which you got. Even if it is please consider all information which was given to you after analyze of those links.

I do not know what you mean by effective threads.

Also please note that we are working on this issue - you got results of our analyze. And please hold with telling what Google considers - because it is you who considers that Google find it not correct. WE have lot of forums which have absolutely no issue with Google and have all languages turned on. And it is not true that content is not physically exists, please do not tell what we can argue because something... The content exists for everyone who visits it and it physically exists in database in original and in cache in translated form. This is not worlds lottery - this is translation it couldn't happen if content didn't exists. And nothing is cheating here - please read Google TOS - this is Google who claims that translation can be cached for max 15 days. It is EVERYTHING OK with Google in case of automatic page translations - it is Google who made this translation

Also no client can tell that any content which is streamed to him is from static file or not. In fact in this days almost nothing is static now. In vBulletin no view hoes from static HTML structure - everything base on data from database (languages, templates, styles). I have no idea what is your knowledge about client server architecture with thin client, but please note that content exists if link responses. Telling that Google feels it cheating because it doesn't exists has nothing to do with reality.

Also please note that You can in every moment use vBET SEO links and have /en/ instead of language= - this configuration is just up to you and vBET supports it without vBSEO.

About note which you gave - as I wrote if you feel you discovered the real reason of your issue then you have solution. Still please consider that this is some generic answer without even checking what happens. I wouldn’t count on it:
1 - you have similar amount of links for half year.
2 - Google mail itself tells:
"Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot."

So once again please consider our advices and analyze result. If you feel it is only issue of number of translations which you turned on - then you know the solution.

Do you still need our help/advise in this issue?

[mario06]

The only thing that was turned off was the Italian language (I turned it on) since it is the forums native language I didn't think it mattered anyway I was good since a year ago so whats up now?

Here's an example (your site) of what I mean when I say effective indexed posts.

Before vBET (effective forum indexed stats):


After vBET (not effective forum indexed stats):


What I'm trying to say is that the 52x language translated posts are now being considered duplicate posts by Google. The email is self explanatory

In my own words when I say a bot attack I mean when you have 300 Google bots heavily crawling your website at the same time eating and tearing away on your bandwidth, ram and cpu and causing forum stalls. And they wrote it in their letter below in a more professional manner but the meaning is the same.

Official Google Email:
Googlebot encountered problems while crawling your site http://mysite.com/.
Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

Did you read what the Google support team wrote me? -> In general, this message is shown when we discover a relatively high number of URLs for your site.

You wrote everything and nothing because I don't understand your English, just give me a fix in simple steps like;
1. Do this
2. Do that
3. And this
etc.

Regards

[mario06]

Still waiting for a remedy.

Regards

[vBET]

The only thing that was turned off was the Italian language (I turned it on) since it is the forums native language I didn't think it mattered anyway I was good since a year ago so whats up now?

It is just advice from our team that without this flag your users are not able to turn back to original language when they click on some flag. So it is wise to keep flag for your default language

Here's an example (your site) of what I mean when I say effective indexed posts.

Before vBET (effective forum indexed stats):


After vBET (not effective forum indexed stats):


What I'm trying to say is that the 52x language translated posts are now being considered duplicate posts by Google. The email is self explanatory

This is NOT true. You have issues with your forum. You have broken archive - and most links you got from Google in email are about archive. You totally ignored our support - you didn't answer questions about version and keep telling what you think and what is not true. If you want to solve your issue please fallow our support hints. We asked you about your vB version because you have broken archive. Is it supported 3.8 or some not supported version? We know that archive is broken in 3.6 because of vBulletin bug. This gives you thousands of not working links - and you got from Google email about wrong links. Instead of listen to our support and cooperate you keep telling some ridiculous idea and you think it is important to put it in red.
What is your purpose here? You want our support to solve your issue or you are just trying to scare others? Because if you want to solve this issue please cooperate with our support.

Also about image which you gave - this is your misinterpretation. Google doesn’t care about number of posts shown by forum. And it cannot care about it. It is because Google indexes all the pages which are not blocked! On forum you have member pages, groups, bogs perhaps, albums and so on... and so on... Also forum can have lot of other plugins which creates additional pages (like arcade).
So please understand: you are wrong. Google do not consider translated content as duplication. You have issue with your own forum and we discovered areas of possible cause. So please - fallow our instructions/questions. Otherwise this can be marked as solved since you seem be not interested with our support just like to repeat what you wrongly think - we already read this - there is no any need to repeat it again. This is not true. Please focus on solving your issue if you want it to be solved. Please answer the question and keep attention on your archive.

Official Google Email:
Googlebot encountered problems while crawling your site http://mysite.com/.
Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

EXACTLY! YOU HAVE WRONG STRUCTURE! YOU HAVE WRONG CONTENT! and we wrote it to you at the very beginning (http://www.vbenterprisetranslator.co....html#post6846) - you have crashed archive where are thousands of not working links! So please start using our support which you asked for instead of trying to scare our users. What are you expecting from us when you ask for support and ignore it?

You wrote everything and nothing because I don't understand your English, just give me a fix in simple steps like;
1. Do this
2. Do that
3. And this
etc.

Regards

Here you got exact information that you should block member pages and how to do this:
http://www.vbenterprisetranslator.co....html#post6845

Here you got information that your archive is crashed, question about your version of vBulletin and what to do if it is not supported version:
http://www.vbenterprisetranslator.co....html#post6846
This is very exact instruction - please tell which version of vBulletin are you using.

And here you got information that you jut shouldn't do anything with posts which are very short (just 1 sentence) because this is issue of the content not of vBET. When content will get longer (users will get replies) those pages content will be not similar anymore:
http://www.vbenterprisetranslator.co....html#post6847

So:
1. Keep your member pages blocked in way like you made it recently (only for registered users) or by robots.txt like it is described here: http://www.vbenterprisetranslator.co....html#post1178
2. Check in Webmaster Tools does your "3 visits for Guest" mod is Google save – check how Google see your forum by giving URL to some thread. Check several times (cannot tell does Google will use same IP for each check). This way you will be really sure that the mod is save for robots.
3. You have broken archive – please tell which version of vBulletin are you using?
4. For short threads you can just give some longer answers there (different of course) or just keep it as it is hoping that people will answer those threads.

PS.
Please take my appologies for late answer. I was at delegation and Kamil was told to leave this issue to me. So once again sorry

[pepito69]

Hello,
I have to say that Michal is right, i had the same warning from google and it's easy to solve if you look at waht URLs are causing the problem. In my example, i had a lot of duplicated content because vbet inset: &language=en, in a French URL. If you go to parameters in webmaster tools you can solve this excluding this parameter. And also look at your robots file. I also have disallow in robots archive files, don't need it. There is a lot of work to do with robots.txt if you use VBET.
And i don't suggest to use more than 20 languages, you don't need it

[mario06]

Double Post, Sorry!!

[mario06]

It is just advice from our team that without this flag your users are not able to turn back to original language when they click on some flag. So it is wise to keep flag for your default language



This is NOT true. You have issues with your forum. You have broken archive - and most links you got from Google in email are about archive. You totally ignored our support - you didn't answer questions about version and keep telling what you think and what is not true. If you want to solve your issue please fallow our support hints. We asked you about your vB version because you have broken archive. Is it supported 3.8 or some not supported version? We know that archive is broken in 3.6 because of vBulletin bug. This gives you thousands of not working links - and you got from Google email about wrong links. Instead of listen to our support and cooperate you keep telling some ridiculous idea and you think it is important to put it in red.
What is your purpose here? You want our support to solve your issue or you are just trying to scare others? Because if you want to solve this issue please cooperate with our support.

Also about image which you gave - this is your misinterpretation. Google doesn’t care about number of posts shown by forum. And it cannot care about it. It is because Google indexes all the pages which are not blocked! On forum you have member pages, groups, bogs perhaps, albums and so on... and so on... Also forum can have lot of other plugins which creates additional pages (like arcade).
So please understand: you are wrong. Google do not consider translated content as duplication. You have issue with your own forum and we discovered areas of possible cause. So please - fallow our instructions/questions. Otherwise this can be marked as solved since you seem be not interested with our support just like to repeat what you wrongly think - we already read this - there is no any need to repeat it again. This is not true. Please focus on solving your issue if you want it to be solved. Please answer the question and keep attention on your archive.



EXACTLY! YOU HAVE WRONG STRUCTURE! YOU HAVE WRONG CONTENT! and we wrote it to you at the very beginning (http://www.vbenterprisetranslator.co....html#post6846) - you have crashed archive where are thousands of not working links! So please start using our support which you asked for instead of trying to scare our users. What are you expecting from us when you ask for support and ignore it?



Here you got exact information that you should block member pages and how to do this:
http://www.vbenterprisetranslator.co....html#post6845

Here you got information that your archive is crashed, question about your version of vBulletin and what to do if it is not supported version:
http://www.vbenterprisetranslator.co....html#post6846
This is very exact instruction - please tell which version of vBulletin are you using.

And here you got information that you jut shouldn't do anything with posts which are very short (just 1 sentence) because this is issue of the content not of vBET. When content will get longer (users will get replies) those pages content will be not similar anymore:
http://www.vbenterprisetranslator.co....html#post6847

So:
1. Keep your member pages blocked in way like you made it recently (only for registered users) or by robots.txt like it is described here: http://www.vbenterprisetranslator.co....html#post1178
2. Check in Webmaster Tools does your "3 visits for Guest" mod is Google save – check how Google see your forum by giving URL to some thread. Check several times (cannot tell does Google will use same IP for each check). This way you will be really sure that the mod is save for robots.
3. You have broken archive – please tell which version of vBulletin are you using?
4. For short threads you can just give some longer answers there (different of course) or just keep it as it is hoping that people will answer those threads.

PS.
Please take my appologies for late answer. I was at delegation and Kamil was told to leave this issue to me. So once again sorry

1. My robots.txt has been the same, I didn't change anything and it seems fine:
User-agent: *
Disallow: /admincp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /members/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /modcp/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
2. No it doesn't save
3. My archive works fine, all links tested and working, where did you see it broken?
4. The only small posts are where the new members present themselves and just say "Hello, my name is Tom" I can't do anything about, 90% of the posts are guides meaning medium sized.


pepito69 said something interesting which involves the parameter url like I said above

In my example, i had a lot of duplicated content because vbet inset: &language=en, in a French URL. If you go to parameters in webmaster tools you can solve this excluding this parameter.

but he only got it for one language I'm getting it for all languages so is this the reason and what must I do exactly?

PS. I don't have vbSEO installed BUT use vBSEO sitemap which already excludes the archive, am I suppose to exclude it the robots.txt to? I never seen anyone here or any other place exclude it in their robots.txt

I want simple detailed instructions,
EXAMPLE:

1. Do this
2. Do that
3. And that

pepito69,
I never seen such a big robots.txt as you have, is all that really necessary?
In realty there are only 3 bots that are considered Bad Bots* and it wouldn't it be easier to put them in your .htaccess file, no?
PS. Your flags are not showing

*Baiduspider
*Mozilla/5(.*)Twiceler
*Yeti

[pepito69]

Thanks Mario,

Yes, my robots file maybe is to big, isn't optimized but it works for me and don't hurt me.
I will include this bots in my htaccess, thanks.

but he only got it for one language I'm getting it for all language

No my friend, i also have it in all languages, that was just an example. But, since i used the parameters in webmaster tools i have increased my google crawl again. I still don't know how to solve other problems with VBET, more important for me, like the obe i have with the code language and the broken words that use some accents (ó, ñ, etc).
Well, i exclude archive in robots.txt, and i don't have any archive webpage in google index.
I thought many times to use vbseo sitemap but my site is very well crawled buy google. I always apply this rule: if something works, don't touch it.

Sorry i can not tell you what you must do, i only share my experience.

Thanks . My flags don't show in /index.php but are visible in the forums site, i will solve it now.

[vBET]

1. My robots.txt has been the same, I didn't change anything and it seems fine:
User-agent: *
Disallow: /admincp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /members/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /modcp/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
2. No it doesn't save
3. My archive works fine, all links tested and working, where did you see it broken?
4. The only small posts are where the new members present themselves and just say "Hello, my name is Tom" I can't do anything about, 90% of the posts are guides meaning medium sized.


pepito69 said something interesting which involves the parameter url like I said above
but he only got it for one language I'm getting it for all languages so is this the reason and what must I do exactly?

PS. I don't have vbSEO installed BUT use vBSEO sitemap which already excludes the archive, am I suppose to exclude it the robots.txt to? I never seen anyone here or any other place exclude it in their robots.txt

I want simple detailed instructions,
EXAMPLE:

1. Do this
2. Do that
3. And that

pepito69,
I never seen such a big robots.txt as you have, is all that really necessary?
In realty there are only 3 bots that are considered Bad Bots* and it wouldn't it be easier to put them in your .htaccess file, no?
PS. Your flags are not showing

*Baiduspider
*Mozilla/5(.*)Twiceler
*Yeti

1. It wasn't same - otherwise Google wouldn't show you link to member page because Google doesn't visits pages blocked by robots.php and you got in warning email this link:

http://YOUR_PAGE/member.php?u=63920&language=ro

Now it is OK. It wasn't some time ago and Google was there. You can be calm now about this part
2. If it is not safe for robots and shows to robots blocking communicate you should seriously consider removing this mod. Otherwise Google will start indexing pages which he see - so pages with blocking communicate. Also in this case it is main cause of your issue - all pages looks for Google same - just blocking communicate.
As I understand you checked in webmaster tools how Google see your forum and you saw that Google see communicate that he should register - yes?
3. As I wrote in the beginning your links to sub-forums in archive aren't working at all. Only to main forum and to threads. This makes thousands of wrong links on your archive pages. For example:

http://YOUR_FORUM/archive/index.php/t-25194.html?language=pl

This is link to tread archive and it is working but in it content there are links on the top of page to sub forums where the thread is included. Example:

http://YOUR_FORUM/archive/index.php/f-182.html

And all those links are not working - only to main forum. All archive sub-forums links are not working. This makes you thousands of not working links on thousands of archive threads pages.
YOU STILL DIDN'T TOLD US WHICH VBULLETIN VERSION ARE YOU USING...
At this moment I can bet that it is not supported vBulletin 3.6 - please do not worry we will not stop supporting you. We need to know it to help you better, because in 3.6 it is known vBulletin bug. If it is other version it is possible that it is vBET bug. It is critical to discover what is the cause to help you.
4. I'm not writing about 90% of your posts. I was writing about posts listed in Google mail. Others are out of scope of this discussion. Google listed you only threads with one sentence as I wrote you (at least those which I was able do see before your mod told me to register). Do you understand now? It is not the issue with whole your forum. As I already wrote - I would not worry about those - people will write answers there and those will not be similar anymore - just like 90% of your posts which looks completely different.

Please note that pepito69 can have different configuration that you. In your case ignoring language parameter is not advised, since you are not using SEO links and this is only way to recognize that URLs are different. If you tell Google to ignore this attribute for Google all URLs witch differs only value of language parameter will be considered as same content - this will create chaos in indexed pages which will randomly change languages in Google index, or Google will just index your main language pages. I think so - newer checked and do not advise for your case. Please do not do that as long as you do not use SEO links.

Simple instructions at this moment:
1. If your "3 times mod" is not Google safe – disable it and contact mod author about it.
2. TELL US WHICH VERSION OF VBULLETIN ARE YOU USING.

This is 3rd time we are asking about this... How you are expecting us to help you if you do not cooperate. WE do not have magic wand which will create solutions. We need to know what is going on on your server.

About big .htaccess - I do not know what you are writing about exactly, but I bet that for you it is not necessary such big - I assume that there are rules to block vBSEO links for member pages and so on. You are not using vBSEO and you are not using SEO links - your small robots.txt is fine.

[mario06]

TELL US WHICH VERSION OF VBULLETIN ARE YOU USING.

I'm using version 3.6.12, you should know this since I paid an additional fee to get it working (make it compatible) because of the archive problem so if it's the archive thats causing this then it's (no offense) YOUR FAULT since you did it. : http://www.vbenterprisetranslator.co...tin-3-6-a.html

If your "3 times mod" is not Google safe – disable it and contact mod author about it.

Author said it doesn't harm google.

1. It wasn't same - otherwise Google wouldn't show you link to member page because Google doesn't visits pages blocked by robots.php and you got in warning email this link:

1. The only thing that i added was /members/ .. everything else including memberslist.php was already there, MY FAULT, so let's close this argument.

2. If it is not safe for robots and shows to robots blocking communicate you should seriously consider removing this mod. Otherwise Google will start indexing pages which he see - so pages with blocking communicate. Also in this case it is main cause of your issue - all pages looks for Google same - just blocking communicate.

2. There is absolutely no need to be rude, I do understand how to use the robots.txt, but in your optimization guides (my Opinion) your lean more towards vBSEO setups, just keep in mind that at least 80% of vBulletin users don't use vBSEO so you should keep 2 separate threads when you talk about tips, tweaks and optimization, example; "vBSEO Optimization" and "Non vBSEO Optimization" to make it more user friendly.


3. All the archive links that you posted as broken seem to be working fine for me, here's a snapshot of your broken archive link example:

http://YOUR_FORUM/archive/index.php/t-25194.html?language=pl

4. htaccess comment wasn't for you but for the member pepito69


Listen Michael I don't want to argue every single letter that's written with you, I just want an easy 1. 2. 3. fix and nothing more.

Best Regards


[OVERTIME]
From what the google representative told me, the url structure below is being indexed 4 times but in realty for them it's considered to be 1 and should be indexed as such and that's probably why I got the email. The underneath post is just an example;


http://www.mysite.com/showthread.php?t=64661 Original
http://www.mysite.com/showthread.php?t=64661&language=pl Translated
http://www.mysite.com/showthread.php?t=64661&language=en Translated
http://www.mysite.com/showthread.php?t=64661&language=es Translated
[OVERTIME]