Google Email Warning: Googlebot found an extremely high number of URLs on your site

[mario06]

Google Email Warning: Googlebot found an extremely high number of URLs on your site

I received this email (warning) from Googles Webmaster Tools

Please Note that I do not have vBSEO Installed.

I want a fix for this problem before google lowers my page rank or bans me altogether, my adsense revenue has already decreased by half and the Google Bots have literally attacked my site.

The is the second case of such a problem with vBET described here: http://www.vbenterprisetranslator.co...ng-google.html

I have sent the email to all vBET admins in PM as requested, I cannot publicly post the email as you should know because it goes against Googles TOS with my Local Privacy Laws as it would be a sure ban for my Google Adsense account so don't ask.

I need an immediate fix for this problem without further delay, it is 3 days that I've been talking about this in the link above.

Regards

[yahoooh]

i think the reason that you enable all 52 languages
i feel that if vbe make the languages requst as ajax by click langages and thin bring the enabled links for other languages may help on this

[vBET]

i think the reason that you enable all 52 languages
i feel that if vbe make the languages requst as ajax by click langages and thin bring the enabled links for other languages may help on this

No this is not the reason. Our and lot of other forums have included all languages and it is fine. So you can use as many languages as you can. The issue is somewhere else and at this moment we know 2 different reasons of that.

EDITED:
Now we know more reasons So please open your own thread and give us your mail from Google

[tavenger5]

i get the same warning and I only have 12 languages enabled

[vBET]

i get the same warning and I only have 12 languages enabled

Hi - please create new thread about your isue with your email. You got different lniks and we kow 2 reasons of this issue. To help we must know your case. So please open new thread about it and we will help you

[vBET]

mario06 - I'm just during checking your links. I will give you answer in short time

Please note that as we described before - our staff is available to start working on issue when new ticked arrives in form of new thread about the issue. Otherwise we are not able to manage it and we inform our customer that he needs to open his own new thread about his issue. Just like you just did and now we are able to help you

[vBET]

Hi

One of reasons (there will be others - still during investigation) of your issue was that you allowed to index such pages as member pages. Those are not relevant and most member pages are in 99% same - only differs by member username. So when it was additionally translated you made 52 times more such pages which are almost same (same pages content after translation will give same translated content). We officially advise to block irrelevant pages by robot.txt - here you will find how to do this: http://www.vbenterprisetranslator.co....html#post1178

Please note that this will make those pages disappear form Google index and this is good for your SEO since Google will see and index only relevant pages with relevant content.

Also as I see you made changes since Google generated this email, because as I see you blocked member pages for guests. This also solves this part of issue. Still if you want guest see member pages, but Google not index it - just use robots.txt for that

I see that there are also links for archive and threads - I'm going to analyze this right now

[vBET]

Hi again

I think I know what can be the other cause - as I'm looking on flags links there is no link without language= at all... Please tell what is your default forum language - for this language link should be without language= because now every page you have duplicated fo Google. Once for normal URL and second time with language=YOUR_DEFAULT_LANGUAGE.

Also please tell did you changed our template named vbenterprisetranslator_flag? We need to know is it vBET bug or generated by customer change.

The reason I think could be also (cannot check it now but we already have TODO task which will allow to check it in the future) that you just had cleaned cache and there was issue with Google connection - in that case pages generate without content (because translation are not obtained) and literally content is same for all translations. But such things if happen are temporary and if you are no blocked by Google those disappears itself - those are issues between your and Google server and vBET can do nothing with this. I do not tell that it was the reason. I think it could be. As I'm looking now on links given by Google I see no described issue at all. So it could be it.

Still - please answer questions about default language and template.

I see there one more thing – your translated archive pages for subforums are not working and give empty results. Main and post pages are OK. For archive subforum pages there is error. This also makes lot of links going to same content – empty content and this also can be reason why Google wrote you this. Please tell which version of vB are you using? Supported 3.8 or some lower – as I remember such thing happens on 3.6 because of vB bug. If it is not supported version and vB bug – I will find thread about it and I strongly recommend to write to vB team about solving this bug. Unfortunately I expect that they will not do this since it is not supported version anymore. But – first please tell which version are you using, so I will no waist more words and your time for speculations

[mario06]

The only thing that was turned off was the Italian language (I turned it on) since it is the forums native language I didn't think it mattered anyway I was good since a year ago so whats up now?

Here's an example (your site) of what I mean when I say effective indexed posts.

Before vBET (effective forum indexed stats):


After vBET (not effective forum indexed stats):


What I'm trying to say is that the 52x language translated posts are now being considered duplicate posts by Google. The email is self explanatory

In my own words when I say a bot attack I mean when you have 300 Google bots heavily crawling your website at the same time eating and tearing away on your bandwidth, ram and cpu and causing forum stalls. And they wrote it in their letter below in a more professional manner but the meaning is the same.

Official Google Email:
Googlebot encountered problems while crawling your site http://mysite.com/.
Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

Did you read what the Google support team wrote me? -> In general, this message is shown when we discover a relatively high number of URLs for your site.

You wrote everything and nothing because I don't understand your English, just give me a fix in simple steps like;
1. Do this
2. Do that
3. And this
etc.

Regards

[vBET]

The only thing that was turned off was the Italian language (I turned it on) since it is the forums native language I didn't think it mattered anyway I was good since a year ago so whats up now?

It is just advice from our team that without this flag your users are not able to turn back to original language when they click on some flag. So it is wise to keep flag for your default language

Here's an example (your site) of what I mean when I say effective indexed posts.

Before vBET (effective forum indexed stats):


After vBET (not effective forum indexed stats):


What I'm trying to say is that the 52x language translated posts are now being considered duplicate posts by Google. The email is self explanatory

This is NOT true. You have issues with your forum. You have broken archive - and most links you got from Google in email are about archive. You totally ignored our support - you didn't answer questions about version and keep telling what you think and what is not true. If you want to solve your issue please fallow our support hints. We asked you about your vB version because you have broken archive. Is it supported 3.8 or some not supported version? We know that archive is broken in 3.6 because of vBulletin bug. This gives you thousands of not working links - and you got from Google email about wrong links. Instead of listen to our support and cooperate you keep telling some ridiculous idea and you think it is important to put it in red.
What is your purpose here? You want our support to solve your issue or you are just trying to scare others? Because if you want to solve this issue please cooperate with our support.

Also about image which you gave - this is your misinterpretation. Google doesn’t care about number of posts shown by forum. And it cannot care about it. It is because Google indexes all the pages which are not blocked! On forum you have member pages, groups, bogs perhaps, albums and so on... and so on... Also forum can have lot of other plugins which creates additional pages (like arcade).
So please understand: you are wrong. Google do not consider translated content as duplication. You have issue with your own forum and we discovered areas of possible cause. So please - fallow our instructions/questions. Otherwise this can be marked as solved since you seem be not interested with our support just like to repeat what you wrongly think - we already read this - there is no any need to repeat it again. This is not true. Please focus on solving your issue if you want it to be solved. Please answer the question and keep attention on your archive.

Official Google Email:
Googlebot encountered problems while crawling your site http://mysite.com/.
Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

EXACTLY! YOU HAVE WRONG STRUCTURE! YOU HAVE WRONG CONTENT! and we wrote it to you at the very beginning (http://www.vbenterprisetranslator.co....html#post6846) - you have crashed archive where are thousands of not working links! So please start using our support which you asked for instead of trying to scare our users. What are you expecting from us when you ask for support and ignore it?

You wrote everything and nothing because I don't understand your English, just give me a fix in simple steps like;
1. Do this
2. Do that
3. And this
etc.

Regards

Here you got exact information that you should block member pages and how to do this:
http://www.vbenterprisetranslator.co....html#post6845

Here you got information that your archive is crashed, question about your version of vBulletin and what to do if it is not supported version:
http://www.vbenterprisetranslator.co....html#post6846
This is very exact instruction - please tell which version of vBulletin are you using.

And here you got information that you jut shouldn't do anything with posts which are very short (just 1 sentence) because this is issue of the content not of vBET. When content will get longer (users will get replies) those pages content will be not similar anymore:
http://www.vbenterprisetranslator.co....html#post6847

So:
1. Keep your member pages blocked in way like you made it recently (only for registered users) or by robots.txt like it is described here: http://www.vbenterprisetranslator.co....html#post1178
2. Check in Webmaster Tools does your "3 visits for Guest" mod is Google save – check how Google see your forum by giving URL to some thread. Check several times (cannot tell does Google will use same IP for each check). This way you will be really sure that the mod is save for robots.
3. You have broken archive – please tell which version of vBulletin are you using?
4. For short threads you can just give some longer answers there (different of course) or just keep it as it is hoping that people will answer those threads.

PS.
Please take my appologies for late answer. I was at delegation and Kamil was told to leave this issue to me. So once again sorry