Site slow after cache clearing

[tavenger5]

I've gone through and implemented all the possible optimization tricks I can find. This includes nginx as a proxy to apache, vbOptimize with memcached, and all the regular vbulletin optimization procedures.

I'm working with two dual quad core processor servers with 12 and *** of ram, and 15k sas drives in raid. So, in other words, the servers have enough power to process everything.

The main site starts to slow right after the vBET cache is cleared every 15 days. (the database gets to just over *** after this 15 day period) > 500k pages a day are being crawled by search engines.

Is there anything I can do to tweak apache to handles these requests better? These are my current apache settings:
from httpd-mpm.conf
# prefork MPM

StartServers 20
MinSpareServers 20
MaxSpareServers 25
MaxClients 180
MaxRequestsPerChild 1000

From httpd-default.conf:

Timeout 150
KeepAlive On
MaxKeepAliveRequests 80
KeepAliveTimeout 3
UseCanonicalName Off

[vBET]

Let me guess - you have vBSEO and lot of links on main page - I'm right?

The trick is - if you do not really have to, then do not use last clearing strategy. I know that there is if - did you checked other clearing strategies? Other will not clear whole cache and will take more resources to clear from other side.

Next vBET 3.x release can help you - we will add new advanced performance parameters for really big pages. We also discovered bottleneck with links translation. At this moment we have implemented solution for vB Friendly URLs in vBET4.x (not released yet) and we will try to adopt it also for vBSEO. If we succeed we will move it also to vBET 3.x The issue is that vBSEO asks for links one by one and this produces dozens of Google requests. aS I wrote we already implemented solution for vB Frinedly URLs - we made delayed translation. Problem with vBSEO is that it works outside vB, after translation happens and also don't tell does needs url to check correctness of actual one
or to put it in output.
Lot of details - shortly we know one bottleneck which happens only when cache is not filled and we are already working on this issue.

So at this moment I can just advise you to play with clearing strategies and other clearing parameters. For other strategies:
- If clearing of one cache table is not killing your server, then set bigger 'Cache clearing timelap' - your server will take a breath between clearings
- Analise your forum traffic and check when it is less - change clearing execution to this time
- Set lower cache TTL - smaller tables will be cleared so clearing itself will take less resources. Other side - server will have to ask Google more often for translations.
- EXPERIMENTAL: set 'Quick local deletion with optimize tables' open /includes/vbenterprisetranslator_functions.php and comment there 3 lines of code with 'OPTIMIZE LOCAL TABLE'. This will make really fast deletion without indexes upgrade. NOTE: indexes will grow, so you will have to execute the query manually - i.e. check it once per week. If it will work for you we will implement new strategy, where indexes will be reorganized not every day.

[tavenger5]

Yes on vbSEO.

I'm using normal deletion at the moment and it doesn't seem to take too long to get things cleared. With quick local deletion are the indexes left in tact, and the normal deletion indexes are cleared? Will having old indexes have any benefit if they are not optimized?

Things only seem to slow down when there's a lot of traffic on the site and the cache is being rebuilt. I'm sure this is because apache processes aren't being closed as fast as they normally would (since data is being requested from google).

It's good to hear that the next version will improve on speed again. I was just making sure there wasn't anything else I could do with tweaking apache.

[vBET]

If you are using normal clearing then forgot about my hints. I thought that you are using last strategy and remove whole cache. Sorry - misunderstanding Just leave it as it is.

In such way I can advise to set bigger Cache TTL. Less data will be removed each time, so less data will be to recover.
As I wrote we already found one bottleneck with vBSEO + empty cache and we are working on it

What you also can do is make sure that your server is not holding outgoing requests. We discovered that some servers behave like this if many outgoing requests are going to same server. Because 100 requests can take 1000 x more time than 1 request (theoretically should take 100 x more time). It can be some firewall, server security issue. Of course it can be that Google puts some little 'punishment' in such case. So if you can find something in this area - it can help. If not please wait for improvements