Aggregators And Large Number Of Feeds, Tips To Make Huge Site, Aggregator Platinum Problem With Cron Job With 7534 Rss Feeds Options

[Web Design Seo]

Aggregator Platinum Problem With Cron Job With 7534 Rss Feeds, tips to make huge website with aggregator and joomla.

Support question in latest week:

Hello,

Thank you for the great component com_aggregator_platinum-k2, I have some problem with cron job with large feeds (7534 rss feed), set with cron scheduler as attached scheduler_monthly.txt , the cron run without problem when I test with few feed (4-10 feeds), but will never run when enable all 7534 feed, it seems that the cron job can not find the correct schedule time to run the job, do you have any suggestion to run this large feed with cron or can you test creat a large feed and use my scheduler_monthly.txt

Thank you.

My answer:

Wow, what a huge site and what a number of feeds to import! Aggregator platinum is developed for huge sites but we dont test component with number of feeds more than 300

Suggestions:
1. Check number of feeds that are imported at once. Will be better that at one moment are imported not more than 5-10 feeds. If you
2. Set cron job in cpanel to run in every one minute.


If You still have problems, switch on error reporting at maximum and open manually in browser url of cron to see what happened. Make this more 50-100 times in day.

I recommend you to lower number of feeds, may be about 1000 feeds are nice number

[kcw]

1) the article per feed to import set to 25
2) the cpanel cron job run every 5 minutes.

Here are some notice on test server when run "administrator/components/com_aggregator/cron.aggregator.php":

Notice: Undefined offset: 8 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 177

Notice: Undefined offset: 4 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 54

Notice: Undefined offset: 5 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 57

Notice: Undefined offset: 4 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 54

Notice: Undefined offset: 5 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 57

Notice: Undefined offset: 4 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 54

Notice: Undefined offset: 5 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 57

Notice: Undefined variable: maxJobs in /home/xxxx/administrator/components/com_aggregator/controller.php on line 0

Notice: Undefined offset: 8 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 177

Notice: Undefined offset: 4 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 54

seems loop forever.

[kcw]

And here on another dedicated server when run the script www.xxxx/administrator/components/com_aggregator/cron.aggregator.php

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@xxxx and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

and error log:

[04-Jan-2011 00:05:59] PHP Warning: file_get_contents(http://www.xxxx/administrator/components/com_aggregator/cron.aggregator.php) [<a href='function.file-get-contents'>function.file-get-contents</a>]: failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error

in /home/xxxxx/public_html/mycron.php on line 2

This post has been edited by 3D Web Design: Jan 3 2011, 08:48 PM

[Web Design Seo]

1) the article per feed to import set to 25
2) the cpanel cron job run every 5 minutes.

Here are some notice on test server when run "administrator/components/com_aggregator/cron.aggregator.php":
Notice: Undefined offset: 8 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 177

I am sure that you must make all of this:

1. Set cpanel cron job to run every 1 minute. This will limit number of feeds that are imported at once if you have configure feeds in right way. Now if you set a cron in feed: 5 * * * * feed on some servers will be imported in next hour in 10 minute.
2. Limit number of feeds to import at once to number lower than 10 or try to set higher memory_limit preference in php. In example you must set memory_limit = 128M

Example code:
max_execution_time = 120 ; Maximum execution time of each php script, in seconds. You must set minimum 30 or better 60 here.
memory_limit = 128M ; Maximum amount of memory a script may consume (default 32MB)


About Notices: Notices are not an errors and php will not stop. This will shown only with maximum error reporting. Php will stop if you get a "fatal error"

[Web Design Seo]

All of this will be not a decision of problem. If your site will be very big you must go to VPS or better to custom server. By 7500 rss feeds that are imported by schedule server must be with minimum 512 mb memory or better with min. 1 GB.

I've never done joomla with more than 150 000 artcles inside. At one vps server we have joomla with aggregator platinum with 80-90000 articles. When articles become more load is high and website is very slow.

When number of content items will be higher, joomla serious load server.

I recommend You to create more sections and categories and post every feed in your own category. When items in one category are lower than 8000-9000 joomla is stable.

Other tip is to use joomla native sef or unicode slugs for seo urls. When you use sef component like sh404sef or artio joomsef load will be to 30-50% higher (only database table with urls of 100000 url addresses is 20-40 mb)

[kcw]

Is my mistake.

3k+ feeds with monthly scheduler cron working now.

My problem is cause by the date format for `last_run` datetime NOT NULL default '0000-00-00 00:00:00',
when create large feed list with spreadsheet the date format may change to 0000/00/00 00-00, the cron.aggregator.php will run all feeds at once if last_run date format wrong, this will kill the server.

Thanks for the support.

[Web Design Seo]

This will be very big website
Would be glad to have more such successful big sites like yours, aggregator is created just for the creation of such sites.

I'll be happy if you post link to your site here.