Welcome Guest ( Log In | Register )

 Forum Rules Aggregators support
 
Reply to this topicStart new topic
> Aggregators And Large Number Of Feeds, Tips To Make Huge Site, Aggregator Platinum Problem With Cron Job With 7534 Rss Feeds
Web Design Seo
post Jan 3 2011, 09:44 AM
Post #1


Web Design Seo
****

Group: Root Admin
Posts: 4,160
Joined: 29-April 09
From: Sofia
Member No.: 1



Aggregator Platinum Problem With Cron Job With 7534 Rss Feeds, tips to make huge website with aggregator and joomla.

Support question in latest week:

Цитат
Hello,

Thank you for the great component com_aggregator_platinum-k2, I have some problem with cron job with large feeds (7534 rss feed), set with cron scheduler as attached scheduler_monthly.txt , the cron run without problem when I test with few feed (4-10 feeds), but will never run when enable all 7534 feed, it seems that the cron job can not find the correct schedule time to run the job, do you have any suggestion to run this large feed with cron or can you test creat a large feed and use my scheduler_monthly.txt

Thank you.



My answer:

Wow, what a huge site and what a number of feeds to import! Aggregator platinum is developed for huge sites but we dont test component with number of feeds more than 300 smile.gif

Suggestions:
1. Check number of feeds that are imported at once. Will be better that at one moment are imported not more than 5-10 feeds. If you
2. Set cron job in cpanel to run in every one minute.


If You still have problems, switch on error reporting at maximum and open manually in browser url of cron to see what happened. Make this more 50-100 times in day.

I recommend you to lower number of feeds, may be about 1000 feeds are nice number smile.gif


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
kcw
post Jan 3 2011, 05:23 PM
Post #2


Newbie
*

Group: Members
Posts: 3
Joined: 3-January 11
Member No.: 469



1) the article per feed to import set to 25
2) the cpanel cron job run every 5 minutes.

Here are some notice on test server when run "administrator/components/com_aggregator/cron.aggregator.php":

Notice: Undefined offset: 8 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 177

Notice: Undefined offset: 4 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 54

Notice: Undefined offset: 5 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 57

Notice: Undefined offset: 4 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 54

Notice: Undefined offset: 5 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 57

Notice: Undefined offset: 4 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 54

Notice: Undefined offset: 5 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 57

Notice: Undefined variable: maxJobs in /home/xxxx/administrator/components/com_aggregator/controller.php on line 0

Notice: Undefined offset: 8 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 177

Notice: Undefined offset: 4 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 54

seems loop forever.
Go to the top of the page
 
+Quote Post
kcw
post Jan 3 2011, 05:26 PM
Post #3


Newbie
*

Group: Members
Posts: 3
Joined: 3-January 11
Member No.: 469



And here on another dedicated server when run the script www.xxxx/administrator/components/com_aggregator/cron.aggregator.php

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@xxxx and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

and error log:

[04-Jan-2011 00:05:59] PHP Warning: file_get_contents(http://www.xxxx/administrator/components/com_aggregator/cron.aggregator.php) [<a href='function.file-get-contents'>function.file-get-contents</a>]: failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error

in /home/xxxxx/public_html/mycron.php on line 2

This post has been edited by 3D Web Design: Jan 3 2011, 08:48 PM
Go to the top of the page
 
+Quote Post
Web Design Seo
post Jan 3 2011, 09:01 PM
Post #4


Web Design Seo
****

Group: Root Admin
Posts: 4,160
Joined: 29-April 09
From: Sofia
Member No.: 1



Цитат(kcw @ Jan 3 2011, 07:23 PM) *
1) the article per feed to import set to 25
2) the cpanel cron job run every 5 minutes.

Here are some notice on test server when run "administrator/components/com_aggregator/cron.aggregator.php":
Notice: Undefined offset: 8 in /home/xxxx/administrator/components/com_aggregator/helpers/cron.php on line 177


I am sure that you must make all of this:

1. Set cpanel cron job to run every 1 minute. This will limit number of feeds that are imported at once if you have configure feeds in right way. Now if you set a cron in feed: 5 * * * * feed on some servers will be imported in next hour in 10 minute.
2. Limit number of feeds to import at once to number lower than 10 or try to set higher memory_limit preference in php. In example you must set memory_limit = 128M

Example code:
max_execution_time = 120 ; Maximum execution time of each php script, in seconds. You must set minimum 30 or better 60 here.
memory_limit = 128M ; Maximum amount of memory a script may consume (default 32MB)


About Notices: Notices are not an errors and php will not stop. This will shown only with maximum error reporting. Php will stop if you get a "fatal error" smile.gif


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
Web Design Seo
post Jan 3 2011, 10:14 PM
Post #5


Web Design Seo
****

Group: Root Admin
Posts: 4,160
Joined: 29-April 09
From: Sofia
Member No.: 1



All of this will be not a decision of problem. If your site will be very big you must go to VPS or better to custom server. By 7500 rss feeds that are imported by schedule server must be with minimum 512 mb memory or better with min. 1 GB.

I've never done joomla with more than 150 000 artcles inside. At one vps server we have joomla with aggregator platinum with 80-90000 articles. When articles become more load is high and website is very slow.

When number of content items will be higher, joomla serious load server.

I recommend You to create more sections and categories and post every feed in your own category. When items in one category are lower than 8000-9000 joomla is stable.

Other tip is to use joomla native sef or unicode slugs for seo urls. When you use sef component like sh404sef or artio joomsef load will be to 30-50% higher (only database table with urls of 100000 url addresses is 20-40 mb)


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
kcw
post Jan 5 2011, 08:03 AM
Post #6


Newbie
*

Group: Members
Posts: 3
Joined: 3-January 11
Member No.: 469



Is my mistake.

3k+ feeds with monthly scheduler cron working now. wink.gif

My problem is cause by the date format for `last_run` datetime NOT NULL default '0000-00-00 00:00:00',
when create large feed list with spreadsheet the date format may change to 0000/00/00 00-00, the cron.aggregator.php will run all feeds at once if last_run date format wrong, this will kill the server.

Thanks for the support.
Go to the top of the page
 
+Quote Post
Web Design Seo
post Jan 5 2011, 08:12 AM
Post #7


Web Design Seo
****

Group: Root Admin
Posts: 4,160
Joined: 29-April 09
From: Sofia
Member No.: 1



This will be very big website smile.gif
Would be glad to have more such successful big sites like yours, aggregator is created just for the creation of such sites.

I'll be happy if you post link to your site here.


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

Collapse

> Similar Topics

  Topic Replies Topic Starter Views Last Action
No new Feeds Only Posting Once A Day
7 Kat 2,908 30th June 2017 - 10:17 AM
Last post by: Web Design Seo
No New Posts Pinned: Android Tv Box
Андроид ТВ бокс устройства, какво е това
0 Web Design Seo 16,073 21st December 2015 - 09:52 AM
Last post by: Web Design Seo
No New Posts Aggregators, Scrapers And Grabbers Demo Site
newest versions now are in demo site for testing
0 Web Design Seo 6,038 31st May 2011 - 09:14 AM
Last post by: Web Design Seo
No new Aggregators And Seos, За агрегаторите и Seo-тата
въпроси и разсъждения
7 Web Design Seo 10,793 18th April 2011 - 12:23 PM
Last post by: Web Design Seo


 



RSS Lo-Fi Version Time is now: 13th October 2019 - 08:04 PM
Clicky Web Analytics