Server Load And Cron In Joomla Scraper, Cron Feeds In Scraper Options

[smok3r]

I use joomla 2.5.9 and on php 5.2.17. Works fine. Could the PHP version be the problem with scraper?

All i was saying is the trial platinum with xajax pulled all my feeds like it should. Just trying to figure a way to get this to pull feeds using cron.

Everything kicks ass with scraper except this one kinda major problem.


This post has been edited by smok3r: Feb 23 2013, 05:44 PM

[smok3r]

No . Updated to PHP version 5.3.20. Still no go. Only last feed

[Web Design Seo]

@smok3r: Especially for your case and similar cases we develop this:

26.02.2013: v.1.8.9 for Joomla 2.5 and v.1.6.8 for joomla 1.5. Added import time and memory usage functions in: manual import, preview, in email notification and in cron. With this improved statistic you can check and diagnose import problems and measure performance of different feeds between different parsers.

With this update you can see where is the problem and why (i think that your problem is max_execution time. When using download of images max_execution time must be 60 seconds or more.). Send me email from email used in your order to receive last version.

[smok3r]

Fingers crossed. Email sent

[smok3r]

Got this message again when i emailed you....

Your message did not reach some or all of the intended recipients.

Subject: email used in your order to receive last version.
Sent: 2/26/2013 3:59 PM

The following recipient(s) cannot be reached:

'3D Уеб дизайн' on 2/26/2013 3:59 PM
503 Valid RCPT command must precede DATA

[Web Design Seo]

I send you PM. Please, read carefull this post: here are posted examples, stats and screenshots from import of one feed and with many rss feeds with and without extras switched on (like scraper and images download).

Now the new version (Joomla Scraper 1.6.6 for Joomla 1.5 and Joomla Scraper 1.8.9 for Joomla 2.5 ) detects time for import and memory usage. With this improved statistic you can make more - accurate assessment and not allow too large or slow feeds.

As you can see from screenshots, time and memory usage depends on items number, usage of scraper, synonyms replacement and content shuffle, image download and feed and site response speed.

Cost more memory (you need to increase memory_limit of php 64 or more mb, recommended over 128):
- large feeds with many items
- usage of scraper, synonyms replacement and content shuffle
- import of many feeds at once

Cost more time (you need to increase max_execution_time of php to 120 seconds or more):
- large feeds with many items
- download of images
- slow websites (response speed). Website that is opened from server in your country is fast, websites from other continents are slower.
- scraper and synonyms replacement

Feed preview without scraper and content shuffle took around second for feed with 40 items from yahoo.



Feed import without scraper and content shuffle took around 6 seconds for feed with 40 items from yahoo.



Feed import with scraper and content shuffle and image download took around 30 seconds for feed with 10 items from iTunes.



Import time for 93 items from feeds with different configuration took around 1.30 - 2 minutes. On most hosting accounts normal php configuration is "max_execution_time = 30" - 60 seconds and "memory_limit = 16M" - 32M megabytes.



Import from crontab with scraper, content shuffle and image download from site with slow response time took around one minute for only 14 items.

Check your php settings and made needed adjustments over custom php.ini file or just place new support ticket to your host support and ask to change these php settings!

[smok3r]

1st feed.. Feed processed in 5 seconds and 887 milliseconds with RAM usage 12.32 MB 3 New content

2nd feed ... Feed processed in 7 seconds and 958 milliseconds with RAM usage 3.15 MB 3 New content

pulled 1st when only it was enabled. When both enabled it pulled the second feed and never pulled the first feed. Its only pulling last feed that's enabled still.

I'm only pulling 3 items per feed, same as when i first purchased it. So not server load. I've tried setting them at same time and separate times... still the same. Tried all 3 different parsers and still the same. Only the last feed enabled gets pulled. If i do it manually all feeds are processed just cron is doing this.

Uses its own php.ini with these settings
allow_url_fopen = On
max_execution_time = 300
memory_limit = 128M
max_input_time = 300
post_max_size = 50M
upload_max_filesize = 20M

Oh well... Don't know what else to do. Thanks for the help. Hopefully i'll stumble on the reason for this.

[Web Design Seo]

Your case is just different - you use some custom php script on your server that parse rss feed and serve it to aggregator.

Possible thing 1: This php script use time and memory also. May be this script stop work.
Possible thing 2: If using cron and simplepie, one feed will be imported again only after min. 15 minutes! May be just simplepie think that urls like these are the same and the only one rss feed:

Код

mysite.com/script.php?url-of-some-rss-feed&result=rss
mysite.com/script.php?url-of-rss-feed-two&result=rss
mysite.com/script.php?url-of-rss-feed-three&result=rss

There is a option to use or not simplepie cache, but i don't think that simplepie work well with this cache.

The ways to solve problem:
1. I recommend you to try our custom RSS parser (you may need to configure it manual) - will work faster and for sure without cache.
2. Use in aggregator direct link to rss feed, not over your custom php script.

[smok3r]

I tried the simplepie cache.

I've been using in cron
php -q /relative path to host directory/public_html/mycron.php >> /dev/null

also triedwith its own php.ini
php -c /relative path to host directory/public_html/administrator/php.ini /relative path to host directory/public_html/mycron.php >> /dev/null


In the mycron.php
<?php
$a = file_get_contents('http://yoursite.com/administrator/components/com_aggregator/cron.aggregator.php');
?>

I'll look into the custom rss parser
Oh and i don't use any scripts... direct link to feed in aggregator with the cron setting above and mycron.php above

This post has been edited by smok3r: Feb 28 2013, 09:44 PM

[smok3r]

Still a no go. Still only pulls last feed

To bad scrapper can't be set with an auto update by feed, without cronjob in control panel, just by setting the seconds (like 3600) for each feed like this other grabber i found.
Or setting multiple cronjobs in the Control Panel an adding the feeds ID number to the cron command like...
5 * * * * <command>/aggegator.php 1,4,6 to run at seperate times like one i used in the past on PHP.

Oh well. I'll manually update until can find something else. Looking at another grabber that the auto update works. scrapper is much easier to use and has more features but like i said the manual updating blows lol.

[smok3r]

Sent you a PM about new update.

Maybe it will get scraper to auto update somehow. I'm still manually updating and using another aggregator thats not as good to auto

[Web Design Seo]

This extension is updated only manual due on license generation. When you want latest version, send me email from email used in your order and we will send you latest version.

P.S. I see your email and will send you update in next hour.

[smok3r]

Thank You. My email still gets returned when emailing you and anyone else... Its ok receiving. Working on getting it fixed

[Web Design Seo]

Still a no go. Still only pulls last feed

Finally we found some bug in cron and all versions of Com_Aggregator are fixed today - scraper and platinum version also.

Bug was hard to find - cron was not work well only in some combination of functions switched on in feed and only on some php versions. But now is fixed.

Request update on our email to receive update.


Attention: if you already have problems, check your file /administrator/components/com_aggregator/helpers/cron.php for these changes.