Server Load And Cron In Joomla Scraper, Cron Feeds In Scraper Options

[smok3r]

aggregator scraper is only reading the last feed. I have 3 feeds in it and only the last one will work. If I turn off the last feed it will start working with the second feed. Turn the second one off and and now only the first will work. Ever with the 1st and 3rd feed enabled only the last one will work.

I saw others have had this problem. Is there a fix? I really need to get this working.

[Web Design Seo]

There is no way, the only chance is some bug in latest version. We have your own 3 aggregator news websites that work with previous version - scraper v.1.8.6 (latest is 1.8.7 from today, will be public in next three days).

May be you have some huge feed or some extras switched on that load server too much like download of images, scraper, synonyms replacement from synonyms database.

Please, read how-to activate cron job first:

You must configure cron on two different places:
- in every rss feed in Joomla Scraper configuration (different settings for every feed)
- in cronjob configuration in your hosting control panel (set to run every 2, every 3 or every 5 minutes)

Create file with name mycron.php. Code for this php file:

Код

<?php
$a = file_get_contents('http://yoursite.com/administrator/components/com_aggregator/cron.aggregator.php');
?>

Upload file mycron.php in your public_html. Create cronjob in your cpanel with command:

Код

/usr/local/bin/php -q /relative path to public_html/mycron.php >> /dev/null

Set cron to run in your control panel (cpanel or other) every 5 or 3 minutes (recommended). In Unix Style view this should be:

Код

*/3 * * * *

How to diagnose cron - Short Guide:

1. First check component requirements - memory limit (128 mb or more) and max execution time (60 seconds or more).
2. Than import manual feeds - one by one. Without extras like download images, synonyms replacement and scraper.
3. After this configure feeds one by one with extras that you want - like download images and scraper and test - again one by one.
4. Finally try automatic import with cron - after you see that any feed is working ok manual.

Notice: Keep in mind that after every test of some feed, you must delete imported items. If you use joomla 2.5/1.7/3.0 you must delete items from trash also! If you use simplepie parser, you must wait min. 15 minutes! (cron cahe is set at 15 minutes, is not normal to import one feed every minute)

Short cron configuration guide:

Recommended configuration for cron is to try to import one rss feed once on 2 hours or max. once on hour.

Configure feeds to be imported in different times, not at once! If in cpanel cron is activated cron on every 3rd minute and feed one is imported in 01:04, feed two is safe to import after min. 5 minutes, in 01:09.

If feeds are 7, i will configure it in these minutes (once on hour or once on 2 hours):
:5
:12
:19
:25
:35
:45
:55

------------End of Short cron configuration guide-----------

So, What are your memory limit and max execution time settings?

[smok3r]

Well there is a way.

Its a new site and pulling only 3 articles per feed. Not a huge feed lol and no extras switched on. memory limit 64m... max execition 300.

So you have no idea it seems. If thats the case then i'd like to get a refund and you can revoke the licenses on both sites i'm licensed for

There is no way, the only chance is some bug in latest version. We have your own 3 aggregator news websites that work with previous version - scraper v.1.8.6 (latest is 1.8.7 from today, will be public in next three days).

May be you have some huge feed or some extras switched on that load server too much like download of images, scraper, synonyms replacement from synonyms database.

What are your memory limit and max execution time settings?

[smok3r]

Oh and it does work great manually... but thats not how i want to use it.

You have any suggestions? I'll see if next version works. This kinda sucks

This post has been edited by smok3r: Feb 11 2013, 06:22 PM

[Web Design Seo]

1. Increase memory limit to 128 mb or more. This is the reason to stop work with many feeds.
2. If you use scraper version, try to use old version of simplepie - is better than new simplepie. Use new version of simplepie only if you got some php errors like deprecated.

[smok3r]

Increased to 128 and will change to old version of simplepie. I'll see if this works when i get home.

I noticed something else. When i choose After N Chars and introtext length is set at any number... using 250 at the moment. The intro text is fine but the full length article picks up where the intro text left off. The full text is missing the intro text part. It was fine in the first version i was sent but i noticed this in v.1.8.6

[Web Design Seo]

There is no warranty that introtext separator will be after closing of some html tag, so end html can be:

Код

<div>some contetnt, text....
<hr id="system-readmore" />
...and text....</div>

and this will broke your website look. Use option "limit chars" carefully.

[smok3r]

Well i added another feed and now it only pulls that one and ignores the first 3.

2 of the feeds pull with no html content... 1 pulls only title and linkback... 1 is full text. Not over taxing at all. I tried all 3 parsers and still only last feed is pulled.

[Web Design Seo]

If you want to help you, please, send on our email login data for admin account in your website.

[smok3r]

I emailed you

[smok3r]

Does aggregator platinum still have xajax? Because it worked perfectly. Pulled all 4 feeds.

I was reading about platinum and it looks like it can does have a feed limit like scraper. It also has the scraper function as an add-on.

Could I trade the aggregator scraper for the platinum and scraper add-on and only license rageandwar.com site? Remove licenses for scraper from both sites? That would be the same cost. (I'm not using th3gate.com anymore)

Scraper was $29.99 and the extra domain was $10.00. The platinum and scraper add-on would be the same amount $19.99 and $19.99. It would be an even exchange.

[Web Design Seo]

Aggregator platinum is without many extras - load server in times less. There is no option to upgrade with scraper plugin. This is only the price for update to scraper version.

[Web Design Seo]

See this scheme to know how work cron in joomla scraper:



Please, send us login data again (all login data sent to us are deleted immediately after i see that username and pass not work)

[smok3r]

I'll sent it this afternoon when i get off work.

So you're saying using curl and excuting cron.aggregator.php won't work.... like below?
*/3 * * * * curl -sS www.my_website.com/administrator/components/com_aggregator/cron.aggregator.php >>/dev/null

This should work the same way? Its doing the same thing just going around mycron.php which is executing cron.aggregator.php and doing it directly?

This post has been edited by smok3r: Feb 18 2013, 06:11 PM

[smok3r]

I got this msg twice...
Your message did not reach some or all of the intended recipients.

Subject: RE: Heres login to site to look at aggregator scraper

Don't know if you got it or not. Let me know. I replied to your earlier email so it must be the correct email

[Web Design Seo]

1. You have configured to use functions that use many server resources - download images local in combination with every feed with exact cron configuration. This mean that all feeds are trying to import at once.
In the world is no server that can do this.

Time of import of different feeds must be cron setting plus min. cron time from cpanel config plus 1!

If in cpanel cron is executed every third minute (*/3 * * * *), config of feeds must be something like:

feed one:

Код

5 0-23 * * *

feed two:

Код

9 0-23 * * *

feed three:

Код

13 0-23 * * *

feed four:

Код

17 0-23 * * *

feed five:

Код

21 0-23 * * *

In this way, feed one will be executed in every hour 05 minute, feed two will be executed in every hour 09 minute and so...

[smok3r]

I had them all staggered except the first 2. One of those was just pulling title and link to it... no other scrapper setting was at the same time.

I'll try what you have though with the staggered time in order of feed

This post has been edited by smok3r: Feb 19 2013, 09:09 PM

[smok3r]

Ok... set it up the way you showed and went the whole hour pulling no other feeds until the last feed.

Again only the last feed was pulled. This is BS.

[smok3r]

Maybe put xajax as another option? the trial Platinum worked.

You know i set this up and it does everything you could want except the not auto pulling feeds through cron... only the last one. I set it up the way you said and still only the last one. Pulling the feeds manually is for the birds.

Do you have any other suggestions? or are you gonna let me hang with it the way it is? Is the next release done so I can see if it works? This is kinda frustrating not getting a response or an email.

[Web Design Seo]

xajax is not supported from php 5.3. And joomla 2.5 require php 5.3. And xajax have nothing to do with server limits and importing of many feeds at once (= needed huge server limits).