Welcome Guest ( Log In | Register )

 Forum Rules Joomla Scraper support
2 Pages V   1 2 >  
Reply to this topicStart new topic
> Server Load And Cron In Joomla Scraper, Cron Feeds In Scraper
smok3r
post Feb 11 2013, 04:57 PM
Post #1


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



aggregator scraper is only reading the last feed. I have 3 feeds in it and only the last one will work. If I turn off the last feed it will start working with the second feed. Turn the second one off and and now only the first will work. Ever with the 1st and 3rd feed enabled only the last one will work.

I saw others have had this problem. Is there a fix? I really need to get this working.
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 11 2013, 05:06 PM
Post #2


Web Design Seo
****

Group: Root Admin
Posts: 4,161
Joined: 29-April 09
From: Sofia
Member No.: 1



There is no way, the only chance is some bug in latest version. We have your own 3 aggregator news websites that work with previous version - scraper v.1.8.6 (latest is 1.8.7 from today, will be public in next three days).

May be you have some huge feed or some extras switched on that load server too much like download of images, scraper, synonyms replacement from synonyms database.

Please, read how-to activate cron job first:

Цитат
You must configure cron on two different places:
- in every rss feed in Joomla Scraper configuration (different settings for every feed)
- in cronjob configuration in your hosting control panel (set to run every 2, every 3 or every 5 minutes)

Create file with name mycron.php. Code for this php file:

Код
<?php
$a = file_get_contents('http://yoursite.com/administrator/components/com_aggregator/cron.aggregator.php');
?>



Upload file mycron.php in your public_html. Create cronjob in your cpanel with command:
Код
/usr/local/bin/php -q /relative path to public_html/mycron.php >> /dev/null


Set cron to run in your control panel (cpanel or other) every 5 or 3 minutes (recommended). In Unix Style view this should be:

Код
*/3 * * * *



How to diagnose cron - Short Guide:

1. First check component requirements - memory limit (128 mb or more) and max execution time (60 seconds or more).
2. Than import manual feeds - one by one. Without extras like download images, synonyms replacement and scraper.
3. After this configure feeds one by one with extras that you want - like download images and scraper and test - again one by one.
4. Finally try automatic import with cron - after you see that any feed is working ok manual.

Цитат
Notice: Keep in mind that after every test of some feed, you must delete imported items. If you use joomla 2.5/1.7/3.0 you must delete items from trash also! If you use simplepie parser, you must wait min. 15 minutes! (cron cahe is set at 15 minutes, is not normal to import one feed every minute)



Short cron configuration guide:

Цитат
Recommended configuration for cron is to try to import one rss feed once on 2 hours or max. once on hour.


Configure feeds to be imported in different times, not at once! If in cpanel cron is activated cron on every 3rd minute and feed one is imported in 01:04, feed two is safe to import after min. 5 minutes, in 01:09.

If feeds are 7, i will configure it in these minutes (once on hour or once on 2 hours):
:5
:12
:19
:25
:35
:45
:55

------------End of Short cron configuration guide-----------

So, What are your memory limit and max execution time settings?


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 11 2013, 05:34 PM
Post #3


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Well there is a way.

Its a new site and pulling only 3 articles per feed. Not a huge feed lol and no extras switched on. memory limit 64m... max execition 300.

So you have no idea it seems. If thats the case then i'd like to get a refund and you can revoke the licenses on both sites i'm licensed for

QUOTE (Web Design Seo @ Feb 11 2013, 12:06 PM) *
There is no way, the only chance is some bug in latest version. We have your own 3 aggregator news websites that work with previous version - scraper v.1.8.6 (latest is 1.8.7 from today, will be public in next three days).

May be you have some huge feed or some extras switched on that load server too much like download of images, scraper, synonyms replacement from synonyms database.

What are your memory limit and max execution time settings?

Go to the top of the page
 
+Quote Post
smok3r
post Feb 11 2013, 05:38 PM
Post #4


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Oh and it does work great manually... but thats not how i want to use it.

You have any suggestions? I'll see if next version works. This kinda sucks

This post has been edited by smok3r: Feb 11 2013, 06:22 PM
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 12 2013, 07:10 AM
Post #5


Web Design Seo
****

Group: Root Admin
Posts: 4,161
Joined: 29-April 09
From: Sofia
Member No.: 1



1. Increase memory limit to 128 mb or more. This is the reason to stop work with many feeds.
2. If you use scraper version, try to use old version of simplepie - is better than new simplepie. Use new version of simplepie only if you got some php errors like deprecated.


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 12 2013, 01:17 PM
Post #6


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Increased to 128 and will change to old version of simplepie. I'll see if this works when i get home.

I noticed something else. When i choose After N Chars and introtext length is set at any number... using 250 at the moment. The intro text is fine but the full length article picks up where the intro text left off. The full text is missing the intro text part. It was fine in the first version i was sent but i noticed this in v.1.8.6
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 12 2013, 01:43 PM
Post #7


Web Design Seo
****

Group: Root Admin
Posts: 4,161
Joined: 29-April 09
From: Sofia
Member No.: 1



There is no warranty that introtext separator will be after closing of some html tag, so end html can be:
Код
<div>some contetnt, text....
<hr id="system-readmore" />
...and text....</div>

and this will broke your website look. Use option "limit chars" carefully.


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 13 2013, 01:21 AM
Post #8


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Well i added another feed and now it only pulls that one and ignores the first 3.

2 of the feeds pull with no html content... 1 pulls only title and linkback... 1 is full text. Not over taxing at all. I tried all 3 parsers and still only last feed is pulled.
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 13 2013, 09:46 AM
Post #9


Web Design Seo
****

Group: Root Admin
Posts: 4,161
Joined: 29-April 09
From: Sofia
Member No.: 1



If you want to help you, please, send on our email login data for admin account in your website.


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 14 2013, 12:29 AM
Post #10


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



I emailed you
Go to the top of the page
 
+Quote Post
smok3r
post Feb 14 2013, 10:29 PM
Post #11


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Does aggregator platinum still have xajax? Because it worked perfectly. Pulled all 4 feeds.

I was reading about platinum and it looks like it can does have a feed limit like scraper. It also has the scraper function as an add-on.

Could I trade the aggregator scraper for the platinum and scraper add-on and only license rageandwar.com site? Remove licenses for scraper from both sites? That would be the same cost. (I'm not using th3gate.com anymore)

Scraper was $29.99 and the extra domain was $10.00. The platinum and scraper add-on would be the same amount $19.99 and $19.99. It would be an even exchange.
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 15 2013, 07:24 AM
Post #12


Web Design Seo
****

Group: Root Admin
Posts: 4,161
Joined: 29-April 09
From: Sofia
Member No.: 1



Aggregator platinum is without many extras - load server in times less. There is no option to upgrade with scraper plugin. This is only the price for update to scraper version.


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 18 2013, 03:28 PM
Post #13


Web Design Seo
****

Group: Root Admin
Posts: 4,161
Joined: 29-April 09
From: Sofia
Member No.: 1



See this scheme to know how work cron in joomla scraper:



Please, send us login data again (all login data sent to us are deleted immediately after i see that username and pass not work)


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 18 2013, 06:09 PM
Post #14


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



I'll sent it this afternoon when i get off work.

So you're saying using curl and excuting cron.aggregator.php won't work.... like below?
*/3 * * * * curl -sS www.my_website.com/administrator/components/com_aggregator/cron.aggregator.php >>/dev/null

This should work the same way? Its doing the same thing just going around mycron.php which is executing cron.aggregator.php and doing it directly?

This post has been edited by smok3r: Feb 18 2013, 06:11 PM
Go to the top of the page
 
+Quote Post
smok3r
post Feb 18 2013, 09:33 PM
Post #15


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



I got this msg twice...
Your message did not reach some or all of the intended recipients.

Subject: RE: Heres login to site to look at aggregator scraper

Don't know if you got it or not. Let me know. I replied to your earlier email so it must be the correct email
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 19 2013, 04:34 PM
Post #16


Web Design Seo
****

Group: Root Admin
Posts: 4,161
Joined: 29-April 09
From: Sofia
Member No.: 1



1. You have configured to use functions that use many server resources - download images local in combination with every feed with exact cron configuration. This mean that all feeds are trying to import at once.
In the world is no server that can do this.

Time of import of different feeds must be cron setting plus min. cron time from cpanel config plus 1!

If in cpanel cron is executed every third minute (*/3 * * * *), config of feeds must be something like:

feed one:
Код
5 0-23 * * *


feed two:
Код
9 0-23 * * *


feed three:
Код
13 0-23 * * *


feed four:
Код
17 0-23 * * *


feed five:
Код
21 0-23 * * *


In this way, feed one will be executed in every hour 05 minute, feed two will be executed in every hour 09 minute and so...


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 19 2013, 08:26 PM
Post #17


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



I had them all staggered except the first 2. One of those was just pulling title and link to it... no other scrapper setting was at the same time.

I'll try what you have though with the staggered time in order of feed

This post has been edited by smok3r: Feb 19 2013, 09:09 PM
Go to the top of the page
 
+Quote Post
smok3r
post Feb 19 2013, 09:36 PM
Post #18


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Ok... set it up the way you showed and went the whole hour pulling no other feeds until the last feed.

Again only the last feed was pulled. This is BS.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 22 2013, 08:45 PM
Post #19


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Maybe put xajax as another option? the trial Platinum worked.

You know i set this up and it does everything you could want except the not auto pulling feeds through cron... only the last one. I set it up the way you said and still only the last one. Pulling the feeds manually is for the birds.

Do you have any other suggestions? or are you gonna let me hang with it the way it is? Is the next release done so I can see if it works? This is kinda frustrating not getting a response or an email.
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 23 2013, 05:04 PM
Post #20


Web Design Seo
****

Group: Root Admin
Posts: 4,161
Joined: 29-April 09
From: Sofia
Member No.: 1



xajax is not supported from php 5.3. And joomla 2.5 require php 5.3. And xajax have nothing to do with server limits and importing of many feeds at once (= needed huge server limits).


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

Collapse

> Similar Topics

  Topic Replies Topic Starter Views Last Action
No new Pinned: Topic has attachmentsJoomla Scraper, Grabber For Joomla
Joomla Scraper Can Grab Any Content From Any Website
82 Web Design Seo 87,714 9th October 2019 - 08:53 AM
Last post by: Web Design Seo
No new Pinned: Topic has attachmentsJoomla Vulnerable Extensions List
list with new Joomla exploits
20 Web Design Seo 24,959 26th September 2018 - 05:07 AM
Last post by: Web Design Seo
No new Pinned: Joomla Pagination Seo Plugin
SEO plugin for Joomla Pagination that work in all Joomla
61 Web Design Seo 64,760 13th March 2018 - 10:05 AM
Last post by: mxcpz
No New Posts Cron Job Working But With Notices
1 Kat 2,015 5th May 2017 - 05:56 AM
Last post by: Web Design Seo
No New Posts Pinned: Joomla Scraper Going Open Source
No licenses, use scraper on unlimited number of web sites
0 Web Design Seo 6,223 8th March 2017 - 07:40 AM
Last post by: Web Design Seo
No New Posts Pinned: Joomla Ден 2016
Joomla Day 2016
1 Web Design Seo 20,136 31st October 2016 - 10:11 AM
Last post by: Web Design Seo
No New Posts Joomla Post By Email To K2 Extra Fields
1 uglykidjoe 13,683 11th February 2016 - 07:45 AM
Last post by: Web Design Seo
No new Pinned: Joomla Scraper Integration With K2
better integration of Joomla Scraper and K2
8 Web Design Seo 25,027 2nd January 2016 - 09:07 AM
Last post by: b_goranov
No New Posts Pinned: Android Tv Box
Андроид ТВ бокс устройства, какво е това
0 Web Design Seo 16,181 21st December 2015 - 09:52 AM
Last post by: Web Design Seo
No New Posts Joomla Scraper Not Working
2 Jan Hädicke 8,212 11th November 2015 - 02:05 PM
Last post by: Web Design Seo


 



RSS Lo-Fi Version Time is now: 19th October 2019 - 07:38 AM
Clicky Web Analytics