Welcome Guest ( Log In | Register )

 Forum Rules Joomla Scraper support
2 Pages V  < 1 2  
Reply to this topicStart new topic
> Server Load And Cron In Joomla Scraper, Cron Feeds In Scraper
smok3r
post Feb 23 2013, 05:37 PM
Post #21


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



I use joomla 2.5.9 and on php 5.2.17. Works fine. Could the PHP version be the problem with scraper?

All i was saying is the trial platinum with xajax pulled all my feeds like it should. Just trying to figure a way to get this to pull feeds using cron.

Everything kicks ass with scraper except this one kinda major problem.


This post has been edited by smok3r: Feb 23 2013, 05:44 PM
Go to the top of the page
 
+Quote Post
smok3r
post Feb 23 2013, 06:25 PM
Post #22


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



No . Updated to PHP version 5.3.20. Still no go. Only last feed
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 26 2013, 11:50 AM
Post #23


Web Design Seo
****

Group: Root Admin
Posts: 4,147
Joined: 29-April 09
From: Sofia
Member No.: 1



@smok3r: Especially for your case and similar cases we develop this:

Цитат
26.02.2013: v.1.8.9 for Joomla 2.5 and v.1.6.8 for joomla 1.5. Added import time and memory usage functions in: manual import, preview, in email notification and in cron. With this improved statistic you can check and diagnose import problems and measure performance of different feeds between different parsers.


With this update you can see where is the problem and why (i think that your problem is max_execution time. When using download of images max_execution time must be 60 seconds or more.). Send me email from email used in your order to receive last version.


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 26 2013, 09:00 PM
Post #24


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Fingers crossed. Email sent
Go to the top of the page
 
+Quote Post
smok3r
post Feb 26 2013, 10:43 PM
Post #25


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Got this message again when i emailed you....

Your message did not reach some or all of the intended recipients.

Subject: email used in your order to receive last version.
Sent: 2/26/2013 3:59 PM

The following recipient(s) cannot be reached:

'3D Уеб дизайн' on 2/26/2013 3:59 PM
503 Valid RCPT command must precede DATA
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 27 2013, 09:25 AM
Post #26


Web Design Seo
****

Group: Root Admin
Posts: 4,147
Joined: 29-April 09
From: Sofia
Member No.: 1



I send you PM. Please, read carefull this post: here are posted examples, stats and screenshots from import of one feed and with many rss feeds with and without extras switched on (like scraper and images download).

Цитат
Now the new version (Joomla Scraper 1.6.6 for Joomla 1.5 and Joomla Scraper 1.8.9 for Joomla 2.5 ) detects time for import and memory usage. With this improved statistic you can make more - accurate assessment and not allow too large or slow feeds.

As you can see from screenshots, time and memory usage depends on items number, usage of scraper, synonyms replacement and content shuffle, image download and feed and site response speed.

Cost more memory (you need to increase memory_limit of php 64 or more mb, recommended over 128):
- large feeds with many items
- usage of scraper, synonyms replacement and content shuffle
- import of many feeds at once

Cost more time (you need to increase max_execution_time of php to 120 seconds or more):
- large feeds with many items
- download of images
- slow websites (response speed). Website that is opened from server in your country is fast, websites from other continents are slower.
- scraper and synonyms replacement


Feed preview without scraper and content shuffle took around second for feed with 40 items from yahoo.



Feed import without scraper and content shuffle took around 6 seconds for feed with 40 items from yahoo.



Feed import with scraper and content shuffle and image download took around 30 seconds for feed with 10 items from iTunes.



Import time for 93 items from feeds with different configuration took around 1.30 - 2 minutes. On most hosting accounts normal php configuration is "max_execution_time = 30" - 60 seconds and "memory_limit = 16M" - 32M megabytes.



Import from crontab with scraper, content shuffle and image download from site with slow response time took around one minute for only 14 items.



Цитат
Check your php settings and made needed adjustments over custom php.ini file or just place new support ticket to your host support and ask to change these php settings!


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 27 2013, 10:50 PM
Post #27


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



1st feed.. Feed processed in 5 seconds and 887 milliseconds with RAM usage 12.32 MB 3 New content

2nd feed ... Feed processed in 7 seconds and 958 milliseconds with RAM usage 3.15 MB 3 New content

pulled 1st when only it was enabled. When both enabled it pulled the second feed and never pulled the first feed. Its only pulling last feed that's enabled still.

I'm only pulling 3 items per feed, same as when i first purchased it. So not server load. I've tried setting them at same time and separate times... still the same. Tried all 3 different parsers and still the same. Only the last feed enabled gets pulled. If i do it manually all feeds are processed just cron is doing this.

Uses its own php.ini with these settings
allow_url_fopen = On
max_execution_time = 300
memory_limit = 128M
max_input_time = 300
post_max_size = 50M
upload_max_filesize = 20M

Oh well... Don't know what else to do. Thanks for the help. Hopefully i'll stumble on the reason for this.
Go to the top of the page
 
+Quote Post
Web Design Seo
post Feb 28 2013, 07:38 AM
Post #28


Web Design Seo
****

Group: Root Admin
Posts: 4,147
Joined: 29-April 09
From: Sofia
Member No.: 1



Your case is just different - you use some custom php script on your server that parse rss feed and serve it to aggregator.

Possible thing 1: This php script use time and memory also. May be this script stop work.
Possible thing 2: If using cron and simplepie, one feed will be imported again only after min. 15 minutes! May be just simplepie think that urls like these are the same and the only one rss feed:
Код
mysite.com/script.php?url-of-some-rss-feed&result=rss
mysite.com/script.php?url-of-rss-feed-two&result=rss
mysite.com/script.php?url-of-rss-feed-three&result=rss


There is a option to use or not simplepie cache, but i don't think that simplepie work well with this cache.

The ways to solve problem:
1. I recommend you to try our custom RSS parser (you may need to configure it manual) - will work faster and for sure without cache.
2. Use in aggregator direct link to rss feed, not over your custom php script.


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Feb 28 2013, 06:53 PM
Post #29


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



I tried the simplepie cache.

I've been using in cron
php -q /relative path to host directory/public_html/mycron.php >> /dev/null

also triedwith its own php.ini
php -c /relative path to host directory/public_html/administrator/php.ini /relative path to host directory/public_html/mycron.php >> /dev/null


In the mycron.php
<?php
$a = file_get_contents('http://yoursite.com/administrator/components/com_aggregator/cron.aggregator.php');
?>

I'll look into the custom rss parser
Oh and i don't use any scripts... direct link to feed in aggregator with the cron setting above and mycron.php above

This post has been edited by smok3r: Feb 28 2013, 09:44 PM
Go to the top of the page
 
+Quote Post
smok3r
post Mar 5 2013, 03:10 PM
Post #30


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Still a no go. Still only pulls last feed

To bad scrapper can't be set with an auto update by feed, without cronjob in control panel, just by setting the seconds (like 3600) for each feed like this other grabber i found.
Or setting multiple cronjobs in the Control Panel an adding the feeds ID number to the cron command like...
5 * * * * <command>/aggegator.php 1,4,6 to run at seperate times like one i used in the past on PHP.

Oh well. I'll manually update until can find something else. Looking at another grabber that the auto update works. scrapper is much easier to use and has more features but like i said the manual updating blows lol.
Go to the top of the page
 
+Quote Post
smok3r
post Mar 29 2013, 07:55 PM
Post #31


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Sent you a PM about new update.

Maybe it will get scraper to auto update somehow. I'm still manually updating and using another aggregator thats not as good to auto
Go to the top of the page
 
+Quote Post
Web Design Seo
post Apr 1 2013, 06:16 AM
Post #32


Web Design Seo
****

Group: Root Admin
Posts: 4,147
Joined: 29-April 09
From: Sofia
Member No.: 1



This extension is updated only manual due on license generation. When you want latest version, send me email from email used in your order and we will send you latest version.

P.S. I see your email and will send you update in next hour.


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post
smok3r
post Apr 1 2013, 07:30 AM
Post #33


Newbie
*

Group: Members
Posts: 39
Joined: 11-February 13
Member No.: 1,561



Thank You. My email still gets returned when emailing you and anyone else... Its ok receiving. Working on getting it fixed
Go to the top of the page
 
+Quote Post
Web Design Seo
post Apr 23 2013, 12:56 PM
Post #34


Web Design Seo
****

Group: Root Admin
Posts: 4,147
Joined: 29-April 09
From: Sofia
Member No.: 1



Цитат(smok3r @ Mar 5 2013, 06:10 PM) *
Still a no go. Still only pulls last feed


Finally we found some bug in cron and all versions of Com_Aggregator are fixed today - scraper and platinum version also.

Bug was hard to find - cron was not work well only in some combination of functions switched on in feed and only on some php versions. But now is fixed.

Request update on our email to receive update.


Attention: if you already have problems, check your file /administrator/components/com_aggregator/helpers/cron.php for these changes.


--------------------
Правила на форума | Forum Rules | How to receive support. 3D Web Design: Уеб дизайн, Seo оптимизация, Web Site Extensions, Oscommerce Addons, Wordpress plugins and Joomla Extensions. Изработка на уеб сайтове и оптимизация на сайт за търсачки и Seo услуги.
Go to the top of the page
 
+Quote Post

2 Pages V  < 1 2
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

Collapse

> Similar Topics

  Topic Replies Topic Starter Views Last Action
No new Pinned: Topic has attachmentsJoomla Vulnerable Extensions List
list with new Joomla exploits
20 Web Design Seo 21,080 26th September 2018 - 05:07 AM
Last post by: Web Design Seo
No new Pinned: Topic has attachmentsJoomla Scraper, Grabber For Joomla
Joomla Scraper Can Grab Any Content From Any Website
81 Web Design Seo 84,175 14th September 2018 - 08:35 AM
Last post by: Web Design Seo
No new Pinned: Joomla Pagination Seo Plugin
SEO plugin for Joomla Pagination that work in all Joomla
61 Web Design Seo 60,008 13th March 2018 - 10:05 AM
Last post by: mxcpz
No New Posts Cron Job Working But With Notices
1 Kat 1,722 5th May 2017 - 05:56 AM
Last post by: Web Design Seo
No New Posts Pinned: Joomla Scraper Going Open Source
No licenses, use scraper on unlimited number of web sites
0 Web Design Seo 5,250 8th March 2017 - 07:40 AM
Last post by: Web Design Seo
No New Posts Pinned: Joomla Ден 2016
Joomla Day 2016
1 Web Design Seo 16,910 31st October 2016 - 10:11 AM
Last post by: Web Design Seo
No New Posts Joomla Post By Email To K2 Extra Fields
1 uglykidjoe 12,781 11th February 2016 - 07:45 AM
Last post by: Web Design Seo
No new Pinned: Joomla Scraper Integration With K2
better integration of Joomla Scraper and K2
8 Web Design Seo 24,035 2nd January 2016 - 09:07 AM
Last post by: b_goranov
No New Posts Pinned: Android Tv Box
Андроид ТВ бокс устройства, какво е това
0 Web Design Seo 14,532 21st December 2015 - 09:52 AM
Last post by: Web Design Seo
No New Posts Joomla Scraper Not Working
2 Jan Hädicke 7,269 11th November 2015 - 02:05 PM
Last post by: Web Design Seo


 



RSS Lo-Fi Version Time is now: 16th July 2019 - 06:26 AM
Clicky Web Analytics