Link to home
Start Free TrialLog in
Avatar of mmcw
mmcw

asked on

Reading page on other site

Hello,

Is it possible to start a script on an other site by clicking a URL and save the result to a file on my site??
Avatar of maneshr
maneshr

here is how i would try to solve the problem.

lets say there are 2 servers (A & B).

The script is on server B and the results are to be stored on server A.

using PERL i can write a script on server A that will remotely invoke a URL on server B.

this URL would then execute a script on server B, do some processing and store the results in a file on server B & send the final results to the calling script on server A.

The calling script on server A can then store the results on server A in a file and/or display it in the browser!!

if this is what you want i can write both the script (for server A and B) for you.
Avatar of mmcw

ASKER

On the server A I can not write/change or edit a script or do something.

I only can start a form with will produce a page with a lot of data. The data I want to use is currency data. This is on this page!!

I want to read this page what has been produced on the server A and write  and save it to my server B. The I can extract the data from this file using my server where I can use a perl script to do so!

Can you help me??
ok. now get me get this clear.

your server, server B, will call some URL on server A. You then want to store the results on server B for later use.

if that is right, i have a script for you.

also if that is right can you provide me the URL for server B, so that i can test the script before i post the final answer here??
Avatar of ozo
Avatar of mmcw

ASKER

ServerB:
mmcw.hypermart.net
Avatar of mmcw

ASKER

Question to OZO:

Where do I place the comment you gave???
Could you give me some more details??

you would type it at your command prompt.
to get more details, type
perldoc LWP::Simple
Avatar of mmcw

ASKER

Question to OZO:

I want to add it to a perl script.
A have no tellnet access!!
use LWP::Simple;
$rc = getstore('https://www.experts-exchange.com/questions/10337436/Reading-page-on-other-site.html);
print status_message($rc), "\n";
Avatar of mmcw

ASKER

Thank you OZO!
But what if the module LWP::simple is not installed and it is not possible to install it.
Avatar of mmcw

ASKER

maneshr:

You told me you had an answer?
But you had to test it first.

Do you still have an answer??

Please lett me hear!
yes i do,

however since you already have one with you, i dont think my solution would add any more to it.

Avatar of mmcw

ASKER

Could you tell me if your solution also uses LWP::Simple.

How to check if this module is installed on my server. On the servers I use I can not install modules.

On one of my servers I use I think I do not have installed this module!!
no it does not use LWP::Simple. my solution uses URI and HTTP modules.

this is how i check if a module is installed on my server.

type perldoc LWP from the command prompt.
if you get the documentation, then the module is installed.

Avatar of mmcw

ASKER

I want to use it into the script!!

and could you pass me your solution for the question?
here is your script...

here i am just fetching the homepage of yahoo.com and storing the output in yahoo.html in my server's tmp directory.

The same script, with slight changes, can also submit results to a search engine and get results or get daily news headlines or get whatever else you might want.

===yahoo.pl
#!/usr/local/bin/perl

require LWP;
require URI::URL;

use strict;
use CGI;

my($hdr,$server_response);
my($statement_URL)="http://www.yahoo.com";

$server_response=&browse($statement_URL,$hdr);  ##  Fire the URL

print "Content-type: text/html\n\n";

open(TMP,">/tmp/yahoo.html") || die $!;
print TMP $server_response;
close(TMP);
print $statement_URL," obtained!!<P>\n";
 
sub browse(){
  my($statement_URL,$hdr)=@_;
  my($content_type,$method);

  $content_type="application/x-www-form-urlencoded";
  $method="GET";

  my($headers)= new HTTP::Headers
    'Content-Type'   =>  $content_type,
    'MIME-Version'   =>  '1.0',
    'Date'           =>  HTTP::Date::time2str(time),
    'Accept'         =>  'text/html';

  my($ua)= new LWP::UserAgent;

  $ua->agent("Mozilla/4.7 [en] (WinNT; U)"); # Define env variable - HTTP_USER_
AGENT

  my($url)= new URI::URL($statement_URL);
  my($request)= new HTTP::Request($method, $url, $headers);

  my($response)= $ua->request($request);

  my($reply);

  if ($response->is_success){
    $reply=$response->content;
  }else{
    $reply=$response->error_as_HTML();
  }
  return $reply;
}
were you able to try out this script??
how did it work??

let me know.
Avatar of mmcw

ASKER

I will try it!!

Another question:
Could zou take another look at my question: Read one value in file

Question: Sometimes the number looks like this: 0.4125

The script workes fine.

But somethimes the number looks like this: 1,926.22

The script will not work.

I have checked the script:

It will work till:
$html_text =~ /$$base_currency\s+=\s+(\d+\.?\d+)/;

The the sesult will be "";

Can zou fix it to make it work for both format of numbers?
Avatar of mmcw

ASKER

Question to OZO and Maneshr:

The script works but I need now to modules to get it to work!!

The module LWP is the same OZO used in his example!
I do not have this module installed on my computer!!

Is there an other way???

I need it urgently!!

greetings Michel
you can go to www.cpan.org (more specifically http://www.cpan.org/authors/id/GAAS/libwww-perl-5.48.tar.gz)
download and install the module.

Libwww-perl is a collection of Perl modules which provides a simple
and consistent application programming interface (API) to the World-Wide Web.  The main focus of the library is to provide classes and functions that allow you to write WWW clients, thus libwww-perl said to be a WWW client library. The library also contain modules that are of more general use and even classes that help you implement simple HTTP servers.

PREREQUISITES

In order to install and use this package you will need Perl version
5.004 or better.  Some modules within this package depend on other
packages that are distributed separately from Perl.  It is recommend that you have the following packages installed before you install
libwww-perl:

  URI
  MIME-Base64
  HTML-Parser
  libnet
  Digest::MD5

These packages should be available on CPAN.

If you want to access sites using the https protocol, then you need to
install the Crypt::SSLeay or the IO::Socket::SSL module.  The
README.SSL file will tell you more about how libwww-perl supports SSL.


INSTALLATION

You install libwww-perl, as you would install any perl module library,
by running these commands:

   perl Makefile.PL
   make
   make test
   make install

Hope that helps.
Avatar of mmcw

ASKER

I do not have tellnet access or something to install a module myself!!
"I do not have tellnet access or something to install a module myself!!" :(

unfortunately then the above script might not be of a lot of help to you.

i have a simpler, socket based version of the script that i gave you above. this does not use the libraries and should work with your server.

==============simple_yahoo.pl
#!/usr/local/bin/perl

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";

$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/
;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;
select (STDOUT);
$| = 1;

print S "$request\n";
while (<S>) {
  print $_;
}
close (S);
Avatar of mmcw

ASKER

I tried the script but I got an 500 error: Premature end of script headers

How is the output file called and where is it situated??

Can you help me??
Oh!! my mistake. you are supposed to run the script from the command prompt.


if you want to run it from the browser pl add .....

print "Content-type: text/html\n\n";

before ......

print S "$request\n";
Avatar of mmcw

ASKER

I tried this code to start from a browser:

#!/usr/local/bin/perl

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";
            
$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;
select (STDOUT);
$| = 1;

print "Content-type: text/html\n\n";

print S "$request\n";
while (<S>) {
  print $_;
}
close (S);

without success!!
What is wrong??

And how is the output file called??
."without success!! "

what  does/does not happen?? please provide details of what you tried and what output you got.

...."And how is the output file called?? "

i am not clear about what you are trying to say here. pl. clarify further.

Thanks
Avatar of mmcw

ASKER

I tried it on a hypermart server (bsdos and serverport=80) and I got an error!

I tried it on another server (freebsd and serverport=80) it workes fine only the output is to the screen. The output has to go to a file!!

How to write it to a file and what could be wrong on the hypermart server??
ASKER CERTIFIED SOLUTION
Avatar of maneshr
maneshr

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of mmcw

ASKER

What I want is to copy the html file to my server.
For example I want to copy the page: http://www.yahoo.com 

to my server at:

/home/geenidee/cgi-bin/Shop/test.html

That was what  want and that is what you firsy script and OZO's script does using modules!

Can you help me??
Avatar of mmcw

ASKER

Thank you fot your quick answer!!

But I still can not get it to work on the other server. It worked here to print it to screen.

I tried your update to write it to a file: See the code I used:

#!/usr/local/bin/perl

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";
            
$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;

print S "$request\n";
$out_file="/tmp/yahoo.html";
open(TMP,">$out_file")||die $!;
select (TMP);
$| = 1;
while (<S>) {
  print TMP $_;
}
close (S);
close(TMP);

print "Content-type: text/html\n\n";
print $out_file," created<P>\n";

But I get a error 500!
In the log file I get the message:
Premature end of script headers: /home/geenidee/cgi-bin/text.pl

What is wrong????
the reason could be because the user under which the CGI  process runs does not have write permission to the /tmp directory or the /tmp directory does not exist.

to confirm that your self move ...

print "Content-type: text/html\n\n"

before....

open(TMP,">$out_file")||die $!;

& see if you can get any message on the browser.

let me know of the same.
Avatar of mmcw

ASKER

I only get the error 500 message.
The log file does only say what I said before!!

Printing to screen workes fine!
Avatar of mmcw

ASKER

When I print to screen the first ines of the page are:

HTTP/1.0 200 OK
Content-Length: 12125
Content-Type: text/html

Can that be the problem when I try to write to a file??
Avatar of mmcw

ASKER

I tried this code and it worked.

#!/bin/perl5.005_03

$AF_INET = 2;
$sockaddr = 'S n a4 x8';
$port = 80;
$them = 'www.yahoo.com';
$function = 'GET';              #Type of HTTP
$versionInfo = "HTTP/1.0\n\n";

$request = "$function / $versionInfo";

$SOCK_STREAM = 2;
$SOCK_STREAM = 1;

($name, $aliases, $proto) = getprotobyname ('tcp');
($name, $aliases, $port) = getservbyname ($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $remoteaddr) = gethostbyname ($them);

$local = pack ($sockaddr, $AF_INET, 0, $localaddr);
$remote = pack ($sockaddr, $AF_INET, $port, $remoteaddr);

if (socket (S, $AF_INET, $SOCK_STREAM, $proto)) {
        print "socket ok\n" if $debug;
} else {
        die "socket: $!\n";
}


if (bind (S, $local)) {
        print "bind ok\n" if $debug;
} else {
        die "bind: $!\n";
}

if (connect (S, $remote)) {
        print "connect ok\n" if $debug;
} else {
        die "connect: $!\n";
}

select (S);
$| = 1;
select (STDOUT);
$| = 1;

print "Content-type: text/html\n\n";

$out_file = "/v/w/w/c/wwckaars/docs/news/data/yahoo.html";
open(TMP,">$out_file")||die $!;
print S "$request\n";
while (<S>) {
      print TMP $_;
}
close (S);
close (TMP);

print "$out_file created\n";

Can I use this code without any problems you think??

When I look into the writen file I see the wanted html file, but their are three lines above it!

HTTP/1.0 200 OK
Content-Length: 11945
Content-Type: text/html

Why are these lines writen to the file??
Avatar of mmcw

ASKER

Is it also possible to check if for this:

i think its because most of these free hosting servers dont allow a socket connection to be opened from their servers to external sites.

So I get no error when try to use the code on a server who will not allow you.
Avatar of mmcw

ASKER

Another little question:

die "socket ok\n" if $debug;

Where do I find this $debug??
and what will it do??

$debug is a variable that is to be declared at the start of the program. (perhaps line 2 or 3).

$debug can take two values 0 or 1.

when $debug is 0 it means debugging is turned off.

when $debug is 1 debugging is on and all those special debugging statements (like die "socket ok\n" if $debug; ) will be executed.

its more a debugging method for the programmer/developer.

Hope that helps.
Avatar of mmcw

ASKER

Thank you a lot!!!
most welcome :-)

hope you have the solution working on all your servers!!

Wish you all the best.

Rgds