October 13th, 2009
Spelling of polite robots
If you want your program based on LWP will be convinced, that, pays attention to files robots.txt and does not do{make} too many searches for the short period of time you can use LWP:: RobotUA instead of LWP:: UserAgent.
If you want your program based on LWP will be convinced, that, pays attention to files robots.txt and does not do{make} too many searches for the short period of time you can use LWP:: RobotUA instead of LWP:: UserAgent.
LWP:: RobotUA is almost LWP:: UserAgent, and you can use it{him} also:
use LWP:: RobotUA;
my $browser = LWP:: RobotUA-> new (
‘ YourSuperBot/1.34 ‘,’ you@yoursite.com ‘);
* Your bot’s name and your email address
my $response = $browser-> get ($url);
But HTTP::RobotUA adds the following opportunities:
*
If robots.txt on the server to which refers $url, forbids to you access to $url then the object $browser (take into account, that he belongs to class LWP:: RobotUA) will not request it{him}, and we shall receive in the answer ($response) a mistake 403 containing a line ” Forbidden by robots.txt “. So, if you have the following line:
die ” $url – “, $response-> status_line, “nAborted”
unless $response-> is_success;
Then the program should will come to the end with the message:
http://whatever.site.int/pith/x.html – 403 Forbidden
by robots.txt
Aborted at whateverprogram.pl line 1234
*
If $browser will see, that communicated with this server not so long ago, then he sdleaet a pause (it is similar sleep) for prevention of realization of a plenty of searches for short term. What delay will be? In general, by default, it – 1 minute, but you can supervise it by change of attribute $browser-> delay (minutes).
For example:
$browser-> delay (7/60);
It means, that the browser will make a pause when it will be necessary while since time of the previous search will not pass 7 seconds.
For the greater information read the full documentation on LWP:: RobotUA.
Use of proxies
In some cases you want or it is necessary for you to use a proxy for access to the certain sites or for use of the certain report. Most often such necessity arises, when your LWP-program is started by the machine which is ” for firewallom “.
That a browser ispol`zovl a proxy which is determined in variables of an environment (HTTP_PROXY), call env_proxy before any searches. In particular:
use LWP:: UserAgent;
my $browser = LWP:: UserAgent-> new;
*? before the first search:
$browser-> env_proxy;
For the greater information on parameters of a proxy read the documentation on LWP:: UserAgent, in particular pay attention to methods proxy, env_proxy and no_proxy.
HTTP Authentication (identification)
Many sites limit access to the pages using ” HTTP Authentication “. It not simply the form where you should enter the password for access to the information, is the special mechanism, when HTTP serverposylaet to a browser the message which says: ” That document is part of a protected ‘ realm ‘, and you can access it only if you re-request it and add some special authorization headers to your request ” (” This document is a part protected ‘ areas ‘ and you can get access to it{him} if you once again will query, having added some specific headings to your search “).
For example, managers of a site Unicode.org limit access for programs of gathering emailov to their archives of electronic dispatches, protecting them with help HTTP Authentication, there is a general{common} login and the password for access (on http: // www.unicode.org/mail-arch/) – a login – “unicode-ml” and the password – “unicode”.
For example, we shall consider this URL which is a part of the protected area of the Website:
http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html
You ate will try to load this page with a browser receive the instruction: ” Enter username and password for ‘ Unicode-MailList-Archives’ at server ‘www.unicode.org’ “, or in a graphic browser something like it:
Screenshot of site with Basic Auth required
In LWP if you start the following:
use LWP 5.64;
my $browser = LWP:: UserAgent-> new;
my $url =
‘ http: // www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html ‘;
my $response = $browser-> get ($url);
die ” Error: “, $response-> header (‘ WWW-Authenticate ‘) ||
‘ Error accessing ‘,
* (‘ WWW-Authenticate ‘ is the realm-name)
“n”, $response-> status_line, ” n at $urln Aborting ”
unless $response-> is_success;
Then receive a mistake:
Error: Basic realm = “Unicode-MailList-Archives”
401 Authorization Required
at http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html
Aborting at auth1.pl line 9. [or wherever]
Because $browser does not know a login and the password for area (“Unicode-MailList-Archives”) on a host (“www.unicode.org”). The simplest method to give to learn{find out} to a browser a login and the password – to use a method credentials. Syntax the following:
$browser-> credentials (
‘ servername:portnumber ‘,
‘ realm-name ‘,
‘ username ‘ => ‘ password ‘
);
In most cases the port number{room} 80 – is TCP/IP port by default for HTTP; and you can use a method credentials up to any searches. For example:
$browser-> credentials (
‘ reports.mybazouki.com:80 ‘,
‘ web_server_usage_reports ‘,
‘ plinky ‘ => ‘ banjo123 ‘
);
So, if we shall add the following right after lines $browser = LWP:: UserAgent-> new;:
$browser-> credentials (* add this to our $browser ‘ s ” key ring ”
‘ www.unicode.org:80 ‘,
‘ Unicode-MailList-Archives ‘,
‘ unicode-ml ‘ => ‘ unicode ‘
);
Also we shall start, the search will pass.
