The World-Wide Web library for Perl, or LWP, is a vital tool in a Perl programmer's arsenal. We can use it to access website APIs like Twitter, download webpages, connect to FTP servers, and a myriad of other web related functions. The module, LWP::Parallel, is a way to do multiple of these requests at once, and thus saving valuable user time by consolidating wait times for requests. Bryan Young discussed these advantages briefly and how to accomplish this here. Although there are not many circumstances where LWP can not handle the job effectively and efficiently, LWP::Parallel, unfortunately, is no longer maintained. Luckily, we have an alternative to LWP::Parallel with WWW::Curl::Multi.

Like LWP, it is easy to grab a webpage with WWW::Curl.

use WWW::Curl::Easy;

my $curl = WWW::Curl::Easy->new;

$curl->setopt(CURLOPT_HEADER, 1);

$curl->setopt(CURLOPT_URL, 'http://perlpronews.com');

my $response;

$curl->setopt(CURLOPT_WRITEDATA, \$response);

my $retcode = $curl->perform;

The above code stores the returned data into >$response. With WWW::Curl::Multi, we initialize multiple Curl handles in the same fashion as above. Then, we use the >add_handle() method to gather these handles. When executing >perform on a Curl::Multi object, it will take all the Curl handles stored in that same object and send the requests in parallel. By utilizing a hash or array for our responses, we can easily access those responses once the parallel requests are complete. For a more complete example of how to utilize Curl::Multi, check out the example on it's CPAN page.