Incorrect example of Net::HTTP::NB usage [rt.cpan.org #107770]

Migrated from rt.cpan.org#107770 (status was 'new')

Requestors:

• oleg@cpan.org

From oleg@cpan.org on 2015-10-14 10:15:16:

In SYNOPSIS section we can see an example of how you can send request and read response with Net::HTTP::NB. But this example has potential problems. Let's see this test server:

use strict;
use IO::Socket;

my $serv = IO::Socket::INET->new(Listen => 10, LocalPort => 8080)
	or die $@;

my $body = 'This is the body';

my $header = join(
	"\r\n",
	"HTTP/1.1 200 OK",
	"Server: nginx/1.0.4",
	"Date: Thu, 06 Oct 2011 16:14:01 GMT",
	"Content-Type: text/html",
	"Content-Length: ".length($body),
	"Connection: keep-alive",
	"Vary: Accept-Encoding",
	"X-Powered-By: PHP/5.3.6",
	"\r\n"
);

while (warn("waiting for next request...\n") and my $client = $serv->accept()) {
	my $req;
	while ($req !~ /\r\n\r\n$/) {
		$client->sysread($req, 1024, length $req) or die $!;
	}
	
	$client->syswrite($header.$body);
	<$client>; # keep-alive ;)
}
__END__

And this client (a little reworked example from SYNOPSIS):

use strict;
use Net::HTTP::NB;

my $s = Net::HTTP::NB->new(Host => "localhost:8080") || die $@;
$s->write_request(GET => "/", 'User-Agent' => "Mozilla/5.0");

use IO::Select;
my $sel = IO::Select->new($s);
 
READ_HEADER: {
   die "Header timeout" unless $sel->can_read(10);
   my($code, $mess, %h) = $s->read_response_headers;
   redo READ_HEADER unless $code;
}

while (1) {
   die "Body timeout" unless $sel->can_read(10);
   my $buf;
   my $n = $s->read_entity_body($buf, 1024);
   last unless $n;
   print $buf;
}
__END__

And output of this client will be "Body timeout" error, instead of expected "This is the body" body content.
The problem is that Net::HTTP::Methods internally uses a buffer when reading data from the server. Here read_response_headers() call readed both headers and data, returned headers for us and stored body in the buffer. So, body now in the buffer instead of a socket and socket will not be available for read anymore (until server will close connection, but our uses keep-alive, so will not do it). This is why our can_read(10) call timed out after 10 seconds.

And this is how this example may looks like to work properly:

use strict;
use Net::HTTP::NB;
use Errno qw/EAGAIN EWOULDBLOCK/;

my $s = Net::HTTP::NB->new(Host => "localhost:8080", KeepAlive => 1) || die $@;
$s->write_request(GET => "/", 'User-Agent' => "Mozilla/5.0");

use IO::Select;
my $sel = IO::Select->new($s);
 
READ_HEADER: {
   die "Header timeout" unless $sel->can_read(10);
   my($code, $mess, %h) = $s->read_response_headers;
   redo READ_HEADER unless $code;
}

# Net::HTTP::NB uses internal buffer, so we should check it before
# socket check by calling read_entity_body()
# make socket non-blocking, so read_entity_body() will not block
$s->blocking(0);

while (1) {
   my $buf;
   my $n;
   # try to read until error or all data received
   while (1) {
		my $tmp_buf;
		$n = $s->read_entity_body($tmp_buf, 1024);
		if ($n == -1 || (!defined($n) && ($! == EWOULDBLOCK || $! == EAGAIN))) {
			last; # no data available this time
		}
		elsif ($n) {
			$buf .= $tmp_buf; # data received
		}
		elsif (defined $n) {
			last; # $n == 0, all readed
		}
		else {
			die "Read error occured: ", $!; # $n == undef
		}
   }
   
   print $buf if length $buf;
   last if defined $n && $n == 0; # all readed
   die "Body timeout" unless $sel->can_read(10); # wait for new data
}

__END__

The bad news is that most modules which uses Net::HTTP::NB doing it wrong, as showed in the example from the documentation.

Start with the Net::HTTP::NB SYNOPSIS example described in the issue and reproduce it against the provided keep-alive test server. Review how read_response_headers and read_entity_body interact with the internal buffer, then update the documentation example so buffered response data is handled correctly. Confirm the revised example receives the body without timing out.