Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Timeout errors using Net::HTTP on Windows

Reply
Thread Tools

Timeout errors using Net::HTTP on Windows

 
 
Toby DiPasquale
Guest
Posts: n/a
 
      04-17-2007
Hi all,

I'm having a problem here running some file upload code on Windows. I've
installed Ruby 1.8.6 from the one-click installer on Windows 2000 Server
in VMware and on real hardware with Windows XP. The client, when running
on Windows, always times/errors out no matter what. Here's the error:

c:/ruby/lib/ruby/1.8/net/http.rb:2094:in `error!': 408 "Request Timeout
" (Net::HTTPServerException)
from c:/ruby/lib/ruby/1.8/net/http.rb:2103:in `value'
from put_client.rb:25:in `upload_file'
from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
from put_client.rb:20:in `upload_file'
from put_client.rb:30

Here is the client code that is causing the problems:

<client>
require 'net/https'

def upload_file url, file
uri = URI.parse url
http = Net::HTTP.new uri.host, uri.port
if uri.scheme == 'https'
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
end

if file.respond_to? :read
file.rewind
stream = file
length = File.size file.path
else
stream = File.open file
length = File.size file
end

http.start do
req = Net::HTTP:ut.new uri.path
req.body_stream = stream
req.content_length = length
resp = http.request req
resp.value
end
end

if __FILE__ == $0
upload_file "http://localhost:8443/junk", ARGV.shift
end
</client>

Here is some server code I was using to test:

<server>
require 'webrick'

s = WEBrick::HTTPServer.new(ort => 8443)

class PutServlet < WEBrick::HTTPServlet::AbstractServlet
def do_PUT(req, res)
res['Content-Type'] = "text/html"
res.body = "Uploaded #{req.body}"
end
end

s.mount("/", PutServlet)
trap("INT"){ s.shutdown }
s.start
</server>

This only happens on Windows and it happens irrespective of the web
server involved. I've tried this on Mac OS X (WEBrick and nginx) and it
works fine. Running the client on Windows, though, I've tried this
against WEBrick and nginx and both exhibit the same behavior when
running the client from Windows.

Any ideas? Thanks.

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Toby Dipasquale
Guest
Posts: n/a
 
      04-17-2007
Toby DiPasquale wrote:
> I'm having a problem here running some file upload code on Windows. I've
> installed Ruby 1.8.6 from the one-click installer on Windows 2000 Server
> in VMware and on real hardware with Windows XP. The client, when running
> on Windows, always times/errors out no matter what. Here's the error:
>
> c:/ruby/lib/ruby/1.8/net/http.rb:2094:in `error!': 408 "Request Timeout
> " (Net::HTTPServerException)
> from c:/ruby/lib/ruby/1.8/net/http.rb:2103:in `value'
> from put_client.rb:25:in `upload_file'
> from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
> from put_client.rb:20:in `upload_file'
> from put_client.rb:30


This error is more indicative of what I've been seeing:

c:/ruby/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill': execution expired
(Timeout::Error)
from c:/ruby/lib/ruby/1.8/timeout.rb:56:in `timeout'
from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
from c:/ruby/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
from c:/ruby/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from c:/ruby/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from c:/ruby/lib/ruby/1.8/net/http.rb:2017:in `read_status_line'
from c:/ruby/lib/ruby/1.8/net/http.rb:2006:in `read_new'
from c:/ruby/lib/ruby/1.8/net/http.rb:1047:in `request'
from put_client.rb:25:in `upload_file'
from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
from put_client.rb:21:in `upload_file'
from put_client.rb:31


I've been looking on the engines for answers to this issue, both before
and after I posted, but it seems that whenever this has come up in the
past, the query has gone unanswered or was unrelated to this particular
problem.

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Toby Dipasquale
Guest
Posts: n/a
 
      04-18-2007
Toby Dipasquale wrote:
> Toby DiPasquale wrote:
>> I'm having a problem here running some file upload code on Windows. I've
>> installed Ruby 1.8.6 from the one-click installer on Windows 2000 Server
>> in VMware and on real hardware with Windows XP. The client, when running
>> on Windows, always times/errors out no matter what. Here's the error:
>>
>> c:/ruby/lib/ruby/1.8/net/http.rb:2094:in `error!': 408 "Request Timeout
>> " (Net::HTTPServerException)
>> from c:/ruby/lib/ruby/1.8/net/http.rb:2103:in `value'
>> from put_client.rb:25:in `upload_file'
>> from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
>> from put_client.rb:20:in `upload_file'
>> from put_client.rb:30

>
> This error is more indicative of what I've been seeing:
>
> c:/ruby/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill': execution expired
> (Timeout::Error)
> from c:/ruby/lib/ruby/1.8/timeout.rb:56:in `timeout'
> from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
> from c:/ruby/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
> from c:/ruby/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
> from c:/ruby/lib/ruby/1.8/net/protocol.rb:126:in `readline'
> from c:/ruby/lib/ruby/1.8/net/http.rb:2017:in `read_status_line'
> from c:/ruby/lib/ruby/1.8/net/http.rb:2006:in `read_new'
> from c:/ruby/lib/ruby/1.8/net/http.rb:1047:in `request'
> from put_client.rb:25:in `upload_file'
> from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
> from put_client.rb:21:in `upload_file'
> from put_client.rb:31
>
>
> I've been looking on the engines for answers to this issue, both before
> and after I posted, but it seems that whenever this has come up in the
> past, the query has gone unanswered or was unrelated to this particular
> problem.


For posterity: as described and discussed here:

http://blade.nagaokaut.ac.jp/cgi-bin...by-talk/229801

it turns out that threads and I/O don't mix in the One-Click Installer
at this time. Neither, apparently, do they mix in the MinGW builds I
made. Cygwin is not an option for me, so I will have to switch to a
different language. Thanks anyway.

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Emilio Tagua
Guest
Posts: n/a
 
      04-18-2007
On 4/17/07, Toby DiPasquale <(E-Mail Removed)> wrote:
> Hi all,
>
> I'm having a problem here running some file upload code on Windows. I've
> installed Ruby 1.8.6 from the one-click installer on Windows 2000 Server
> in VMware and on real hardware with Windows XP. The client, when running
> on Windows, always times/errors out no matter what. Here's the error:
>
> c:/ruby/lib/ruby/1.8/net/http.rb:2094:in `error!': 408 "Request Timeout
> " (Net::HTTPServerException)
> from c:/ruby/lib/ruby/1.8/net/http.rb:2103:in `value'
> from put_client.rb:25:in `upload_file'
> from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
> from put_client.rb:20:in `upload_file'
> from put_client.rb:30
>
> Here is the client code that is causing the problems:
>
> <client>
> require 'net/https'
>
> def upload_file url, file
> uri = URI.parse url
> http = Net::HTTP.new uri.host, uri.port
> if uri.scheme == 'https'
> http.use_ssl = true
> http.verify_mode = OpenSSL::SSL::VERIFY_NONE
> end
>
> if file.respond_to? :read
> file.rewind
> stream = file
> length = File.size file.path
> else
> stream = File.open file
> length = File.size file
> end
>
> http.start do
> req = Net::HTTP:ut.new uri.path
> req.body_stream = stream
> req.content_length = length
> resp = http.request req
> resp.value
> end
> end
>
> if __FILE__ == $0
> upload_file "http://localhost:8443/junk", ARGV.shift
> end
> </client>
>
> Here is some server code I was using to test:
>
> <server>
> require 'webrick'
>
> s = WEBrick::HTTPServer.new(ort => 8443)
>
> class PutServlet < WEBrick::HTTPServlet::AbstractServlet
> def do_PUT(req, res)
> res['Content-Type'] = "text/html"
> res.body = "Uploaded #{req.body}"
> end
> end
>
> s.mount("/", PutServlet)
> trap("INT"){ s.shutdown }
> s.start
> </server>
>
> This only happens on Windows and it happens irrespective of the web
> server involved. I've tried this on Mac OS X (WEBrick and nginx) and it
> works fine. Running the client on Windows, though, I've tried this
> against WEBrick and nginx and both exhibit the same behavior when
> running the client from Windows.
>
> Any ideas? Thanks.


You could try something like:
def method
begin
response = Timeout::timeout(5) { Net:whatever u want from NET Class) }
...
rescue Timeout::Error
puts 'Timeout!!!'
....
end
end

Hope it helps you!

 
Reply With Quote
 
Toby Dipasquale
Guest
Posts: n/a
 
      04-23-2007
Emilio Tagua wrote:
> On 4/17/07, Toby DiPasquale <(E-Mail Removed)> wrote:
>> from put_client.rb:25:in `upload_file'
>> uri = URI.parse url
>> else
>> end
>> require 'webrick'
>> s.mount("/", PutServlet)
>> Any ideas? Thanks.

> You could try something like:
> def method
> begin
> response = Timeout::timeout(5) { Net:whatever u want from NET
> Class) }
> ...
> rescue Timeout::Error
> puts 'Timeout!!!'
> ....
> end
> end
>
> Hope it helps you!


This would not help as the problem it the use of the timeout module in
the first place. The net/protocol module already wraps the call in a
Timeout::timeout() call and this is the source of the issue. That call
spawns a Ruby thread to enforce the timeout and the combination of the
thread use and I/O is what is triggering the problem. In any case, I've
already rewritten the code in Python and it works fine on all platforms.
Thanks anyway.

--
Toby DiPasquale

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Marcin Coles
Guest
Posts: n/a
 
      05-01-2007
Well, I'm no expert, and I certainly wasn't going to learn another
language, so I decided to do some tests because this was a problem for
me too.

I went into protocol.rb, to the rbuf_fill method (where it actually
starts the timeout thread).

the code there was

def rbuf_fill
timeout(@read_timeout) {
@rbuf << @io.sysread(1024)
}
end

now timeout takes 2 parameters - a time in seconds(?) and an exception
class to raise (defaults to Error).

When I changed the code to the following, it started to work for me.
New code:

def rbuf_fill
timeout(@read_timeout,ProtocolError) {
@rbuf << @io.sysread(1024)
}
end

Obviously this is not exactly extensively tested - but an interesting
results.

Marcin


Toby Dipasquale wrote:
> Emilio Tagua wrote:
>> On 4/17/07, Toby DiPasquale <(E-Mail Removed)> wrote:
>>> from put_client.rb:25:in `upload_file'
>>> uri = URI.parse url
>>> else
>>> end
>>> require 'webrick'
>>> s.mount("/", PutServlet)
>>> Any ideas? Thanks.

>> You could try something like:
>> def method
>> begin
>> response = Timeout::timeout(5) { Net:whatever u want from NET
>> Class) }
>> ...
>> rescue Timeout::Error
>> puts 'Timeout!!!'
>> ....
>> end
>> end
>>
>> Hope it helps you!

>
> This would not help as the problem it the use of the timeout module in
> the first place. The net/protocol module already wraps the call in a
> Timeout::timeout() call and this is the source of the issue. That call
> spawns a Ruby thread to enforce the timeout and the combination of the
> thread use and I/O is what is triggering the problem. In any case, I've
> already rewritten the code in Python and it works fine on all platforms.
> Thanks anyway.
>
> --
> Toby DiPasquale



--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Ben Brightwell
Guest
Posts: n/a
 
      06-05-2007
Interesting results!!! What an understatement!! How on earth did you
figure that out? It worked like a charm. Initially my problem was just
that the HTTP request took FOREVER... then i started randomly getting
the timeout errors mentioned above. I went into the net/protocol.rb file
and made this change and voila!. Not only did the errors go away, but
the request takes literally 1/30th of the time it did before it
eventually got the data or even timed out.. Marcin Coles, you are the
genius of the day! Thank you!

Marcin Coles wrote:
> Well, I'm no expert, and I certainly wasn't going to learn another
> language, so I decided to do some tests because this was a problem for
> me too.
>
> I went into protocol.rb, to the rbuf_fill method (where it actually
> starts the timeout thread).
>
> the code there was
>
> def rbuf_fill
> timeout(@read_timeout) {
> @rbuf << @io.sysread(1024)
> }
> end
>
> now timeout takes 2 parameters - a time in seconds(?) and an exception
> class to raise (defaults to Error).
>
> When I changed the code to the following, it started to work for me.
> New code:
>
> def rbuf_fill
> timeout(@read_timeout,ProtocolError) {
> @rbuf << @io.sysread(1024)
> }
> end
>
> Obviously this is not exactly extensively tested - but an interesting
> results.
>
> Marcin


--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Marcin Coles
Guest
Posts: n/a
 
      06-05-2007
Hey Ben
I'm glad it worked for you too, and thanks for the compliment. I was
happy to just get rid of those untrappable errors, so I didn't even
profile to see whether there were any other speed improvements -
performance is not a critical issue in my script.

Cheers,
M

Ben Brightwell wrote:
> Interesting results!!! What an understatement!! How on earth did you
> figure that out? It worked like a charm. Initially my problem was just
> that the HTTP request took FOREVER... then i started randomly getting
> the timeout errors mentioned above. I went into the net/protocol.rb file
> and made this change and voila!. Not only did the errors go away, but
> the request takes literally 1/30th of the time it did before it
> eventually got the data or even timed out.. Marcin Coles, you are the
> genius of the day! Thank you!
>
> Marcin Coles wrote:
>> Well, I'm no expert, and I certainly wasn't going to learn another
>> language, so I decided to do some tests because this was a problem for
>> me too.
>>
>> I went into protocol.rb, to the rbuf_fill method (where it actually
>> starts the timeout thread).
>>
>> the code there was
>>
>> def rbuf_fill
>> timeout(@read_timeout) {
>> @rbuf << @io.sysread(1024)
>> }
>> end
>>
>> now timeout takes 2 parameters - a time in seconds(?) and an exception
>> class to raise (defaults to Error).
>>
>> When I changed the code to the following, it started to work for me.
>> New code:
>>
>> def rbuf_fill
>> timeout(@read_timeout,ProtocolError) {
>> @rbuf << @io.sysread(1024)
>> }
>> end
>>
>> Obviously this is not exactly extensively tested - but an interesting
>> results.
>>
>> Marcin



--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Raghu Kumar
Guest
Posts: n/a
 
      06-26-2007
Hi,

Thanks a lot... I was struck with this problem ... now I have no
problem.

But in my project I have lot of different web pages to scrap, is
anything possible to tweak so that responses are fast ?

Raghu

Ben Brightwell wrote:
> Interesting results!!! What an understatement!! How on earth did you
> figure that out? It worked like a charm. Initially my problem was just
> that the HTTP request took FOREVER... then i started randomly getting
> the timeout errors mentioned above. I went into the net/protocol.rb file
> and made this change and voila!. Not only did the errors go away, but
> the request takes literally 1/30th of the time it did before it
> eventually got the data or even timed out.. Marcin Coles, you are the
> genius of the day! Thank you!
>
> Marcin Coles wrote:
>> Well, I'm no expert, and I certainly wasn't going to learn another
>> language, so I decided to do some tests because this was a problem for
>> me too.
>>
>> I went into protocol.rb, to the rbuf_fill method (where it actually
>> starts the timeout thread).
>>
>> the code there was
>>
>> def rbuf_fill
>> timeout(@read_timeout) {
>> @rbuf << @io.sysread(1024)
>> }
>> end
>>
>> now timeout takes 2 parameters - a time in seconds(?) and an exception
>> class to raise (defaults to Error).
>>
>> When I changed the code to the following, it started to work for me.
>> New code:
>>
>> def rbuf_fill
>> timeout(@read_timeout,ProtocolError) {
>> @rbuf << @io.sysread(1024)
>> }
>> end
>>
>> Obviously this is not exactly extensively tested - but an interesting
>> results.
>>
>> Marcin



--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Steve Midgley
Guest
Posts: n/a
 
      06-30-2007
Hi Raghu,

Since this weird Net timeout issue has a solution now, I'd recommend
that you spin out a series of threads to scrape your websites. That way,
even if each thread spends a long time waiting on Net:HTTP, you still
have lots of other threads working on other sites - aggregate wait time
goes way down if you have 100 simultaneous HTTP requests all waiting on
different sites. Be careful not to have too many threads request from
the same site at the same time - not only is this bad manners but some
systems will block you instantly if you do this.

Pragprog guys have a good little tutorial on threading which even
includes some info on net requests (though be sure to make the changes
described above for execution expired issues first):

http://www.rubycentral.com/book/tut_threads.html

Steve

Raghu Kumar wrote:
> Hi,
>
> Thanks a lot... I was struck with this problem ... now I have no
> problem.
>
> But in my project I have lot of different web pages to scrap, is
> anything possible to tweak so that responses are fast ?
>
> Raghu
>
> Ben Brightwell wrote:
>> Interesting results!!! What an understatement!! How on earth did you
>> figure that out? It worked like a charm. Initially my problem was just
>> that the HTTP request took FOREVER... then i started randomly getting
>> the timeout errors mentioned above. I went into the net/protocol.rb file
>> and made this change and voila!. Not only did the errors go away, but
>> the request takes literally 1/30th of the time it did before it
>> eventually got the data or even timed out.. Marcin Coles, you are the
>> genius of the day! Thank you!
>>
>> Marcin Coles wrote:
>>> Well, I'm no expert, and I certainly wasn't going to learn another
>>> language, so I decided to do some tests because this was a problem for
>>> me too.
>>>
>>> I went into protocol.rb, to the rbuf_fill method (where it actually
>>> starts the timeout thread).
>>>
>>> the code there was
>>>
>>> def rbuf_fill
>>> timeout(@read_timeout) {
>>> @rbuf << @io.sysread(1024)
>>> }
>>> end
>>>
>>> now timeout takes 2 parameters - a time in seconds(?) and an exception
>>> class to raise (defaults to Error).
>>>
>>> When I changed the code to the following, it started to work for me.
>>> New code:
>>>
>>> def rbuf_fill
>>> timeout(@read_timeout,ProtocolError) {
>>> @rbuf << @io.sysread(1024)
>>> }
>>> end
>>>
>>> Obviously this is not exactly extensively tested - but an interesting
>>> results.
>>>
>>> Marcin



--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Obsolete Errors and Deprecated Errors using Dotnet Framework 2.0 SenthilVel ASP .Net 0 06-07-2006 11:48 AM
Timeout::timeout and Socket timeout Mark Probert Ruby 1 10-06-2004 09:30 AM
Errors, errors, errors Mark Goldin ASP .Net 2 01-17-2004 08:05 PM
Re: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Bob Johnson ASP .Net 0 08-07-2003 12:52 PM



Advertisments