Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Odd result when attempting to use Mechanize in parallel with Threads

Reply
Thread Tools

Odd result when attempting to use Mechanize in parallel with Threads

 
 
Richard Conroy
Guest
Posts: n/a
 
      12-07-2006
I wrote a simple tool to iterate a network to try and find web servers
running on specific ports. We have a lot of devices & software with a
web UI, and I thought that this would be a handy way to find them,
and even tell what they are.

I thought this would be a handy coding project too, and a good way to
cut my teeth on Ruby threads, and build up some usage with
Mechanize.

BTW I am running this on *Windows XP*.

However my code is quite obviously executing this serially. Is there
something obviously wrong with my code below? (results after
code snippet). I am aware this could make my machine choke from
thread overkill, but I wanted to get it working in parallel first.
Perhaps Mechanize instances have some shared elements?

============================
require 'mechanize'

threads = Array.new
puts "sweep of 153.200.72.* segment http ports"
(1..254).each do |ran|
threads << Thread.new(ran) { |r|
agent = WWW::Mechanize.new
agent.user_agent_alias = 'Windows Mozilla'
ports = [80,8080]
ports.each do |p|
begin
page = agent.get("http://153.200.72."+r.to_s+":"+p.to_s)
puts "153.200.72."+r.to_s+":"+p.to_s+" - "+page.title
rescue
puts "153.200.72."+r.to_s+":"+p.to_s+" - NOTHING"
end
end
}
threads.each { |aThread| aThread.join }
end
============================

153.200.72.10:80 - NOTHING
153.200.72.10:8080 - NOTHING
153.200.72.11:80 - NOTHING
153.200.72.11:8080 - NOTHING
153.200.72.12:80 - NOTHING
153.200.72.12:8080 - NOTHING
153.200.72.13:80 - NOTHING
153.200.72.13:8080 - NOTHING
153.200.72.14:80 - NOTHING
153.200.72.14:8080 - NOTHING
153.200.72.15:80 - NOTHING
153.200.72.15:8080 - NOTHING
153.200.72.16:80 - NOTHING
153.200.72.16:8080 - NOTHING
153.200.72.17:80 - NOTHING
153.200.72.17:8080 - NOTHING

 
Reply With Quote
 
 
 
 
ara.t.howard@noaa.gov
Guest
Posts: n/a
 
      12-07-2006
On Thu, 7 Dec 2006, Richard Conroy wrote:

> I wrote a simple tool to iterate a network to try and find web servers
> running on specific ports. We have a lot of devices & software with a
> web UI, and I thought that this would be a handy way to find them,
> and even tell what they are.
>
> I thought this would be a handy coding project too, and a good way to
> cut my teeth on Ruby threads, and build up some usage with
> Mechanize.
>
> BTW I am running this on *Windows XP*.
>
> However my code is quite obviously executing this serially. Is there
> something obviously wrong with my code below? (results after
> code snippet). I am aware this could make my machine choke from
> thread overkill, but I wanted to get it working in parallel first.
> Perhaps Mechanize instances have some shared elements?
>
> ============================


require 'mechanize'

threads = Array.new

puts "sweep of 153.200.72.* segment http ports"

(1..254).each do |ran|
threads << Thread.new(ran) { |r|
agent = WWW::Mechanize.new
agent.user_agent_alias = 'Windows Mozilla'
ports = [80,8080]
ports.each do |p|
begin
page = agent.get("http://153.200.72."+r.to_s+":"+p.to_s)
puts "153.200.72."+r.to_s+":"+p.to_s+" - "+page.title
rescue
puts "153.200.72."+r.to_s+":"+p.to_s+" - NOTHING"
end
end
}
end

threads.each { |aThread| aThread.join } # THIS MUST BE OUTSIDE THE LOOP!



fyi. starting a thread, and then immediately joining it is the same as not
using a thread at all!

another fyi - threads are io (even socket io) is a dealy combination on
windows. run this on linux/mac if possible.

regards.

-a
--
if you want others to be happy, practice compassion.
if you want to be happy, practice compassion. -- the dalai lama

 
Reply With Quote
 
 
 
 
Richard Conroy
Guest
Posts: n/a
 
      12-07-2006
On 12/7/06, http://www.velocityreviews.com/forums/(E-Mail Removed) <(E-Mail Removed)> wrote:
> threads.each { |aThread| aThread.join } # THIS MUST BE OUTSIDE THE LOOP!


<homer>*d'Oh</homer>

> fyi. starting a thread, and then immediately joining it is the same as not
> using a thread at all!


Ah yes, cutting & pasting a line too high ....

> another fyi - threads are io (even socket io) is a dealy combination on
> windows. run this on linux/mac if possible.


Has to be windows, but this isn't mission critical code - just a
development tool that may eventually post the results to a wiki or
something. I can break this
up a bit so it doesn't kill my laptop later.

> regards.


Thanks. I knew it had to a WTF.

 
Reply With Quote
 
Richard Conroy
Guest
Posts: n/a
 
      12-12-2006
On 12/12/06, Shiwei Zhang <(E-Mail Removed)> wrote:
> Hi, Richard,
>
> Actually in Ruby, only by the method ".new" we can make threads
> run in parallel rather than serially. And I think it can meet your
> requirement, pls see the programs<multithreads_ProbingHttp.rb> I post at
> the end of this mail, plus the running results.
> Firstly pls notice the following points: 1) The method ".new"
> means "Creates and runs a new thread to execute the instructions given
> in block". 2) The method ".join" means "The calling thread will suspend
> execution and run the called thread. Does not return until the called
> thread exits or until limit seconds have passed".
> ".new" doesn't only mean "creates", it means both "creates" and
> "runs". So ".new" can make son threads run in parallel. And ".join"
> needs to wait for the exit of the called thread, so it gives you the
> illusion that the theads are running serially, but in fact ".join" just
> wraps up the threads. It is inappropriate for us to say whether ".join"
> is making threads run in parallel or serially. We can say ".join" is
> serially waiting for the exits of threads that might be already running
> in parallel.


This is what I noticed. I join up 5 threads at a time, the output jumps
up in batches of 5. This does slow down the algorithm, especially
if there is a lot of positive results - most of these threads are
waiting for the http
connection to timeout.

But I run this thing at night anyway.

As an aside, I have had difficulty getting more than ~ 5 joined threads
to work at all in windows.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Parallel in, Parallel out shift register Vivek Menon VHDL 0 06-10-2011 10:15 PM
Parallel in, Parallel out shift register Vivek Menon VHDL 5 06-08-2011 03:56 PM
Parallel port control with USB->Parallel converter Soren Python 4 02-14-2008 03:18 PM
VERY odd routing behavior when attempting VPN connections over Wifi Robert Gordon Wireless Networking 0 08-25-2005 04:04 PM
1. Ruby result: 101 seconds , 2. Java result:9.8 seconds, 3. Perl result:62 seconds Michael Tan Ruby 32 07-21-2005 03:23 PM



Advertisments