Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > looping through & downloading web page - WATIR codes

Reply
Thread Tools

looping through & downloading web page - WATIR codes

 
 
curious
Guest
Posts: n/a
 
      12-05-2006
What I am trying to realize through code below is to read a line by
line in items.txt, and download each ensuing pages by having WATIR
click 'Next' link.

Now what I wish this code to do is when WATIR reaches the last page,
and there is no more "Next" link on the page, then go to the second
line in items.txt, searching again, and download ensuing pages again.

But unfortunately, the WATIR code close down when there is no more
"Next" link on the first item search.. and it does not go to searching
with the second line in items.txt.

Could anybody help me how I should adjust the code below so that when
there is no more "Next" link on the page, then the second line in
items.txt goes into searching, download each page by clicking "Next"
link.. on and on??

any suggestion will be deeply appreciated.

Thanks.

================================================== ===================
require 'watir'

ie = Watir::IE.start("http://www.test.com")

file = File.open("c:/items.txt", "r")
lines = file.readlines()

PageNo = 1

0.upto(lines.length - 1) {|j|

$item = lines[j]

ie.text_field(:name, "item").value = $item

ie.button(:name, "Search").click

a = ie.contains_text("Next")

while a

File.open("#{PageNo}.html", "wb") { |f|

f << '<html>'
f << ie.html
f << '</html>'

PageNo = PageNo + 1

ie.link(:text, "Next").click

}

end

}

file.close()
lines.clear()

 
Reply With Quote
 
 
 
 
curious
Guest
Posts: n/a
 
      12-05-2006
I found out what the problem is..

it is the last part..

ie.link(:text, "Next").click.. in the last web page, although "Next"
text is contained, link is not present since it is the last page..

Is there anyway, I can control "ie.link(:text, "Next").click" by
checking the presence of link beforehand?? so if link is present, then
proceed, otherwise.. halt.. I think my code will do as I wish once
this gets done..

thanks.

Paul Lutus wrote:
> curious wrote:
>
> > What I am trying to realize through code below is to read a line by
> > line in items.txt, and download each ensuing pages by having WATIR
> > click 'Next' link.
> >
> > Now what I wish this code to do is when WATIR reaches the last page,
> > and there is no more "Next" link on the page, then go to the second
> > line in items.txt, searching again, and download ensuing pages again.
> >
> > But unfortunately, the WATIR code close down when there is no more
> > "Next" link on the first item search.. and it does not go to searching
> > with the second line in items.txt.
> >
> > Could anybody help me how I should adjust the code below so that when
> > there is no more "Next" link on the page, then the second line in
> > items.txt goes into searching, download each page by clicking "Next"
> > link.. on and on??
> >
> > any suggestion will be deeply appreciated.

>
> Rather than simply rewriting your code for you, I offer these suggestions:
>
> 1. Always close the opened file handle when the write is completed. You are
> not closing your file handles until the last file is read, which in the
> worst case leaves many file handles open and abandoned.
>
> 2. Instead of doing this:
>
> 0.upto(lines.length - 1) {|j|
> $item = lines[j]
>
> Do it this way:
>
> lines.each do |item|
> # ... your code here
> end
>
> 3. Avoid use of global variables when possible.
>
> I would offer more suggestions but I don't actually understand what your
> program is meant to accomplish.
>
> --
> Paul Lutus
> http://www.arachnoid.com


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
looping through json array loops through the characters instead ofthe values Aaron Javascript 2 04-10-2011 05:58 PM
Watir: How to return a Watir::IE object for an existing IE? Anukul Singhal Ruby 1 05-15-2008 04:57 PM
downloading web page as HTML accessed through WATIR michael Ruby 3 10-23-2006 11:52 PM
downloading web pages in watir michael Ruby 3 08-08-2006 11:35 PM
Virtual Key Codes, Scan Codes and ASCII Codes in C gj_williams2000@yahoo.co.uk C Programming 2 08-20-2005 11:04 AM



Advertisments
 



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57