Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > ASP .Net > Screen Scraper

Reply
Thread Tools

Screen Scraper

 
 
_eee_
Guest
Posts: n/a
 
      02-25-2004
Does anyone know of a simple code module that can do screen scraping,
including simulating user-entered pushbuttons, etc.

I can get the first screen on a website with HttpWebRequest, but I
need to follow up to that by simulating user entries, then get
subsequent response screens to that entered data.

I figure someone has done this elegantly.
 
Reply With Quote
 
 
 
 
Steve Chatham
Guest
Posts: n/a
 
      02-25-2004
This works for me:

objRequest =
System.Net.HttpWebRequest.Create("http://myserver.com/sales/oldsystemreport.
asp?User=" & Session("User") & "&Yr=" & Request.QueryString("yr") & "&Rpt="
& sRpt & "&P=" & Session("Permission") & "&T=" &
cmbTerritory.SelectedItem.Value & "&O=" & cmbGroup.SelectedItem.Value &
"&R=" & cmbRegion.SelectedItem.Value & "&Chan=" & Chan & "&sSoldto=" &
ssoldto & "&sShipto=" & J6shipto & "&J7Periodselect=" & J7Period & "&J7Opt="
& J7Option & "&LC=" & J6lawcon & "&msoldto=" & msoldto & "&BA=" & xBA &
"&RG=" & xRG & "&FC=" & xFC & "&SC=" & xSC)

'objRequest.Credentials = New NetworkCredential("sql", "sql", "REMINGTON")

objRequest.Credentials = New NetworkCredential("username", "password",
"domain")

objResponse = objRequest.GetResponse()

Dim sr As New StreamReader(objResponse.GetResponseStream())

result = sr.ReadToEnd()

iLength = Len(result)

' commented out to debug 03/27/03

''''If InStr(result, "No records matched") > 0 Then

'''' result = "<br>No Records Matched <br>"

''''Else

'''' If sRpt = "Samples" Then

'''' Dim sb As New StringBuilder()

'''' sb.Append("<table border=""9"" ")

'''' iBegin = InStr(result, sb.ToString)

'''' Else

'''' iBegin = InStr(result, "<td><table border=")

'''' End If

'''' 'iBegin = InStr(result, "<td><table border=")

'''' '<table border="9" align="left"

'''' Dim iend = InStr(result, "<p><small>The report ran")

'''' result = Mid(result, iBegin, (iLength - ((iLength - iend) + iBegin)))

'''' result = result.Replace("<td><table border=""9"" align=""left""",
"<td><table border=""9"" align=""center""")

'''' 'result = "<align=""center"">" & result

''''End If

MyPage.Text = result

Button1.Visible = False



The commented code is out for debugging purposes, but I call an asp page
that parses out the stuff I call it with, and calls a stored procedure.



SC

"_eee_" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Does anyone know of a simple code module that can do screen scraping,
> including simulating user-entered pushbuttons, etc.
>
> I can get the first screen on a website with HttpWebRequest, but I
> need to follow up to that by simulating user entries, then get
> subsequent response screens to that entered data.
>
> I figure someone has done this elegantly.



 
Reply With Quote
 
 
 
 
Peter Rilling
Guest
Posts: n/a
 
      02-25-2004
This assumes that the request will accept parameters using a "GET" commend.
If the site you are capturing uses "POST", you may need to use the Headers
collection of the HttpWebRequest object. Never done this myself, so this is
about as much info as I can give you.

"Steve Chatham" <(E-Mail Removed)> wrote in message
news:Ot3BI59#(E-Mail Removed)...
> This works for me:
>
> objRequest =
>

System.Net.HttpWebRequest.Create("http://myserver.com/sales/oldsystemreport.
> asp?User=" & Session("User") & "&Yr=" & Request.QueryString("yr") &

"&Rpt="
> & sRpt & "&P=" & Session("Permission") & "&T=" &
> cmbTerritory.SelectedItem.Value & "&O=" & cmbGroup.SelectedItem.Value &
> "&R=" & cmbRegion.SelectedItem.Value & "&Chan=" & Chan & "&sSoldto=" &
> ssoldto & "&sShipto=" & J6shipto & "&J7Periodselect=" & J7Period &

"&J7Opt="
> & J7Option & "&LC=" & J6lawcon & "&msoldto=" & msoldto & "&BA=" & xBA &
> "&RG=" & xRG & "&FC=" & xFC & "&SC=" & xSC)
>
> 'objRequest.Credentials = New NetworkCredential("sql", "sql", "REMINGTON")
>
> objRequest.Credentials = New NetworkCredential("username", "password",
> "domain")
>
> objResponse = objRequest.GetResponse()
>
> Dim sr As New StreamReader(objResponse.GetResponseStream())
>
> result = sr.ReadToEnd()
>
> iLength = Len(result)
>
> ' commented out to debug 03/27/03
>
> ''''If InStr(result, "No records matched") > 0 Then
>
> '''' result = "<br>No Records Matched <br>"
>
> ''''Else
>
> '''' If sRpt = "Samples" Then
>
> '''' Dim sb As New StringBuilder()
>
> '''' sb.Append("<table border=""9"" ")
>
> '''' iBegin = InStr(result, sb.ToString)
>
> '''' Else
>
> '''' iBegin = InStr(result, "<td><table border=")
>
> '''' End If
>
> '''' 'iBegin = InStr(result, "<td><table border=")
>
> '''' '<table border="9" align="left"
>
> '''' Dim iend = InStr(result, "<p><small>The report ran")
>
> '''' result = Mid(result, iBegin, (iLength - ((iLength - iend) + iBegin)))
>
> '''' result = result.Replace("<td><table border=""9"" align=""left""",
> "<td><table border=""9"" align=""center""")
>
> '''' 'result = "<align=""center"">" & result
>
> ''''End If
>
> MyPage.Text = result
>
> Button1.Visible = False
>
>
>
> The commented code is out for debugging purposes, but I call an asp page
> that parses out the stuff I call it with, and calls a stored procedure.
>
>
>
> SC
>
> "_eee_" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed)...
> > Does anyone know of a simple code module that can do screen scraping,
> > including simulating user-entered pushbuttons, etc.
> >
> > I can get the first screen on a website with HttpWebRequest, but I
> > need to follow up to that by simulating user entries, then get
> > subsequent response screens to that entered data.
> >
> > I figure someone has done this elegantly.

>
>



 
Reply With Quote
 
_eee_
Guest
Posts: n/a
 
      02-25-2004
>> "_eee_" <(E-Mail Removed)> wrote in message
>> > Does anyone know of a simple code module that can do screen scraping,
>> > including simulating user-entered pushbuttons, etc.
>> >
>> > I can get the first screen on a website with HttpWebRequest, but I
>> > need to follow up to that by simulating user entries, then get
>> > subsequent response screens to that entered data.


>"Steve Chatham" <(E-Mail Removed)> wrote in message
>news:Ot3BI59#(E-Mail Removed)...
>> objRequest =
>>System.Net.HttpWebRequest.Create("http://myserver.com/sales/oldsystemreport.
>> asp?User=" & Session("User") & "&Yr=" & Request.QueryString("yr") &


[code snipped. Thanks, Steve!]

<(E-Mail Removed)> wrote:
>This assumes that the request will accept parameters using a "GET" commend.
>If the site you are capturing uses "POST", you may need to use the Headers
>collection of the HttpWebRequest object. Never done this myself, so this is
>about as much info as I can give you.


Thanks to Steve for the posted code. But Peter is correct--I do need
to do this with 'Post'. I should have mentioned that.

I didn't think it would be an uncommon thing to do. Otherwise, how is
anyone scraping data from a site that requires entering text or
pushing a button?

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
my first screen scraper mensanator@aol.com Python 0 12-02-2007 04:25 AM
Python Screen Scraper James Stroud Python 7 04-25-2007 05:19 PM
Simple screen scraper using scrAPI doog Ruby 8 11-30-2006 08:27 PM
Screen scraper again _eee_ ASP .Net 6 02-28-2004 05:34 PM
Screen Scraper with Java API Dave Monroe Java 1 10-17-2003 01:51 PM



Advertisments