Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > ASP .Net > General Web Scraping Question

Reply
Thread Tools

General Web Scraping Question

 
 
Selden McCabe
Guest
Posts: n/a
 
      02-22-2004
I've been working on a web scraping program, and have the basics down.

But I don't understand the parameters.
Normally, you go to a URL (say a reverse yellow pages directory), and enter
some parameters (like area code, phone number, etc.) and POST this back to
the web. Then you parse the response, looking for the data you need.

Ofen I see examples where the data you post contains something like
"AreaCode=503&Number=5551212&x=1&y=2"

Where do the "x=1 and y=2" come from? I have some sites where my post
doesn't work. In one case, you are supposed to enter a contractor's license
number, and then click a button, and the result contains information about
the license. After I post what I think should work, the result coming back
is the same web page, with the contractor's number filled in.

Do the X and Y parameters involve invoking a button? How do you determine
what to use for the parameters?

Thanks in advance for any advice or pointers!
---Selden McCabe


 
Reply With Quote
 
 
 
 
Eric Lawrence [MSFT]
Guest
Posts: n/a
 
      02-22-2004
I suspect X and Y are passed by the browser when the user clicks on an image
map. Have you tried passing &x=1&y=1 in your post?

--
Thanks,

Eric Lawrence
Program Manager
Assistance and Worldwide Services

This posting is provided "AS IS" with no warranties, and confers no rights.


"Selden McCabe" <(E-Mail Removed)> wrote in message
news:#lFAcCO#(E-Mail Removed)...
> I've been working on a web scraping program, and have the basics down.
>
> But I don't understand the parameters.
> Normally, you go to a URL (say a reverse yellow pages directory), and

enter
> some parameters (like area code, phone number, etc.) and POST this back to
> the web. Then you parse the response, looking for the data you need.
>
> Ofen I see examples where the data you post contains something like
> "AreaCode=503&Number=5551212&x=1&y=2"
>
> Where do the "x=1 and y=2" come from? I have some sites where my post
> doesn't work. In one case, you are supposed to enter a contractor's

license
> number, and then click a button, and the result contains information about
> the license. After I post what I think should work, the result coming

back
> is the same web page, with the contractor's number filled in.
>
> Do the X and Y parameters involve invoking a button? How do you determine
> what to use for the parameters?
>
> Thanks in advance for any advice or pointers!
> ---Selden McCabe
>
>



 
Reply With Quote
 
 
 
 
Joerg Jooss
Guest
Posts: n/a
 
      02-22-2004
Selden McCabe wrote:
> I've been working on a web scraping program, and have the basics down.
>
> But I don't understand the parameters.
> Normally, you go to a URL (say a reverse yellow pages directory), and
> enter some parameters (like area code, phone number, etc.) and POST
> this back to the web. Then you parse the response, looking for the
> data you need.
>
> Ofen I see examples where the data you post contains something like
> "AreaCode=503&Number=5551212&x=1&y=2"
>
> Where do the "x=1 and y=2" come from? I have some sites where my post
> doesn't work. In one case, you are supposed to enter a contractor's
> license number, and then click a button, and the result contains
> information about the license. After I post what I think should
> work, the result coming back is the same web page, with the
> contractor's number filled in.
>
> Do the X and Y parameters involve invoking a button? How do you
> determine what to use for the parameters?


These could be hidden fields used by web application to store session state
on the client. Actually, it's not easy to implement web scraping for
"foreign" web applications where you don't have access to the code or at
least some inside knowledge.

Cheers,

--
Joerg Jooss
http://www.velocityreviews.com/forums/(E-Mail Removed)

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Scraping data from a web form submit Brent ASP .Net 8 10-19-2010 06:26 AM
General....very general.... no important for u forever hi Python 0 03-18-2009 08:21 AM
General Q about screen scraping KMA ASP .Net Web Services 3 10-21-2005 08:11 PM
Web Scraping/Site Scraping David Jones Python 4 07-13-2004 01:05 AM
Web Scraping Question Selden McCabe ASP .Net 1 02-21-2004 04:43 PM



Advertisments