Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > ASP .Net > ASP .Net Web Services > Screen Scraping HTTPS

Reply
Thread Tools

Screen Scraping HTTPS

 
 
Michael C#
Guest
Posts: n/a
 
      06-02-2005
Hi all,

I'm writing a web service that uses HttpWebRequest/HttpWebResponse to screen
scrape. It works fine for regular HTTP sites, but for an HTTPS site (no
username/password) it gets stuck... I think the certificate is causing it
to fail. Here's the error message I get: "The underlying connection was
closed: Could not establish trust relationship with remote server." Any
ideas? Thanks


 
Reply With Quote
 
 
 
 
Michael C#
Guest
Posts: n/a
 
      06-02-2005
Scratch that, I found a solution. Thanks.

"Michael C#" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Hi all,
>
> I'm writing a web service that uses HttpWebRequest/HttpWebResponse to
> screen scrape. It works fine for regular HTTP sites, but for an HTTPS
> site (no username/password) it gets stuck... I think the certificate is
> causing it to fail. Here's the error message I get: "The underlying
> connection was closed: Could not establish trust relationship with remote
> server." Any ideas? Thanks
>
>



 
Reply With Quote
 
 
 
 
Jon Maz
Guest
Posts: n/a
 
      06-06-2005
Can you post your solution? That way you'll help anyone else who has the
same issue.


 
Reply With Quote
 
Michael C#
Guest
Posts: n/a
 
      06-07-2005
Yeah sure. The problem was that the site name and certificate name didn't
match. It's a government site so they were using the certificate of their
'parent department'. This was causing the Certificate Validation to fail.
To get around it, I created a Class that implements the ICertificatePolicy
interface. In the CheckValidationResult public function of the Class, I
returned True. Finally I assigned an instance of the Class to the
ServicePointManager.CertificatePolicy property.

One word of caution: this pretty much beats any security via certificates
completely into the ground. To get some level of security back, you can
code the CheckValidationResult function to check values of the Certificate
when it's passed in, like certificate.GetName(), GetEffectiveDateString(),
GetExpirationDateString(), etc.

"Jon Maz" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Can you post your solution? That way you'll help anyone else who has the
> same issue.
>
>



 
Reply With Quote
 
hellrazor
Guest
Posts: n/a
 
      06-10-2005
"Michael C#" <(E-Mail Removed)> wrote in
news:(E-Mail Removed):

> Scratch that, I found a solution. Thanks.


You should have said "Scrape that"
 
Reply With Quote
 
Michael C#
Guest
Posts: n/a
 
      06-10-2005
LOL. There's one in every crowd, right?

"hellrazor" <(E-Mail Removed)> wrote in message
news:Xns96717024654D7jorgeanotherworldcom@207.46.2 48.16...
> "Michael C#" <(E-Mail Removed)> wrote in
> news:(E-Mail Removed):
>
>> Scratch that, I found a solution. Thanks.

>
> You should have said "Scrape that"



 
Reply With Quote
 
J. Michael Terenin
Guest
Posts: n/a
 
      12-18-2005
Michael,
Could you revisit this thread discussion and perhaps show an example of your
code ? I am attempting the same thing, using Fiddler as a debugger, and am
noticing a similar breakdown when I try an access a web page over SSL (https)
.. I actually found your solution on another blog site, but it still didn't
help me resolve my block.

Much thanks !
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
newbie: HTTPS screen scraping user@domain.invalid Python 2 04-21-2007 06:36 PM
Scheduling a screen-scraping progam on a locked PC? =?Utf-8?B?VGludGluTWlsb3U=?= Microsoft Certification 7 01-12-2005 06:19 PM
Web Scraping/Site Scraping David Jones Python 4 07-13-2004 01:05 AM
HTML Screen Scraping Q George Durzi ASP .Net 2 02-25-2004 05:17 PM
Screen Scraping C# Robert Martinez ASP .Net 0 08-26-2003 09:59 PM



Advertisments