Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > C Web crawler code

Reply
Thread Tools

C Web crawler code

 
 
Sanjay Patra
Guest
Posts: n/a
 
      11-18-2004
Hi All,

I am looking for a simple C/ C++ web crawler code.

It should be very simple with minimal functionality.
I am particularly interested in the code to grab the content of a url
and the code to search this content for other urls.

Thanks.
 
Reply With Quote
 
 
 
 
Raymond Martineau
Guest
Posts: n/a
 
      11-18-2004
On 17 Nov 2004 16:01:44 -0800, http://www.velocityreviews.com/forums/(E-Mail Removed) (Sanjay Patra) wrote:

>Hi All,
>
>I am looking for a simple C/ C++ web crawler code.
>
>It should be very simple with minimal functionality.


You will have to consult the newsgroup of the platform you wish the web
crawler to run on.

For example:

news:comp.os.msdos.programmer DOS, BIOS, Memory Models,
interrupts, screen handling,
hardware
news:comp.os.ms-windows.programmer.misc MS/Windows: Mice, DLLs, hardware
news:comp.os.ms-windows.programmer.win32 MS 32-bit API
news:comp.os.os2.programmer.misc OS/2 Programming
news:comp.sys.mac.programmer.misc Macintosh Programming
news:comp.unix.programmer General Unix: processes, pipes,
POSIX, curses, sockets
news:comp.unix.[vendor] Various Unix vendors
news:comp.os.linux.development.apps Linux application programming


>I am particularly interested in the code to grab the content of a url
>and the code to search this content for other urls.


Code for retrieving URLs is not possible in standard C and needs a
third-party library (or a set of wrapper functions), but it is on-topic to
discuss how to retrieve URLs from the retrieved content.

The simplest way to detect a Url is to call char *strstr( const char
*string, const char *strCharSet ), the first parameter being a line of the
file being scanned, and the second parameter being a simple "http://". If
you want to do a case-insensitive search, you will have to roll your own
function, since there isn't even a "stristr" in a non-standard
implementation.

I can't tell you how to detect the end-point of the URL - you will have to
take a look at the RFCs for that.


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Web Crawler Paul Morrison Java 3 06-30-2012 04:17 PM
web crawler in python or C? abhinav Python 13 02-20-2006 09:07 PM
Web Crawler Sanjay Patra C++ 2 11-18-2004 06:12 AM
Web Crawler / Spider Commercial Software Info Request Gray Ghost Computer Support 1 11-07-2004 01:48 PM
Web Crawler Hans Computer Support 1 07-20-2003 03:20 PM



Advertisments