Server response validation?
 

This may be a vague question but which perl network tools or procedures are
good to determine if a hostname or IP exist and that is suitable for a cgi
request that allows user-input of any URL string?

An HTTP status code may return any response, such as 200, 404 or something
else. Whatever the response may be, as long as there is an actual response,
DNS record or server existing, a subroutine should return true. If however
the hostname or IP is non-existing, or if the user input has a wrong
syntax, the perl code should return false.

Presumably countless modules and procedures exist that do this and more.
Any ideas for good solutions would be much appreciated.

Many thanks,
Tuxedo

Re: Server response validation?
 

Ben Morrow wrote:
>
> Quoth Tuxedo <tuxedo@mailinator.com>:
> > This may be a vague question but which perl network tools or procedures
> > are good to determine if a hostname or IP exist and that is suitable for
> > a cgi request that allows user-input of any URL string?
>
> The core gethostbyname function will tell you if a given name has an
> address in the DNS. The Net::DNS module will let you make more specific
> queries if you need to.

Thanks for the advise, most helpful!

I'm not sure what I need yet but before making DNS requests I think would
be a good idea to validate the URL input.

I read that gethostbyname is not IPv6 compatible which may run into trouble:
http://stackoverflow.com/questions/1...-when-given-ip

The Net::DNS tools look interesting and in combination with Net::LibIDN it
appears to support IDNs.

I'm not quite sure how to validate a URI and I guess there is no simple way
to do so by existing core modules.

To cover a growing number of IDNs, IDNsNet-IDN-Encode-2.003 seem to be a
good tool in converting IDNs to their ASCII representations:
http://search.cpan.org/~cfaerber/Net.../IDN/Encode.pm

I will test Data::Validate::URI first, which require various non-core
modules, namely Data::Validate::IP, Net::Domain::TLD as well as
Net::Netmask.

> Determining if an IP exists is harder, since you can't in general assume
> any given IP will respond to any particular protocol. Net::Ping would be
> a start, but a lot of sites block external pings (even though they
> shouldn't).
>
> > An HTTP status code may return any response, such as 200, 404 or
> > something else. Whatever the response may be, as long as there is an
> > actual response, DNS record or server existing, a subroutine should
> > return true. If however the hostname or IP is non-existing, or if the
> > user input has a wrong syntax, the perl code should return false.
>
> Oh, you didn't say 'HTTP'. If you want to check an HTTP server exists,
> you can either just open a connection to port 80 with IO::Socket::INET
> (and close it again without sending a response), or send a HEAD request
> with LWP::Simple.
>
> You should be aware that some sites may consider this sort of behaviour
> to be port-scanning, and may take action to prevent you from doing it.
> In particular, you should be very careful not to probe any particular
> server too often, as that would almost certainly be considered abuse.

Thanks for the words of warning.

Tuxedo

>
> Ben
>

Re: Server response validation?
 

Ben Morrow wrote:

[...]

> Socket6.pm will give you the getaddrinfo function for making v4/v6
> hostname lookups.

[...]

> Well, URI.pm will do basic syntactic validation, which is a start, or
> you could use Regexp::Common, which has patterns for validating URIs.

[...]

> Hah! You seem to be solving this on your own... good. I didn't know
> about that. Note that it doesn't appear to do any network tests, so a
> URI like http://example.com/foo/bar will pass.

Yes, I read that in the docs. My first step will be to validate a URI
format and not that it actually exists through a network request. The URI
may be standard domain, a new style IDN representation or even an IP. The
network request comes after. So thanks for Socket6.pm, URI.pm and
Regexp::Common. I will test all!

Tuxedo

Re: Server response validation?
 

On 2012-11-10 19:48, Ben Morrow <ben@morrow.me.uk> wrote:
> Note that it doesn't appear to do any network tests, so a URI like
> http://example.com/foo/bar will pass.

% wget http://example.com/foo/bar
--2012-11-11 13:43:26-- http://example.com/foo/bar
Resolving example.com (example.com)... 2001:500:88:200::10, 192.0.43.10
Connecting to example.com (example.com)|2001:500:88:200::10|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.iana.org/domains/example/ [following]
--2012-11-11 13:43:27-- http://www.iana.org/domains/example/
Resolving www.iana.org (www.iana.org)... 2620:0:2d0:200::8, 192.0.32.8
Connecting to www.iana.org (www.iana.org)|2620:0:2d0:200::8|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `bar'

[ <=> ] 2,966 --.-K/s in 0.001s

2012-11-11 13:43:28 (3.39 MB/s) - `bar' saved [2966]

SCNR,
hp


--
_ | Peter J. Holzer | Fluch der elektronischen Textverarbeitung:
|_|_) | Sysadmin WSR | Man feilt solange an seinen Text um, bis
| | | hjp@hjp.at | die Satzbestandteile des Satzes nicht mehr
__/ | http://www.hjp.at/ | zusammenpaßt. -- Ralph Babel

Re: Server response validation?
 

Shmuel (Seymour J.) Metz wrote:

> In <k7k6jr$148$1@news.albasani.net>, on 11/10/2012
> at 01:20 AM, Tuxedo <tuxedo@mailinator.com> said:
>
> >This may be a vague question but which perl network tools or
> >procedures are good to determine if a hostname or IP exist
>
> What does it mean for an IP to exist? Does a hostname exist if there
> is an AAAA but no A? How do you want to treat CNAME?

I'm not quite sure.

> You really need to nail down the specifics of the problem before you
> look for a solution.

You are right, my questions were posted somewhat prematurely.

> Have you looked at Regexp-Common or URI
> <http://search.cpan.org/~gaas/URI-1.60/>?

Thanks, I'm looking into these. In the meantime, I found that
Data::Validate::URI appears to be working well in validating http and/or
https URI's in case they are the only things one need to do.

Tuxedo