Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > finding subdirectories from starting URL

Reply
Thread Tools

finding subdirectories from starting URL

 
 
Alan
Guest
Posts: n/a
 
      11-12-2007
I want to find subdirectories from a starting URL. For example,
if I start with http://www.someplace.net, I want to be able to find
the subdirectories there, e.g.:

http://www.someplace.net/documentation/
http://www.someplace.net/about/
http://www.someplace.net/images/

Are there Java methods that facilitate this?

Thanks, Alan

 
Reply With Quote
 
 
 
 
Daniel Pitts
Guest
Posts: n/a
 
      11-12-2007
Alan wrote:
> I want to find subdirectories from a starting URL. For example,
> if I start with http://www.someplace.net, I want to be able to find
> the subdirectories there, e.g.:
>
> http://www.someplace.net/documentation/
> http://www.someplace.net/about/
> http://www.someplace.net/images/
>
> Are there Java methods that facilitate this?
>
> Thanks, Alan
>

There is no easy way to do that, unless someplace.net gives you a
listing page. Generally, in order to do that, you have to have either
direct access to the disk, access to an FTP account on that machine, or
you have to crawl the web site and parse out the urls.

Note, this is not a limitation of Java, but simply a result of the way
http works.

There are plenty of web-crawling libraries/programs out there, I suggest
you Google for them.

Good luck,
Daniel

--
Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
 
Reply With Quote
 
 
 
 
Andrew Thompson
Guest
Posts: n/a
 
      11-12-2007
Alan wrote:
>I want to find subdirectories from a starting URL. For example,
>if I start with http://www.someplace.net, I want to be able to find
>the subdirectories there, ..


Why? What business is that of yours?
What does this ability offer to the end user?

--
Andrew Thompson
http://www.athompson.info/andrew/

Message posted via JavaKB.com
http://www.javakb.com/Uwe/Forums.asp...neral/200711/1

 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      11-12-2007
On Mon, 12 Nov 2007 00:20:17 -0000, Alan <(E-Mail Removed)>
wrote, quoted or indirectly quoted someone who said :

>http://www.someplace.net/documentation/
>http://www.someplace.net/about/
>http://www.someplace.net/images/
>
> Are there Java methods that facilitate this?


In general no. It is considered confidential information. Sometimes a
server will give you a directory listing in HTML if you give it an URL
of a directory without in index.html file in it.
--
Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com
 
Reply With Quote
 
Sherman Pendley
Guest
Posts: n/a
 
      11-12-2007
Daniel Pitts <(E-Mail Removed)> writes:

> Alan wrote:
>> I want to find subdirectories from a starting URL. For example,
>> if I start with http://www.someplace.net, I want to be able to find
>> the subdirectories there, e.g.:
>>
>> http://www.someplace.net/documentation/
>> http://www.someplace.net/about/
>> http://www.someplace.net/images/
>>
>> Are there Java methods that facilitate this?
>>
>> Thanks, Alan
>>

> There is no easy way to do that, unless someplace.net gives you a
> listing page. Generally, in order to do that, you have to have either
> direct access to the disk, access to an FTP account on that machine,
> or you have to crawl the web site and parse out the urls.
>
> Note, this is not a limitation of Java, but simply a result of the way
> http works.


In particular, note that documentation, about, and images may not even be
directories at all. A content-management system could use them as keys
into a database of managed documents, for example, or as category keywords
that are used to dynamically assemble a list of documents in the specified
category.

sherm--

--
WV News, Blogging, and Discussion: http://wv-www.com
Cocoa programming in Perl: http://camelbones.sourceforge.net
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      11-12-2007
Roedy Green wrote:
> On Mon, 12 Nov 2007 00:20:17 -0000, Alan <(E-Mail Removed)>
> wrote, quoted or indirectly quoted someone who said :
>
>> http://www.someplace.net/documentation/
>> http://www.someplace.net/about/
>> http://www.someplace.net/images/
>>
>> Are there Java methods that facilitate this?

>
> In general no. It is considered confidential information. Sometimes a
> server will give you a directory listing in HTML if you give it an URL
> of a directory without in index.html file in it.


In addition, many directories on the server hard drive, while they are
subdirectories of the web document directory, will not be accessible to public
clients. A classic example is the WEB-INF/ directory tree in Java EE apps,
but Apache .htaccess can also restrict directories.

--
Lew
 
Reply With Quote
 
Andrew Thompson
Guest
Posts: n/a
 
      11-12-2007
Roedy Green wrote:
>>http://www.someplace.net/documentation/

...
>> Are there Java methods that facilitate this?

>
>In general no. It is considered confidential information. Sometimes a
>server will give you a directory listing in HTML if you give it an URL
>of a directory without in index.html file in it.


An ironic side note is that I'd like my server to *allow*
automatic 'directory listing' for dirs with no index.html,
but cannot figure how to achieve it using the arcane
..contol panel ..thingy the host offers.

If directory indexing *is* turned on, it is a perhaps tedious
but mundane task to parse the resulting HTML, looking for
links to sub-dirs and resources.

--
Andrew Thompson
http://www.athompson.info/andrew/

Message posted via JavaKB.com
http://www.javakb.com/Uwe/Forums.asp...neral/200711/1

 
Reply With Quote
 
Chris ( Val )
Guest
Posts: n/a
 
      11-12-2007
On Nov 12, 5:11 pm, "Andrew Thompson" <u32984@uwe> wrote:
> Roedy Green wrote:
> >>http://www.someplace.net/documentation/

> ..
> >> Are there Java methods that facilitate this?

>
> >In general no. It is considered confidential information. Sometimes a
> >server will give you a directory listing in HTML if you give it an URL
> >of a directory without in index.html file in it.

>
> An ironic side note is that I'd like my server to *allow*
> automatic 'directory listing' for dirs with no index.html,
> but cannot figure how to achieve it using the arcane
> .contol panel ..thingy the host offers.


[snip]

If it is an IIS web server, you will usually find a checkbox
in the IIS server properties, or for memory, you can even get
to it by right clicking on the virtual directory and editing
the properties.

Tomcat for example has the following setting in its web.xml file:

<init-param>
<param-name>listings</param-name>
<param-value>true</param-value>
</init-param>

But I think it's only good whilst developing.

--
Chris

 
Reply With Quote
 
Alan
Guest
Posts: n/a
 
      11-12-2007
Thanks for the information. I think I shall just follow href
links instead of finding directories.

Thanks, Alan


 
Reply With Quote
 
Andrew Thompson
Guest
Posts: n/a
 
      11-12-2007
Alan wrote:
>Thanks for the information. I think I shall just follow href
>links instead of finding directories.


In that case, as Daniel mentioned, search "Web crawler"/
"web crawling". There have been some interesting discussions
about crawlers in these groups, across the ages. As I vaguely
recall there was a source posted for one by Mr Omar Khan
..ahh yes.
<http://groups.google.com/group/comp....4a6f43d57e3e6a
>


But please (please, please) respect the directions of the
site's robots.txt (if it has one).

--
Andrew Thompson
http://www.athompson.info/andrew/

Message posted via JavaKB.com
http://www.javakb.com/Uwe/Forums.asp...neral/200711/1

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
3 ESSENTIAL TOOLS FOR STARTING AND MAINTAINING...3 ESSENTIAL TOOLSFOR STARTING AND MAINTAINING...3 ESSENTIAL TOOLS FOR STARTING ANDMAINTAINING... Oanh Bui C++ 0 04-27-2009 12:51 PM
3 ESSENTIAL TOOLS FOR STARTING AND MAINTAINING...3 ESSENTIAL TOOLSFOR STARTING AND MAINTAINING...3 ESSENTIAL TOOLS FOR STARTING ANDMAINTAINING... Oanh Bui C Programming 0 04-27-2009 12:51 PM
3 ESSENTIAL TOOLS FOR STARTING AND MAINTAINING...3 ESSENTIAL TOOLSFOR STARTING AND MAINTAINING...3 ESSENTIAL TOOLS FOR STARTING ANDMAINTAINING... Oanh Bui Python 0 04-27-2009 12:46 PM
redirect URL's, return URL's, and URL Parameters Jon paugh ASP .Net 1 07-10-2004 05:29 AM
finding subdirectories without parsing every file Helen Perl Misc 7 08-19-2003 05:02 AM



Advertisments