Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > XML > Java + DOM + extracting text from XHTML

Reply
Thread Tools

Java + DOM + extracting text from XHTML

 
 
Damo
Guest
Posts: n/a
 
      01-30-2007
I have a program, That retrieves a webpage , such as a search engine
results page from the web, Then I need to go through the document and
retrieve just the search results. The problem is I want to omit the
sponsored results. So is there a way I can start analysing the
document at a specific point (ie after the sponsored results)
Thanks

 
Reply With Quote
 
 
 
 
Joseph Kesselman
Guest
Posts: n/a
 
      01-30-2007
Damo wrote:
> I have a program, That retrieves a webpage , such as a search engine
> results page from the web, Then I need to go through the document and
> retrieve just the search results. The problem is I want to omit the
> sponsored results. So is there a way I can start analysing the
> document at a specific point (ie after the sponsored results)


If you can describe what that specific point is, you should be able to
write code that navigates through the document to get to that point, and
then do whatever processing you need. If you want a more detailed
answer, you'll have to ask a more specific question...


--
Joe Kesselman / Beware the fury of a patient man. -- John Dryden
 
Reply With Quote
 
 
 
 
Peter Flynn
Guest
Posts: n/a
 
      01-30-2007
Damo wrote:
> I have a program, That retrieves a webpage , such as a search engine
> results page from the web, Then I need to go through the document and
> retrieve just the search results. The problem is I want to omit the
> sponsored results. So is there a way I can start analysing the
> document at a specific point (ie after the sponsored results)


Pass the page through HTML Tidy so that it becomes well-formed XHTML.
Inspect it manually in an XML editor and find where the bits you want to
keep start. Use some kind of XPath inspector (standalone or built into
your editor) to derive a reliably unique XPath statement that will
return the bits you want. Write some XSLT code to implement this and
transform the result to the kind of output you want. Once you've done
this once, you can automate it and it will continue until some
dipsh^H^H^H^H^Hbright spark the far end decides to change the HTML of
the original page, at which point it will all break down and you'll have
to repeat the process. Organisations who don't like their pages being
sucked clean of information sometimes have random variations built into
their server to make it unpredictable where the information will appear
in the document structure, even though it appears unchanged in the
browser. Good luck

Alternative: ask them if they provide an XML web service API which you
can use (probably for payment) to deliver the information in a
display-neutral form.

Warning: vacuuming pages from a web site too frequently will get your IP
address-block blacklisted.

///Peter
--
XML FAQ: http://xml.silmaril.ie
 
Reply With Quote
 
Joseph Kesselman
Guest
Posts: n/a
 
      01-31-2007
Peter Flynn wrote:
> Pass the page through HTML Tidy so that it becomes well-formed XHTML.


Or use an HTML-to-XML-API parser, such as NekoHTML (part of the Xerces
family). Though Tidy has the advantage that, like a parser, it will
attempt to guess what completely bogus/broken/atrocious HTML was
intended to mean; I think NekoHTML is intended mostly for HTML that is
at least vaguely reasonable.

As Damo said, and as I hinted: If you're doing this as a personal tool,
and are willing to continue to maintain it every time the folks running
the search engine break it, you can probably make this work well enough.
If you're doing it as a business tool, whoever runs that search engine
is going to work very hard to shut you down unless you've contracted
with them -- and if you've got a contract, you can probably pay them for
an XML interface for the search that doesn't include the advertising,
avoiding the whole problem.

Remember, search results are their product. They're putting a lot of
money into the software, machines, and network resources, and they're
providing the service to noncommercial users for no fee. They really are
entitled to make a a fair profit on the commercial users and/or those
who aren't willing to look at advertising.

--
Joe Kesselman / Beware the fury of a patient man. -- John Dryden
 
Reply With Quote
 
Joseph Kesselman
Guest
Posts: n/a
 
      01-31-2007
Peter Flynn wrote:
> Pass the page through HTML Tidy so that it becomes well-formed XHTML.


Or use an HTML-to-XML-API parser, such as NekoHTML (part of the Xerces
family). Though Tidy has the advantage that, like a parser, it will
attempt to guess what completely bogus/broken/atrocious HTML was
intended to mean; I think NekoHTML is intended mostly for HTML that is
at least vaguely reasonable.

As Peter said, and as I hinted: If you're doing this as a personal tool,
and are willing to continue to maintain it every time the folks running
the search engine break it, you can probably make this work well enough.
If you're doing it as a business tool, whoever runs that search engine
is going to work very hard to shut you down unless you've contracted
with them -- and if you've got a contract, you can probably pay them for
an XML interface for the search that doesn't include the advertising,
avoiding the whole problem.

Remember, search results are their product. They're putting a lot of
money into the software, machines, and network resources, and they're
providing the service to noncommercial users for no fee. They really are
entitled to make a a fair profit on the commercial users and/or those
who aren't willing to look at advertising.

--
Joe Kesselman / Beware the fury of a patient man. -- John Dryden
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
XHTML Orphaned text representation in a DOM Mike XML 4 08-01-2007 11:57 AM
Ignoring tags when extracting data from xhtml Damo_Suzuki Java 1 12-08-2006 01:37 AM
parse URL (href) from xhtml, xhtml -> text, for data hawat.thufir@gmail.com XML 7 02-08-2006 07:39 PM
XML DOM: XML/XHTML inside a text node noahlt@gmail.com Python 5 11-04-2005 07:45 PM
Extracting the XHTML "lang" parameter. lucanos@gmail.com Javascript 1 09-21-2005 04:18 PM



Advertisments