Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > BeautifulSoup

Reply
Thread Tools

BeautifulSoup

 
 
Steve Young
Guest
Posts: n/a
 
      08-19-2005
I tried using BeautifulSoup to make changes to the url
links on html pages, but when the page was displayed,
it was garbled up and didn't look right (even when I
didn't actually change anything on the page yet). I
ran these steps in python to see what was up:

>>from BeautifulSoup import BeautifulSoup
>>from urllib2 import build_opener, Request
>>
>>req = Request('http://www.python.org/')
>>f = build_opener().open(req)
>>page = f.read()
>>f.close()
>>
>>len(page)

12040
>>
>>soup = BeautifulSoup()
>>soup.feed(page)
>>page2 = soup.renderContents()
>>len(page2)

11889

I have version 2.1 of BeautifulSoup. It seems that
other ppl have used BeautifulSoup and it works fine
for them so I'm not sure what I'm doing wrong. Any
help would be appreciated, thanks.

-Steve



__________________________________________________ __
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs

 
Reply With Quote
 
 
 
 
Paul McGuire
Guest
Posts: n/a
 
      08-19-2005
Steve -

Is there a chance you could post a before and after example, so we can
see just what you are trying to do instead of talking conceptually all
around it and making us guess? If you are just doing some spot
translations of specific values in an HTML file, you can probably get
away with a simple (but readable) program using pyparsing.

-- Paul

 
Reply With Quote
 
 
 
 
Paul McGuire
Guest
Posts: n/a
 
      08-19-2005
Here's a pyparsing program that reads my personal web page, and spits
out HTML with all of the HREF's reversed.

-- Paul
(Download pyparsing at http://pyparsing.sourceforge.net.)



from pyparsing import Literal, quotedString
import urllib

LT = Literal("<")
GT = Literal(">")
EQUALS = Literal("=")
htmlAnchor = LT + "A" + "HREF" + EQUALS +
quotedString.setResultsName("href") + GT

def convertHREF(s,l,toks):
# do HREF conversion here - for demonstration, we will just reverse
them
print toks.href
return "<A HREF=%s>" % toks.href[::-1]

htmlAnchor.setParseAction( convertHREF )

inputURL = "http://www.geocities.com/ptmcg"
inputPage = urllib.urlopen(inputURL)
inputHTML = inputPage.read()
inputPage.close()

print htmlAnchor.transformString( inputHTML )

 
Reply With Quote
 
Mike Meyer
Guest
Posts: n/a
 
      08-20-2005
"Paul McGuire" <(E-Mail Removed)> writes:

> Here's a pyparsing program that reads my personal web page, and spits
> out HTML with all of the HREF's reversed.


Parsing HTML isn't easy, which makes me wonder how good this solution
really is. Not meant as a comment on the quality of this code or
PyParsing, but as curiosity from someone who does a lot of [X}HTML
herding.

> -- Paul
> (Download pyparsing at http://pyparsing.sourceforge.net.)


If it were in the ports tree, I'd have grabbed it and tried it
myself. But it isn't, so I'm going to be lazy and ask. If PyParsing
really makes dealing with HTML this easy, I may package it as a port
myself.

> from pyparsing import Literal, quotedString
> import urllib
>
> LT = Literal("<")
> GT = Literal(">")
> EQUALS = Literal("=")
> htmlAnchor = LT + "A" + "HREF" + EQUALS +
> quotedString.setResultsName("href") + GT
>
> def convertHREF(s,l,toks):
> # do HREF conversion here - for demonstration, we will just reverse
> them
> print toks.href
> return "<A HREF=%s>" % toks.href[::-1]
>
> htmlAnchor.setParseAction( convertHREF )
>
> inputURL = "http://www.geocities.com/ptmcg"
> inputPage = urllib.urlopen(inputURL)
> inputHTML = inputPage.read()
> inputPage.close()
>
> print htmlAnchor.transformString( inputHTML )


How well does it deal with other attributes in front of the href, like
<A onClick="..." href="...">?

How about if my HTML has things that look like HTML in attributes,
like <TAG ATTRIBUTE="stuff<A HREF=stuff">?

Thanks,
<mike
--
Mike Meyer <(E-Mail Removed)> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
 
Reply With Quote
 
Paul McGuire
Guest
Posts: n/a
 
      08-20-2005
Mike -

Thanks for asking. Typically I hang back from these discussions of
parsing HTML or XML (*especially* XML), since there are already a
number of parsers out there that can handle the full language syntax.
But it seems that many people trying to parse HTML aren't interested in
fully parsing an HTML page, so much as they are trying to match some
tag pattern, to extract or modify the embedded data. In these cases,
fully comprehending HTML syntax is rarely required.

In this particular instance, the OP had started another thread in which
he was trying to extract some HTML content using regexp's, and this
didn't seem to be converging to a workable solution. When he finally
revealed that what he was trying to do was extract and modify the URL's
in a web pages HTML source, this seemed like a tractable problem for a
quick pyparsing program. In the interests of keeping things simple, I
admittedly provided a limited solution. As you mentioned, no
additional attributes are handled by this code. But many HTML scrapers
are able to make simplifying assumptions about what HTML features can
be expected, and I did not want to spend a lot of time solving problems
that may never come up.

So you asked some good questions, let me try to give some reasonable
answers, or at least responses:

1. "If it were in the ports tree, I'd have grabbed it and tried it
myself."
By "ports tree", I assume you mean some directory of your Linux
distribution. I'm sure my Linux ignorance is showing here, most of my
work occurs on Windows systems. I've had pyparsing available on SF for
over a year and a half, and I do know that it has been incorporated (by
others) into a couple of Linux distros, including Debian, ubuntu,
gentoo, and Fedora. If you are interested in doing a port to another
Linux, that would be great! But I was hoping that hosting pyparsing on
SF would be easy enough for most people to be able to get at it.

2. "How well does it deal with other attributes in front of the href,
like <A onClick="..." href="...">?"
*This* version doesn't deal with other attributes at all, in the
interests of simplicity. However, pyparsing includes a helper method,
makeHTMLTags(), that *does* support arbitrary attributes within an
opening HTML tag. It is called like:

anchorStart,anchorEnd = makeHTMLTags("A")

makeHTMLTags returns a pyparsing subexpression that *does* comprehend
attributes, as well as opening tags that include their own closing '/'
(indicating an empty tag body). Tag attributes are accessible by name
in the returned results tokens, without requiring setResultsName()
calls (as in the example).

3. "How about if my HTML has things that look like HTML in attributes,
like <TAG ATTRIBUTE="stuff<A HREF=stuff">?"
Well, again, the simple example wont be able to tell the difference,
and it would process the ATTRIBUTE string as a real tag. To address
this, we would expand our statement to process quoted strings
explicitly, and separately from the htmlAnchor, as in:

htmlPatterns = quotedString | htmlAnchor

and then use htmlPatterns for the transformString call:

htmlPatterns.transformString( inputHTML )

You didn't ask, but one feature that is easy to handle is comments.
pyparsing includes some common comment syntaxes, such as cStyleComment
and htmlComment. To ignore them, one simply calls ignore() on the root
pyparsing node. In the simple example, this would look like:

htmlPatterns.ignore( htmlComment )

By adding this single statement, all HTML comments would be ignored.


Writing a full HTML parser with pyparsing would be tedious, and not a
great way to spend your time, given the availability of other parsing
tools. But for simple scraping and extracting, it can be a very
efficient way to go.

-- Paul

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
BeautifulSoup error William Xu Python 4 06-16-2006 06:34 PM
scraping nested tables with BeautifulSoup Gonzillaaa@gmail.com Python 7 04-04-2006 05:21 PM
how to run BeautifulSoup in Jython ye juan Python 1 02-05-2006 09:28 AM
BeautifulSoup fetch help ted Python 2 01-07-2006 06:11 AM
HTML purifier using BeautifulSoup? Dan Stromberg Python 1 01-07-2005 06:10 PM



Advertisments