Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > BeautifulSoup vs. loose & chars

Reply
Thread Tools

BeautifulSoup vs. loose & chars

 
 
John Nagle
Guest
Posts: n/a
 
      12-26-2006
I've been parsing existing HTML with BeautifulSoup, and occasionally
hit content which has something like "Design & Advertising", that is,
an "&" instead of an "&". Is there some way I can get BeautifulSoup
to clean those up? There are various parsing options related to "&"
handling, but none of them seem to do quite the right thing.

If I write the BeautifulSoup parse tree back out with "prettify",
the loose "&" is still in there. So the output is
rejected by XML parsers. Which is why this is a problem.
I need valid XML out, even if what went in wasn't quite valid.

John Nagle
 
Reply With Quote
 
 
 
 
placid
Guest
Posts: n/a
 
      12-26-2006

John Nagle wrote:
> I've been parsing existing HTML with BeautifulSoup, and occasionally
> hit content which has something like "Design & Advertising", that is,
> an "&" instead of an "&". Is there some way I can get BeautifulSoup
> to clean those up? There are various parsing options related to "&"
> handling, but none of them seem to do quite the right thing.
>
> If I write the BeautifulSoup parse tree back out with "prettify",
> the loose "&" is still in there. So the output is
> rejected by XML parsers. Which is why this is a problem.
> I need valid XML out, even if what went in wasn't quite valid.
>
> John Nagle



So do you want to remove "&" or replace them with "&" ? If you want
to replace it try the following;

import urllib, sys

try:
location = urllib.urlopen(url)
except IOError, (errno, strerror):
sys.exit("I/O error(%s): %s" % (errno, strerror))

content = location.read()
content = content.replace("&", "&")


To do this with BeautifulSoup, i think you need to go through every
Tag, get its content, see if it contains an "&" and then replace the
Tag with the same Tag but the content contains "&"

Hope this helps.
Cheers

 
Reply With Quote
 
 
 
 
Felipe Almeida Lessa
Guest
Posts: n/a
 
      12-26-2006
On 26 Dec 2006 04:22:38 -0800, placid <(E-Mail Removed)> wrote:
> So do you want to remove "&" or replace them with "&amp;" ? If you want
> to replace it try the following;


I think he wants to replace them, but just the invalid ones. I.e.,

This & this &amp; that

would become

This &amp; this &amp; that


No, i don't know how to do this efficiently. =/...
I think some kind of regex could do it.

--
Felipe.
 
Reply With Quote
 
Duncan Booth
Guest
Posts: n/a
 
      12-26-2006
"Felipe Almeida Lessa" <(E-Mail Removed)> wrote:

> On 26 Dec 2006 04:22:38 -0800, placid <(E-Mail Removed)> wrote:
>> So do you want to remove "&" or replace them with "&amp;" ? If you
>> want to replace it try the following;

>
> I think he wants to replace them, but just the invalid ones. I.e.,
>
> This & this &amp; that
>
> would become
>
> This &amp; this &amp; that
>
>
> No, i don't know how to do this efficiently. =/...
> I think some kind of regex could do it.
>


Since he's asking for valid xml as output, it isn't sufficient just to
ignore entity definitions: HTML has a lot of named entities such as
&nbsp; but xml only has a very limited set of predefined named entities.
The safest technique is to convert them all to numeric escapes except
for the very limited set also guaranteed to be available in xml.

Try this:

from cgi import escape
import re
from htmlentitydefs import name2codepoint
name2codepoint = name2codepoint.copy()
name2codepoint['apos']=ord("'")

EntityPattern =
re.compile('&(?:#(\d+)|(?:#x([\da-fA-F]+))|([a-zA-Z]+));')

def decodeEntities(s, encoding='utf-8'):
def unescape(match):
code = match.group(1)
if code:
return unichr(int(code, 10))
else:
code = match.group(2)
if code:
return unichr(int(code, 16))
else:
return unichr(name2codepoint[match.group(3)])
return EntityPattern.sub(unescape, s)

>>> escape(

decodeEntities("This & this &amp; that&nbsp;&eacute;")).encode(
'ascii', 'xmlcharrefreplace')
'This &amp; this &amp; that é'


P.S. apos is handled specially as it isn't technically a
valid html entity (and Python doesn't include it in its entity
list), but it is an xml entity and recognised by many browsers so some
people might use it in html.

 
Reply With Quote
 
Andreas Lysdal
Guest
Posts: n/a
 
      12-26-2006


Duncan Booth skrev:
> "Felipe Almeida Lessa" <(E-Mail Removed)> wrote:
>
>
>> On 26 Dec 2006 04:22:38 -0800, placid <(E-Mail Removed)> wrote:
>>
>>> So do you want to remove "&" or replace them with "&amp;" ? If you
>>> want to replace it try the following;
>>>

>> I think he wants to replace them, but just the invalid ones. I.e.,
>>
>> This & this &amp; that
>>
>> would become
>>
>> This &amp; this &amp; that
>>
>>
>> No, i don't know how to do this efficiently. =/...
>> I think some kind of regex could do it.
>>
>>

>
> Since he's asking for valid xml as output, it isn't sufficient just to
> ignore entity definitions: HTML has a lot of named entities such as
> &nbsp; but xml only has a very limited set of predefined named entities.
> The safest technique is to convert them all to numeric escapes except
> for the very limited set also guaranteed to be available in xml.
>
> Try this:
>
> from cgi import escape
> import re
> from htmlentitydefs import name2codepoint
> name2codepoint = name2codepoint.copy()
> name2codepoint['apos']=ord("'")
>
> EntityPattern =
> re.compile('&(?:#(\d+)|(?:#x([\da-fA-F]+))|([a-zA-Z]+));')
>
> def decodeEntities(s, encoding='utf-8'):
> def unescape(match):
> code = match.group(1)
> if code:
> return unichr(int(code, 10))
> else:
> code = match.group(2)
> if code:
> return unichr(int(code, 16))
> else:
> return unichr(name2codepoint[match.group(3)])
> return EntityPattern.sub(unescape, s)
>
>
>>>> escape(
>>>>

> decodeEntities("This & this &amp; that&nbsp;&eacute;")).encode(
> 'ascii', 'xmlcharrefreplace')
> 'This &amp; this &amp; that é'
>
>
> P.S. apos is handled specially as it isn't technically a
> valid html entity (and Python doesn't include it in its entity
> list), but it is an xml entity and recognised by many browsers so some
> people might use it in html.
>

Hey i fund this site:
http://www.htmlhelp.com/reference/ht...s/symbols.html

I hope that its what you mean.

/Scripter47





 
Reply With Quote
 
John Nagle
Guest
Posts: n/a
 
      12-26-2006
Felipe Almeida Lessa wrote:
> On 26 Dec 2006 04:22:38 -0800, placid <(E-Mail Removed)> wrote:
>
>> So do you want to remove "&" or replace them with "&amp;" ? If you want
>> to replace it try the following;

>
>
> I think he wants to replace them, but just the invalid ones. I.e.,
>
> This & this &amp; that
>
> would become
>
> This &amp; this &amp; that
>
>
> No, i don't know how to do this efficiently. =/...
> I think some kind of regex could do it.


Yes, and the appropriate one is:

krefindamp = re.compile(r'&(?!(\w|#)+')
...
xmlsection = re.sub(krefindamp,'&amp;',xmlsection)

This will replace an '&' with '&amp' if the '&' isn't
immediately followed by some combination of letters, numbers,
and '#' ending with a ';' Admittedly this would let something
like '&xx#2;', which isn't a legal entity, through unmodified.

There's still a potential problem with unknown entities in the output XML, but
at least they're recognized as entities.

John Nagle


 
Reply With Quote
 
Duncan Booth
Guest
Posts: n/a
 
      12-26-2006
Andreas Lysdal <(E-Mail Removed)> wrote:

>> P.S. apos is handled specially as it isn't technically a
>> valid html entity (and Python doesn't include it in its entity
>> list), but it is an xml entity and recognised by many browsers so some
>> people might use it in html.
>>

> Hey i fund this site:
> http://www.htmlhelp.com/reference/ht...s/symbols.html
>
> I hope that its what you mean.


Try
http://www.w3.org/TR/html4/sgml/entities.html#entities
for a more complete list.
 
Reply With Quote
 
Frederic Rentsch
Guest
Posts: n/a
 
      12-26-2006
John Nagle wrote:
> Felipe Almeida Lessa wrote:
>
>> On 26 Dec 2006 04:22:38 -0800, placid <(E-Mail Removed)> wrote:
>>
>>
>>> So do you want to remove "&" or replace them with "&amp;" ? If you want
>>> to replace it try the following;
>>>

>> I think he wants to replace them, but just the invalid ones. I.e.,
>>
>> This & this &amp; that
>>
>> would become
>>
>> This &amp; this &amp; that
>>
>>
>> No, i don't know how to do this efficiently. =/...
>> I think some kind of regex could do it.
>>

>
> Yes, and the appropriate one is:
>
> krefindamp = re.compile(r'&(?!(\w|#)+')
> ...
> xmlsection = re.sub(krefindamp,'&amp;',xmlsection)
>
> This will replace an '&' with '&amp' if the '&' isn't
> immediately followed by some combination of letters, numbers,
> and '#' ending with a ';' Admittedly this would let something
> like '&xx#2;', which isn't a legal entity, through unmodified.
>
> There's still a potential problem with unknown entities in the output XML, but
> at least they're recognized as entities.
>
> John Nagle
>
>
>


Here's another idea:

>>> s = '''<html> htm tag should not translate

> & should be &amp;
> &xx#2; isn't a legal entity and should translate
> { is a legal entity and should not translate

</html>

>>> import SE # http://cheeseshop.python.org/pypi/SE/2.3
>>> HTM_Escapes = SE.SE (definitions) # See definitions below the

dotted line

>>> print HTM_Escapes (s)

<html> htm tag should not translate
&gt; &amp; should be &amp;
&gt; &amp;xx#2; isn&quot;t a legal entity and should translate
&gt; { is a legal entity and should not translate
</html>

Regards

Frederic


------------------------------------------------------------------------------


definitions = '''

# Do # Don't do
# " =&nbsp;" &nbsp;== # 32 20
(34)=&dquot; &dquot;== # 34 22
&=&amp; &amp;== # 38 26
'=&quot; &quot;== # 39 27
<=&lt; &lt;== # 60 3c
>=&gt; &gt;== # 62 3e

=&copy; &copy;== # 169 a9
=&middot; &middot;== # 183 b7
=&raquo; &raquo;== # 187 bb
=&Agrave; &Agrave;== # 192 c0
=&Aacute; &Aacute;== # 193 c1
=&Acirc; &Acirc;== # 194 c2
=&Atilde; &Atilde;== # 195 c3
=&Auml; &Auml;== # 196 c4
=&Aring; &Aring;== # 197 c5
=&AElig; &AElig;== # 198 c6
=&Ccedil; &Ccedil;== # 199 c7
=&Egrave; &Egrave;== # 200 c8
=&Eacute; &Eacute;== # 201 c9
=&Ecirc; &Ecirc;== # 202 ca
=&Euml; &Euml;== # 203 cb
=&Igrave; &Igrave;== # 204 cc
=&Iacute; &Iacute;== # 205 cd
=&Icirc; &Icirc;== # 206 ce
=&Iuml; &Iuml;== # 207 cf
=&Eth; &Eth;== # 208 d0
=&Ntilde; &Ntilde;== # 209 d1
=&Ograve; &Ograve;== # 210 d2
=&Oacute; &Oacute;== # 211 d3
=&Ocirc; &Ocirc;== # 212 d4
=&Otilde; &Otilde;== # 213 d5
=&Ouml; &Ouml;== # 214 d6
=&Oslash; &Oslash;== # 216 d8
=&Ugrve; &Ugrve;== # 217 d9
=&Uacute; &Uacute;== # 218 da
=&Ucirc; &Ucirc;== # 219 db
=&Uuml; &Uuml;== # 220 dc
=&Yacute; &Yacute;== # 221 dd
=&Thorn; &Thorn;== # 222 de
=&szlig; &szlig;== # 223 df
=&agrave; &agrave;== # 224 e0
=&aacute; &aacute;== # 225 e1
=&acirc; &acirc;== # 226 e2
=&atilde; &atilde;== # 227 e3
=&auml; &auml;== # 228 e4
=&aring; &aring;== # 229 e5
=&aelig; &aelig;== # 230 e6
=&ccedil; &ccedil;== # 231 e7
=&egrave; &egrave;== # 232 e8
=&eacute; &eacute;== # 233 e9
=&ecirc; &ecirc;== # 234 ea
=&euml; &euml;== # 235 eb
=&igrave; &igrave;== # 236 ec
=&iacute; &iacute;== # 237 ed
=&icirc; &icirc;== # 238 ee
=&iuml; &iuml;== # 239 ef
=&eth; &eth;== # 240 f0
=&ntilde; &ntilde;== # 241 f1
=&ograve; &ograve;== # 242 f2
=&oacute; &oacute;== # 243 f3
=&ocric; &ocric;== # 244 f4
=&otilde; &otilde;== # 245 f5
=&ouml; &ouml;== # 246 f6
=&oslash; &oslash;== # 248 f8
=&ugrave; &ugrave;== # 249 f9
=&uacute; &uacute;== # 250 fa
=&ucirc; &ucirc;== # 251 fb
=&uuml; &uuml;== # 252 fc
=&yacute; &yacute;== # 253 fd
=&thorn; &thorn;== # 254 fe
(xff)=ÿ # 255 ff
&#== # All numeric codes
"~<(.|\n)*?>~==" # All HTM tags '''

If the ampersand is all you need to handle you can erase the others
in the first column. You need to keep the second column though, except
the last entry, because the tags don't need protection if '<' and
'>' in the first column are gone.
Definitions are easily edited and can be kept in text files.
The SE constructor accepts a file name instead of a definitions string:

>>> HTM_Escapes = SE.SE ('definition_file_name')



-------------------------------------------------------------------

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to truncate char string fromt beginning and replace chars instring by other chars in C or C++? Hongyu C++ 9 08-08-2008 12:18 PM
Floats to chars and chars to floats Kosio C Programming 44 09-23-2005 09:49 AM
BeautifulSoup Steve Young Python 4 08-20-2005 06:47 AM
HTML purifier using BeautifulSoup? Dan Stromberg Python 1 01-07-2005 06:10 PM
receiving ??? chars instead of "special" chars M.Posseth ASP .Net Web Services 3 11-16-2004 07:00 PM



Advertisments