Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > BeautifulSoup and Problem Tables

Reply
Thread Tools

BeautifulSoup and Problem Tables

 
 
academicedgar@gmail.com
Guest
Posts: n/a
 
      09-21-2008
Hi

I would appreciate some help. I am trying to learn Python and want to
use BeautifulSoup to pull some data from tables. I was really psyched
earlier tonight when I discovered that I could do this

from BeautifulSoup import BeautifulSoup
bst=file(r"c:\bstest.htm").read()
soup=BeautifulSoup(bst)
rows=soup.findAll('tr')
len(rows)
a=len(rows[0].findAll('td'))
b=len(rows[1].findAll('td'))
c=len(rows[2].findAll('td'))
d=len(rows[3].findAll('td'))
e=len(rows[4].findAll('td'))
f=len(rows[5].findAll('td'))
g=len(rows[6].findAll('td'))
h=len(rows[8].findAll('td'))
i=len(rows[9].findAll('td'))
j=len(rows[10].findAll('td'))
k=rows[1].findAll('td')[1].contents[0]


So here I am chortling to myself thinking this is too easy. I know
that the data columns are in rows[0] and so I can learn some more
python to figure out how to create tuples so I can lable each data
item using the row and column headings plucked from the contents.

However, I discovered that my tables have inconsistent numbers of
rows. Even though the tables look pretty. It might be that the
column heading for the third column is "Apples" but the value for
"Apples" in the fourth row is not in the third position in the row but
the fourth.

Now I am reluctant to make any assumptions because the tables were
created inconsistently. What I mean is that in some tables if there is
no value for a row/column intersection then there is a blank line, in
other tables if there is no value for a row/column intersection then
the length of k (as above) is 0.

I have been Googling for some insight into this and I have not been
successful finding anything. I would really appreciate any suggestions
or some direction about how to better describe the problem.
 
Reply With Quote
 
 
 
 
Peter Pearson
Guest
Posts: n/a
 
      09-21-2008
On Sat, 20 Sep 2008 20:51:52 -0700 (PDT), http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
[snip]
> from BeautifulSoup import BeautifulSoup
> bst=file(r"c:\bstest.htm").read()
> soup=BeautifulSoup(bst)
> rows=soup.findAll('tr')
> len(rows)
> a=len(rows[0].findAll('td'))
> b=len(rows[1].findAll('td'))
> c=len(rows[2].findAll('td'))
> d=len(rows[3].findAll('td'))
> e=len(rows[4].findAll('td'))
> f=len(rows[5].findAll('td'))
> g=len(rows[6].findAll('td'))
> h=len(rows[8].findAll('td'))
> i=len(rows[9].findAll('td'))
> j=len(rows[10].findAll('td'))
> k=rows[1].findAll('td')[1].contents[0]

[snip]
> However, I discovered that my tables have inconsistent numbers of
> rows.

[snip]
> I have been Googling for some insight into this and I have not been
> successful finding anything. I would really appreciate any suggestions
> or some direction about how to better describe the problem.


Would it be accurate to describe the problem as wanting to
extract the contents of the cth column of the rth row of a
table in spite of various pathologies in the construction of
the table?

If so, maybe it would help to post sample HTML (trimmed to a
minimum) of the pathologies that must be handled. I gotta
confess, though, that it doesn't take many rowspans or colspans
to put this problem beyond my reach.

--
To email me, substitute nowhere->spamcop, invalid->net.
 
Reply With Quote
 
 
 
 
clurks
Guest
Posts: n/a
 
      09-22-2008
(E-Mail Removed) wrote:

> Hi
>
> I would appreciate some help. I am trying to learn Python and want to
> use BeautifulSoup to pull some data from tables. I was really psyched
> earlier tonight when I discovered that I could do this
>
> from BeautifulSoup import BeautifulSoup
> bst=file(r"c:\bstest.htm").read()
> soup=BeautifulSoup(bst)
> rows=soup.findAll('tr')
> len(rows)
> a=len(rows[0].findAll('td'))
> b=len(rows[1].findAll('td'))
> c=len(rows[2].findAll('td'))
> d=len(rows[3].findAll('td'))
> e=len(rows[4].findAll('td'))
> f=len(rows[5].findAll('td'))
> g=len(rows[6].findAll('td'))
> h=len(rows[8].findAll('td'))
> i=len(rows[9].findAll('td'))
> j=len(rows[10].findAll('td'))
> k=rows[1].findAll('td')[1].contents[0]
>
>
> So here I am chortling to myself thinking this is too easy. I know
> that the data columns are in rows[0] and so I can learn some more
> python to figure out how to create tuples so I can lable each data
> item using the row and column headings plucked from the contents.
>
> However, I discovered that my tables have inconsistent numbers of
> rows. Even though the tables look pretty. It might be that the
> column heading for the third column is "Apples" but the value for
> "Apples" in the fourth row is not in the third position in the row but
> the fourth.
>
> Now I am reluctant to make any assumptions because the tables were
> created inconsistently. What I mean is that in some tables if there is
> no value for a row/column intersection then there is a blank line, in
> other tables if there is no value for a row/column intersection then
> the length of k (as above) is 0.
>
> I have been Googling for some insight into this and I have not been
> successful finding anything. I would really appreciate any suggestions
> or some direction about how to better describe the problem.


This may help, and it may not.

One of the most useful features of BeautifulSoup is the .prettify()
function. Use it, and you can examine your xml with a browser or
decent editor. You will be able, hopefully, to see attributes that
may explain your difficulties. The one that snuck up and bit me was
table:number-columns-repeated -- I don't know what is biting you,
but it is weird the way known cells are completely skipped with this
attribute -- you have to know about it and check for it.

If there is a way to deal with that attribute using BeautifulSoup I
was unable to find it given the documentation and examples available
to me. I took a step back and tried xml.sax -- it is straighforward
and easy to use -- you build an xml.sax.handler for your xml, and it
deals with it.

It worked for me, but, as always, YMMV

sc
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
encoding problem with BeautifulSoup - problem when writing parsedtext to file Greg Python 9 10-08-2011 03:30 PM
PAMIE and beautifulsoup problem elca Python 0 10-23-2009 06:03 AM
BeautifulSoup problem placid Python 2 10-20-2006 10:38 AM
How to get database metadata information (i.e. existing tables and columns in tables) Chris Brat Python 5 08-22-2006 05:25 PM
scraping nested tables with BeautifulSoup Gonzillaaa@gmail.com Python 7 04-04-2006 05:21 PM



Advertisments