Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > how come i cant grab all rss items from a feed

Reply
Thread Tools

how come i cant grab all rss items from a feed

 
 
Adam Akhtar
Guest
Posts: n/a
 
      04-04-2009
When i try and access the bbc feed it will only return the latest 40
results. However if i use the same feed in google reader, it can return
a lot more. Why is this and how do i modify the code below so that i can
return more results.

Heres some code

require 'rss/1.0'
require 'rss/2.0'
require 'open-uri'
require 'rss/parser'


source =
"http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/rss.xml"
# url or local file
content = "" # raw content of rss feed will be loaded here
open(source) do |s| content = s.read end
rss = RSS:arser.parse(content, false)

(rss.items.length == 40ish)
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Ben Lovell
Guest
Posts: n/a
 
      04-04-2009
[Note: parts of this message were removed to make it a legal post.]

I'd suggest you check which headers your browser is sending. Particularly
the Last-Modified and Etag. Replicate those and you will see the same amount
of items.
Ben

On Sat, Apr 4, 2009 at 11:28 AM, Adam Akhtar <(E-Mail Removed)>wrote:

> When i try and access the bbc feed it will only return the latest 40
> results. However if i use the same feed in google reader, it can return
> a lot more. Why is this and how do i modify the code below so that i can
> return more results.
>
> Heres some code
>
> require 'rss/1.0'
> require 'rss/2.0'
> require 'open-uri'
> require 'rss/parser'
>
>
> source =
> "http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/rss.xml"
> # url or local file
> content = "" # raw content of rss feed will be loaded here
> open(source) do |s| content = s.read end
> rss = RSS:arser.parse(content, false)
>
> (rss.items.length == 40ish)
> --
> Posted via http://www.ruby-forum.com/.
>
>


 
Reply With Quote
 
 
 
 
Adam Akhtar
Guest
Posts: n/a
 
      04-04-2009
hi ben, thanks for the reply. Im a bit confused though.

When i type the address into the browser it shows 40 items just as the
code above does.

When i access the feed via google reader it shows more. I wondering what
magic google is using to get more items than my browser or my rss code
can.

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Snaury Miyoto
Guest
Posts: n/a
 
      04-04-2009
Adam Akhtar wrote:
> When i access the feed via google reader it shows more. I wondering what
> magic google is using to get more items than my browser or my rss code
> can.


I think this could be because google caches items internally, from the
moment someone subscribes to the feed for the first time. Then it just
periodically updates the feed and caches new items. To simulate google
behavior you would need to go back in time and start fetching items from
the moment you need.

Unless there's some api on the feed itself, in which case you'd have to
ask owners of the feed.
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Ben Lovell
Guest
Posts: n/a
 
      04-04-2009
[Note: parts of this message were removed to make it a legal post.]

>
>
> When i access the feed via google reader it shows more. I wondering what
> magic google is using to get more items than my browser or my rss code
> can.
>
>

Ah I have to admit I glossed over the fact you were using Google Reader. As
Snaury said, they would certainly employ caching at their end which should
display results beyond the ones you are seeing.

Ben

 
Reply With Quote
 
Adam Akhtar
Guest
Posts: n/a
 
      04-05-2009
ahhh thank you for confirming my suspicions. Customized Deloreans with
flux capacitors are a bit hard to come by these days so I guess ill have
to settle with 40 a day.

Thank you for your help!

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Ben Lovell
Guest
Posts: n/a
 
      04-05-2009
[Note: parts of this message were removed to make it a legal post.]

If it is utterly important you _could_ possibly use the google reader api to
get at those entries. Might be a little easier than going back in time

On Sun, Apr 5, 2009 at 4:16 AM, Adam Akhtar <(E-Mail Removed)> wrote:

> ahhh thank you for confirming my suspicions. Customized Deloreans with
> flux capacitors are a bit hard to come by these days so I guess ill have
> to settle with 40 a day.
>
> Thank you for your help!
>
> --
> Posted via http://www.ruby-forum.com/.
>
>


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Two ways to generate RSS - rss/maker and rss/2.0 - which is better? Jonathan Groll Ruby 1 06-27-2009 03:53 AM
Post RSS feed w/o RSS-to-Javascript.com Scott Gordo HTML 5 08-29-2006 01:34 AM
RSS Feed - need an Idiot's Guide to RSS News on my website teach_me6@hotmail.com HTML 5 02-25-2005 11:01 AM
Searches in multiple RSS feeds -> new rss feed Motta XML 1 06-09-2004 10:55 PM



Advertisments