Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > XML > Removing duplicate entries/stories from a RSS feed?

Reply
Thread Tools

Removing duplicate entries/stories from a RSS feed?

 
 
gaikokujinkyofusho@gmail.com
Guest
Posts: n/a
 
      12-06-2006
Hi, I have been enjoying being able to subscribe to RSS
(http://kinja.com/user/thedigestibleaggie) for awhile and have come up
with a fairly nice list of feeds but I have run into an annoying
(though not critical) problem, duplicate stories. Apparently there is
overlap with some of the sites I subscribe to so I get duplicate
stories. Does anyone know of some sort of filter (software or online
service) that can remove duplicate stories? Any help or suggestions
would really be appreciated!

Cheers

-Gaiko

 
Reply With Quote
 
 
 
 
Paul Lutus
Guest
Posts: n/a
 
      12-06-2006
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> Hi, I have been enjoying being able to subscribe to RSS
> (http://kinja.com/user/thedigestibleaggie) for awhile and have come up
> with a fairly nice list of feeds but I have run into an annoying
> (though not critical) problem, duplicate stories. Apparently there is
> overlap with some of the sites I subscribe to so I get duplicate
> stories. Does anyone know of some sort of filter (software or online
> service) that can remove duplicate stories? Any help or suggestions
> would really be appreciated!


Write a script in a language that supports associative arrays (as do Java,
Perl, Ruby, Python, and even JavaScript). Key the associative array to a
unique key created out of elements in the various RSS feed items. Fill the
associative array using the generated key.

Unfortunately, it is rare for two RSS feed items to be truly identical.
Often, they tell the same story with small differences in wording (to avoid
accusations of plagiarism) and of course the URL is normally different.

Without some complex coding to detect items that are almost the same, the
above method will remove only genuinely identical items from different RSS
feeds.

--
Paul Lutus
http://www.arachnoid.com
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Two ways to generate RSS - rss/maker and rss/2.0 - which is better? Jonathan Groll Ruby 1 06-27-2009 03:53 AM
Removing duplicate entries/stories from a RSS feed? gaikokujinkyofusho@gmail.com HTML 2 12-06-2006 07:46 PM
is RSS 2.0 still RSS 2.0 if we add our own unique tags to it? Jake Barnes XML 1 11-14-2005 01:54 AM
RSS Feed - need an Idiot's Guide to RSS News on my website teach_me6@hotmail.com HTML 5 02-25-2005 11:01 AM
Searches in multiple RSS feeds -> new rss feed Motta XML 1 06-09-2004 10:55 PM



Advertisments