Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Removing Duplicate entries in a file...

Reply
Thread Tools

Removing Duplicate entries in a file...

 
 
sri2097
Guest
Posts: n/a
 
      01-06-2006
Hi all, I'm storing number of dictionary values into a file using the
'cPickle' module and then am retrieving it. The following is the code
for it -

# Code for storing the values in the file
import cPickle

book = {raw_input("Name: "): [int(raw_input("Phone: ")),
raw_input("Address: ")] }
file_object = file(database, 'w+')
cPickle.dump(book, file_object)
file_object.close()

# Code for retrieving values and modifiing them.
tobe_modified_name = raw_input("Enter name to be modified: ")
file_object = file(database)

while file_object.tell() != EOFError:
try:
stored_dict = cPickle.load(file_object)
if stored_dict.has_key(tobe_modified_name):
print ("Entry found !")
# I want to modify the values retrieved from the file and
then put it back to the file without duplicate entry.
file_object = file(database, 'a+')
except EOFError:
break
file_object.close()


Now, my problem is after finding the entry in the file, I want to make
changes to the 'values' under the searched 'key' and then insert it
back to the file. But in doing so I'm having duplicate entries for the
same key. I want to remove the previous key and value entry in the file
and key the latest one. How to solve this problem ?

I actually thought of 2 ways -

1) In Java there is something called 'file_pointer' concept where in
after you find the entry you are looking for you move all the entries
below this entry. Then you get the searched entry at the bottom of the
file. After this truncate the file by a certain bytes to remove the old
entry. Can we do this in Python using the file.truncate([size]) method
?

2) Although this is a really crappy way but nevertheless I'll put it
across. First after finding the entry you are looking for in the file,
make a copy of this file without the entry found in the previous file.
Make the changes to the 'values' under this key and insert this into
the second file what you have created. Before exiting delete the first
file.

Are there any more ways to solve my problem ? Any criticisms are
welcome....

 
Reply With Quote
 
 
 
 
Fuzzyman
Guest
Posts: n/a
 
      01-06-2006

sri2097 wrote:
> Hi all, I'm storing number of dictionary values into a file using the
> 'cPickle' module and then am retrieving it. The following is the code
> for it -
>
> # Code for storing the values in the file
> import cPickle
>
> book = {raw_input("Name: "): [int(raw_input("Phone: ")),
> raw_input("Address: ")] }
> file_object = file(database, 'w+')
> cPickle.dump(book, file_object)
> file_object.close()
>


I may be misunderstanding you - but it seems you just want to read a
pickle, modify it, and then write it back ?

What you're doing is appending the modified pickle to the original one
- which is more complicated than what you want to achieve.

file_object = open(filename, 'rb')
stored_dict = cPickle.load(file_object)
file_object.close()

.... code that modifies stored_dict

file_object = open(filename, 'wb')
cPickle.dump(stored_dict, file_object)
file_object.close()

Any reason why that shouldn't do what you want ?

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

 
Reply With Quote
 
 
 
 
sri2097
Guest
Posts: n/a
 
      01-06-2006
Hi there, I'm just curious to know as to how the changes you have
suggested will solve the problem. Instead of appending (what I was
doing), now we are opening and storing the files in 'binary' format.
All the other entries in my file will be gone when I write into the
file again.

What I actuall need is this -

I have some dictionary values stored in a file. I retrieve these
entries based on the key value specified by the user. Now if I want to
modify the values under a particular key, I first search if that key
exists in the file and if yes retrieve the values associated with the
key and modify them. Now when I re-insert this modified key-value pair
back in the file. I have 2 entries now (one is the old wntry and the
second is the new modified one). So if I search for that key the next
time I'll have 2 entries for it. That's not what we want. So how do I
remove the old entry without the other values getting deleted ? In
other words, keeping the other entries as it is, I want to update a
particular key-value pair.

Let me know in case any bright idea strikes...

 
Reply With Quote
 
Mike Meyer
Guest
Posts: n/a
 
      01-06-2006
"sri2097" <(E-Mail Removed)> writes:
> Hi all, I'm storing number of dictionary values into a file using the
> 'cPickle' module and then am retrieving it. The following is the code
> for it -
>
> # Code for storing the values in the file
> import cPickle
>
> book = {raw_input("Name: "): [int(raw_input("Phone: ")),
> raw_input("Address: ")] }
> file_object = file(database, 'w+')
> cPickle.dump(book, file_object)
> file_object.close()
>
> # Code for retrieving values and modifiing them.
> tobe_modified_name = raw_input("Enter name to be modified: ")
> file_object = file(database)
>
> while file_object.tell() != EOFError:
> try:
> stored_dict = cPickle.load(file_object)
> if stored_dict.has_key(tobe_modified_name):
> print ("Entry found !")
> # I want to modify the values retrieved from the file and
> then put it back to the file without duplicate entry.
> file_object = file(database, 'a+')
> except EOFError:
> break
> file_object.close()
>
>
> Now, my problem is after finding the entry in the file, I want to make
> changes to the 'values' under the searched 'key' and then insert it
> back to the file. But in doing so I'm having duplicate entries for the
> same key. I want to remove the previous key and value entry in the file
> and key the latest one. How to solve this problem ?


First, file_object.tell won't return EOFError. Nothing should return
EOFError - it's an exception. It should be raised.

As you noticed, cPickle.load will raise EOFError when called on a file
that you've reached the end of. However, you want to narrow the
try clause as much as possible:

try:
stored_dict = cPickle.load(file_object)
except EOFError:
break

# Work with stored dict here.

If you weren't doing a break in the except clause, you'd work with the
dictionary in an else clause.

> I actually thought of 2 ways -
>
> 1) In Java there is something called 'file_pointer' concept where in
> after you find the entry you are looking for you move all the entries
> below this entry. Then you get the searched entry at the bottom of the
> file. After this truncate the file by a certain bytes to remove the old
> entry. Can we do this in Python using the file.truncate([size]) method
> ?


Yup, this would work. You'd have to save the value from
file_object.tell() before calling cPickle.load, so you could go back
to that point to write the next object. You'd either have to load all
the following objects into memory, or shuttle back and forth between
the read and write positions. The latter sounds "really crappy" to me.

> 2) Although this is a really crappy way but nevertheless I'll put it
> across. First after finding the entry you are looking for in the file,
> make a copy of this file without the entry found in the previous file.
> Make the changes to the 'values' under this key and insert this into
> the second file what you have created. Before exiting delete the first
> file.


Actually, there's a good reason for doing it that way. But first,
another alternative.

Unless your file is huge (more than a few hundred megabytes), you
might consider loading the entire thing into memory. Instead of
calling cPickle.dump multiple times, put all the dictionaries in a
list, then call cPickle.dump on the list. When you want to update the
list, cPickle.load will load the entire list, so you can use Python to
work on it.

As for saving the file, best practice for updating a file is to write
it to a temporary file, and then rename the new file to the old name
after the write has successfully finished. This way, if the write
fails for some reason, your working file isn't corrupted. Doing it
this way also makes dealing with the case of the the list being to big
load into memory easy:

# Warning, untested code

while 1:
try:
stored_dict = cPickle.load(input_file)
except EOFError:
break
if stored_dict.has_key(tobe_modified_name):
print "Entry found !"
# Modify stored_dict here
cPickle.dump(stored_dict, output_file)

output_file.close()
os.unlink(database) # May not be required; depends on your os
os.rename(datebase_temp, database)


You'll probably want to handle exceptions from cPickle.dump and
output_file.close cleanly as well.

<mike
--
Mike Meyer <(E-Mail Removed)> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
 
Reply With Quote
 
sri2097
Guest
Posts: n/a
 
      01-10-2006
Thanx Mike, My problem solved !! I loaded the entire file contnets into
list and my job was a piece of cake after that.

Srikar

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Perl Format the Output in table,by removing duplicate entries ppnair Perl Misc 0 10-11-2012 12:43 PM
Removing duplicate entries/stories from a RSS feed? gaikokujinkyofusho@gmail.com HTML 2 12-06-2006 07:46 PM
Removing duplicate entries/stories from a RSS feed? gaikokujinkyofusho@gmail.com XML 1 12-06-2006 06:25 PM
removing duplicate entries in a list. Amit C++ 3 09-05-2005 11:16 AM
Tying up Port Login table entries with Port Table Entries in CISCO SNMP John Ramsden Cisco 0 07-24-2004 04:03 PM



Advertisments