Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > moving data in a file without using system memory

Reply
Thread Tools

moving data in a file without using system memory

 
 
Pascal Bourguignon
Guest
Posts: n/a
 
      01-13-2006
"ulyses" <(E-Mail Removed)> writes:

> David Schwartz napisal(a):
>> "ulyses" <(E-Mail Removed)> wrote in message
>> news:(E-Mail Removed) ups.com...
>>
>> > Let's assume I have following file:
>> >
>> > 2938929384902491233.....
>> > 923949919199191919112....
>> >
>> > File contains INTs only. What is more they are huge. For example first
>> > row in file may contain integer which size is 50MB and the second 30MB.
>> > Now we come to my problem. Is there possibility to swap this rows
>> > without using system memory (preferably in Unix/Linux)? Is there any
>> > function in C to do this?

>>
>> It's not clear what you mean by "without using system memory". Why
>> wouldn't you want to use memory if this results in faster operation?
>>

>
> I thouhg about something that would be some kind of low level function
> that would modify pointers not real data, the same way as we do with
> pointer to data in memory, e.g.:
> int a*, b*, temp*;
> ...
> //the swap
> temp = a;
> a = b;
> b = temp;
>
> And the data was swapped without moving it in memory. I asked if there
> is such functionality that would enable me to do something like that
> but with rows in file. Swap them without moving them on disk.


You can build an index yourself.

Read the data file byte by byte, and note the offset of all newline
characters. Save the list of offsets to an index file.

Later you can read the index file, and when you want to read the nth
number, you seek in the data file to the nth offset.

If you want to swap two numbers, you just swap the two offsets in the
index, which involves reading and writing at most two blocks. You can
also easily "delete" or "insert" numbers. To "delete" a number you
just remove its offset in the index (or put it in a "free list" if you
want to be able to reuse the space). To "insert" a number, you just
note the file size as the offset to the new number which you merely
append to the end of the file, and insert the offset in the index.


--
__Pascal Bourguignon__ http://www.informatimago.com/

"Debugging? Klingons do not debug! Our software does not coddle the
weak."
 
Reply With Quote
 
 
 
 
Barry Margolin
Guest
Posts: n/a
 
      01-14-2006
In article <(E-Mail Removed)>,
Pascal Bourguignon <(E-Mail Removed)> wrote:

> You can build an index yourself.
>
> Read the data file byte by byte, and note the offset of all newline
> characters. Save the list of offsets to an index file.


Even better would be if you could get the program that creates the file
in the first place to create an index at the same time.

--
Barry Margolin, http://www.velocityreviews.com/forums/(E-Mail Removed)
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Moving Guide Moving Companies Movers Storage linkswanted ASP .Net 0 01-23-2008 01:08 AM
Free Moving Estimate, Local Movers, Long Distance Moving, PackingSupplies, Storage Rental, Home Moving, Apartment Moving, Office Moving,Commercial Moving linkswanted ASP .Net 0 01-06-2008 04:45 AM
Moving to California Moving to L.A linkswanted Digital Photography 3 01-05-2008 04:22 PM
movers los angeles moving storage services movers in L.A companymoving companies moving in calfornia los angeles linkswanted HTML 0 12-21-2007 10:55 PM



Advertisments