Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > how to save a file to memory rather to disk?

Reply
Thread Tools

how to save a file to memory rather to disk?

 
 
zl2k
Guest
Posts: n/a
 
      03-28-2011
hi, there,

Is there any way that I can save a file (a few k) in to the memory
than on disk (and then delete it)? Thanks for help.

zl2k
 
Reply With Quote
 
 
 
 
Victor Bazarov
Guest
Posts: n/a
 
      03-28-2011
On 3/28/2011 11:04 AM, zl2k wrote:
> hi, there,
>
> Is there any way that I can save a file (a few k) in to the memory
> than on disk (and then delete it)? Thanks for help.


What do you mean by "save a file"? You can have an ostringstream and
write to it. Read up on string-based streams.

V
--
I do not respond to top-posted replies, please don't ask
 
Reply With Quote
 
 
 
 
zl2k
Guest
Posts: n/a
 
      03-28-2011
On Mar 28, 10:21*am, cg_chas <(E-Mail Removed)> wrote:
> On Mon, 28 Mar 2011 08:04:48 -0700 (PDT), zl2k <(E-Mail Removed)> wrote:
> >hi, there,

>
> >Is there any way that I can save a file (a few k) in to the memory
> >than on disk (and then delete it)? Thanks for help.

>
> >zl2k

>
> I am not 100% sure what you are asking here since "saving a file in to the
> memory" seems to be a bit of an odd way to say load a file into memory. *In
> which case the anwer is yes.
>
> If you know the size of the data you are reading, you can construct a
> vector<unsigned char>(size) and call fstream::read passing it a pointer to the
> first element of the vector. i.e. loads it into memory. *When you're done, you
> can fstream::write it to a file on disk., let the vector go out of scope or
> delete it depending on how it was allocated initially.


Please allow me to clarify the question. In my code it generates a jpg
image, and another api will consume the jpg. I can write the jpg to
disk as "abc.jpg" and pass that to the consumer program, but that may
cost IO time. What I think is if there is a way I can pass it to the
consumer program without the disk IO cost. Thanks.
 
Reply With Quote
 
Öö Tiib
Guest
Posts: n/a
 
      03-28-2011
On Mar 28, 7:38*pm, zl2k <(E-Mail Removed)> wrote:
>
> Please allow me to clarify the question. In my code it generates a jpg
> image, and another api will consume the jpg. I can write the jpg to
> disk as "abc.jpg" and pass that to the consumer program, but that may
> cost IO time. What I think is if there is a way I can pass it to the
> consumer program without the disk IO cost.


Then you take and read up on named pipes

http://en.wikipedia.org/wiki/Named_pipe

 
Reply With Quote
 
Jorgen Grahn
Guest
Posts: n/a
 
      03-28-2011
On Mon, 2011-03-28, zl2k wrote:
> On Mar 28, 10:21*am, cg_chas <(E-Mail Removed)> wrote:
>> On Mon, 28 Mar 2011 08:04:48 -0700 (PDT), zl2k <(E-Mail Removed)> wrote:
>> >hi, there,

>>
>> >Is there any way that I can save a file (a few k) in to the memory
>> >than on disk (and then delete it)? Thanks for help.

>>
>> >zl2k

>>
>> I am not 100% sure what you are asking here since "saving a file in to the
>> memory" seems to be a bit of an odd way to say load a file into memory. *In
>> which case the anwer is yes.

....

> Please allow me to clarify the question. In my code it generates a jpg
> image, and another api will consume the jpg. I can write the jpg to
> disk as "abc.jpg" and pass that to the consumer program, but that may
> cost IO time. What I think is if there is a way I can pass it to the
> consumer program without the disk IO cost. Thanks.


That's still an unclear description which partly contradicts your
first one. I guess you mean this:

"My program (A) can generate an image, but it needs to pass it in
jpeg format to this other program (B), using some unspecified
interface of B. I can do this by storing the image to the file
system (B handles that case), but I want to avoid that."

I'd say it depends entirely on that other program, and your OS.
If you're lucky you're on Unix and B can read the image from standard
input. If you're unlucky B requires a seekable file.

I don't like doing IPC via the file system either, but it's not so
much an issue of I/O cost. I don't like it because
- it kills parallelism: B cannot start its work until A has written
everything
- the file system size sets a limit; I cannot process 10GB pictures
if I don't have 10GB free disk space
- I risk leaving garbage lying around in the file system
if either process crashes or is killed by the user.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
 
Reply With Quote
 
Sjouke Burry
Guest
Posts: n/a
 
      03-28-2011
zl2k wrote:
> hi, there,
>
> Is there any way that I can save a file (a few k) in to the memory
> than on disk (and then delete it)? Thanks for help.
>
> zl2k

Open a ramdisk at startup.
Then treat it as any other drive.
 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      03-28-2011
On Mar 28, 5:38 pm, zl2k <(E-Mail Removed)> wrote:
> On Mar 28, 10:21 am, cg_chas <(E-Mail Removed)> wrote:
> > On Mon, 28 Mar 2011 08:04:48 -0700 (PDT), zl2k <(E-Mail Removed)> wrote:


> > >Is there any way that I can save a file (a few k) in to the
> > >memory than on disk (and then delete it)? Thanks for help.


> > I am not 100% sure what you are asking here since "saving a
> > file in to the memory" seems to be a bit of an odd way to
> > say load a file into memory. In which case the anwer is
> > yes.


> > If you know the size of the data you are reading, you can
> > construct a vector<unsigned char>(size) and call
> > fstream::read passing it a pointer to the first element of
> > the vector. i.e. loads it into memory. When you're done,
> > you can fstream::write it to a file on disk., let the vector
> > go out of scope or delete it depending on how it was
> > allocated initially.


> Please allow me to clarify the question. In my code it generates a jpg
> image, and another api will consume the jpg. I can write the jpg to
> disk as "abc.jpg" and pass that to the consumer program, but that may
> cost IO time.


Or it may not. On most systems, if the file isn't too big, and
is deleted quickly enough after it is written, it may never hit
the disk. Or you can use shared memory---but if the data hangs
around enough in shared memory, it may end up on disk. In
general, the OS makes the decision about such things, not you
(although you can generally force it when necessary, e.g. for
reasons of transactional integrity).

--
James Kanze
 
Reply With Quote
 
puppi
Guest
Posts: n/a
 
      03-29-2011
You could try using shared memory, by looking into the functions
mmap() and shmget(). However, note that it's quite a low level
approach. Also, they are not Standard C++ functions, they are POSIX's.
If you have specific OSs in mind, such as Linux, then it's an
alternative. Just don't expect the portability to be the same as using
the filesystem.
 
Reply With Quote
 
Joshua Maurice
Guest
Posts: n/a
 
      03-29-2011
On Mar 28, 3:36*pm, James Kanze <(E-Mail Removed)> wrote:
> On Mar 28, 5:38 pm, zl2k <(E-Mail Removed)> wrote:
> > Please allow me to clarify the question. In my code it generates a jpg
> > image, and another api will consume the jpg. I can write the jpg to
> > disk as "abc.jpg" and pass that to the consumer program, but that may
> > cost IO time.

>
> Or it may not. *On most systems, if the file isn't too big, and
> is deleted quickly enough after it is written, it may never hit
> the disk. *Or you can use shared memory---but if the data hangs
> around enough in shared memory, it may end up on disk. *In
> general, the OS makes the decision about such things, not you
> (although you can generally force it when necessary, e.g. for
> reasons of transactional integrity).


Allow me to take disagreement with your points in parentheses. From
what little reading I've done, ensuring an actual flush to non-
volatile storage on POSIX OSs and win32 is not easy. Across all
platforms, you do not have a real-life guarantee that sync, fsync, and
fdatasync force the data to non-volatile storage. The last time I
researched this problem, I learned that there are no easy answers. It
depends on the exact OS, its current configuration, its available non-
standard APIs, and whether or not your exact hard disk model and hard
disk driver want to play nice.

In short, I suggest that people do their research before attempting to
write some software with guarantees in the face of failures from power
loss or other catastrophic program or OS failures.
 
Reply With Quote
 
James Kanze
Guest
Posts: n/a
 
      04-03-2011
On Mar 29, 8:35 pm, Joshua Maurice <(E-Mail Removed)> wrote:
> On Mar 28, 3:36 pm, James Kanze <(E-Mail Removed)> wrote:


> > On Mar 28, 5:38 pm, zl2k <(E-Mail Removed)> wrote:
> > > Please allow me to clarify the question. In my code it generates a jpg
> > > image, and another api will consume the jpg. I can write the jpg to
> > > disk as "abc.jpg" and pass that to the consumer program, but that may
> > > cost IO time.


> > Or it may not. On most systems, if the file isn't too big, and
> > is deleted quickly enough after it is written, it may never hit
> > the disk. Or you can use shared memory---but if the data hangs
> > around enough in shared memory, it may end up on disk. In
> > general, the OS makes the decision about such things, not you
> > (although you can generally force it when necessary, e.g. for
> > reasons of transactional integrity).


> Allow me to take disagreement with your points in parentheses. From
> what little reading I've done, ensuring an actual flush to non-
> volatile storage on POSIX OSs and win32 is not easy. Across all
> platforms, you do not have a real-life guarantee that sync, fsync, and
> fdatasync force the data to non-volatile storage. The last time I
> researched this problem, I learned that there are no easy answers. It
> depends on the exact OS, its current configuration, its available non-
> standard APIs, and whether or not your exact hard disk model and hard
> disk driver want to play nice.


Well, it's not portable, that's for sure. And my actual
experience in this domain is limited to Solaris Sparc. But as
long as the disk was local (no NFS involved), it seemed to work.
Or perhaps we were just lucky.

Posix does make specific requirements, as well; presumably, a
system where it didn't work wouldn't be Posix compliant. But of
course, there are always bugs, and if you're remote mounting
drives, the system is dependent on what the protocol provides,
and how well the remote host complies, regardless of what Posix
says, or even how well it has been written itself.

> In short, I suggest that people do their research before attempting to
> write some software with guarantees in the face of failures from power
> loss or other catastrophic program or OS failures.


Definitly. It's important to understand the specific platform
you're programming to, and what guarantees it actually gives.
And to maintain a good deal of scepticism; I'm not aware of any
specific issues with NFS, for example, but as soon as two or
more entities are involved, I become very mistrustful.

--
James Kanze
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
jar file from Eclipse at the root of the jar file rather than nestedin directories wong_powah@yahoo.ca Java 0 03-27-2008 06:26 PM
maps to hold ultra large data sets using customer allocators to allocate disk space rather than main memory CMOS C++ 15 05-17-2007 10:12 PM
Force browser to open a file (rather than save/open/cancel) tiewknvc9 Java 12 02-19-2007 08:21 PM
prompting to "save as" rather than opening images Richard Hollenbeck Javascript 5 01-13-2004 05:36 AM



Advertisments