Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > Faster way to write in a file

Reply
Thread Tools

Faster way to write in a file

 
 
LilacSkin
Guest
Posts: n/a
 
      02-13-2008
Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16* / ;

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

Tk.
 
Reply With Quote
 
 
 
 
santosh
Guest
Posts: n/a
 
      02-13-2008
LilacSkin wrote:

> Hi,
>
> Currently, I write an signed long long in a file with the fprintf
> function:
>
> signed long long * pData = NULL;
> unsigned long long k = 0;
> unsigned long DataAvail = 0 ;
>
> pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
> (65536*16* / ;
>
> for ( k = 0; k < DataAvail/8; k ++ ) {
> fprintf ( pFilec, "%lli\t", pData[k]);
> }
>
> The problem is that function is very very slow !
>
> In several forums, I saw that fwrite function is better, but I don't
> know how to use it.
> Can you help me, please ?


size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

 
Reply With Quote
 
 
 
 
LilacSkin
Guest
Posts: n/a
 
      02-13-2008
On 13 fév, 13:47, santosh <(E-Mail Removed)> wrote:
> LilacSkin wrote:
> > Hi,

>
> > Currently, I write an signed long long in a file with the fprintf
> > function:

>
> > signed long long * pData = NULL;
> > unsigned long long k = 0;
> > unsigned long DataAvail = 0 ;

>
> > pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
> > (65536*16* / ;

>
> > for ( k = 0; k < DataAvail/8; k ++ ) {
> > fprintf ( pFilec, "%lli\t", pData[k]);
> > }

>
> > The problem is that function is very very slow !

>
> > In several forums, I saw that fwrite function is better, but I don't
> > know how to use it.
> > Can you help me, please ?

>
> size_t rc;
> rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
> if (rc != 1048576) {
> puts("Write error.");
> /* ... */
> }


It's not working !

 
Reply With Quote
 
santosh
Guest
Posts: n/a
 
      02-13-2008
LilacSkin wrote:

> On 13 fév, 13:47, santosh <(E-Mail Removed)> wrote:
>> LilacSkin wrote:
>> > Hi,

>>
>> > Currently, I write an signed long long in a file with the fprintf
>> > function:

>>
>> > signed long long * pData = NULL;
>> > unsigned long long k = 0;
>> > unsigned long DataAvail = 0 ;

>>
>> > pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
>> > (65536*16* / ;

>>
>> > for ( k = 0; k < DataAvail/8; k ++ ) {
>> > fprintf ( pFilec, "%lli\t", pData[k]);
>> > }

>>
>> > The problem is that function is very very slow !

>>
>> > In several forums, I saw that fwrite function is better, but I
>> > don't know how to use it.
>> > Can you help me, please ?

>>
>> size_t rc;
>> rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
>> if (rc != 1048576) {
>> puts("Write error.");
>> /* ... */
>> }

>
> It's not working !


How exactly? Did you check that the elements parameter is correct? What
is the value that rc has after the fwrite() call? Is pFilec pointing to
a valid stream for which you have appropriate permissions?

 
Reply With Quote
 
Mark Bluemel
Guest
Posts: n/a
 
      02-13-2008
LilacSkin wrote:
> Hi,
>
> Currently, I write an signed long long in a file with the fprintf
> function:


No. You don't. You write the textual representation of the signed long
long value to the file. I will return to this point shortly.

> signed long long * pData = NULL;
> unsigned long long k = 0;
> unsigned long DataAvail = 0 ;
>
> pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
> (65536*16* / ;
>
> for ( k = 0; k < DataAvail/8; k ++ ) {
> fprintf ( pFilec, "%lli\t", pData[k]);
> }


This snipped code is clearly incomplete, so tells us virtually nothing.

pDataRx is undefined. DataAvail is 0, so the loop will not be entered...

> The problem is that function is very very slow !


Compared to what? How do you know?

> In several forums, I saw that fwrite function is better,


Than what? In what way? Do you know (by measuring) that fprintf is the
cause of your problem?

As I pointed out above, fprintf is converting your signed long long
values to their textual representation. fwrite can't do that - it would
write the internal binary representation of the data.

I suppose you could write your own single-purpose function to convert
signed long long values into text and write that text with fwrite, and
that may perform better than fprintf...

> but I don't know how to use it.


It (fwrite) is a simple enough function which is documented in the
standard and any good text.

> Can you help me, please ?


You'd need to give us more information about the problem before we can
help you with the solution.
 
Reply With Quote
 
Mark Bluemel
Guest
Posts: n/a
 
      02-13-2008
santosh wrote:
> LilacSkin wrote:
>
>> Hi,
>>
>> Currently, I write an signed long long in a file with the fprintf
>> function:
>>
>> signed long long * pData = NULL;
>> unsigned long long k = 0;
>> unsigned long DataAvail = 0 ;
>>
>> pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
>> (65536*16* / ;
>>
>> for ( k = 0; k < DataAvail/8; k ++ ) {
>> fprintf ( pFilec, "%lli\t", pData[k]);
>> }
>>
>> The problem is that function is very very slow !
>>
>> In several forums, I saw that fwrite function is better, but I don't
>> know how to use it.
>> Can you help me, please ?

>
> size_t rc;
> rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
> if (rc != 1048576) {
> puts("Write error.");
> /* ... */
> }


How does dumping the data in binary form compare to writing a
tab-delimited textual representation?
 
Reply With Quote
 
Morris Dovey
Guest
Posts: n/a
 
      02-13-2008
Mark Bluemel wrote:

> How does dumping the data in binary form compare to writing a
> tab-delimited textual representation?


If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

A special-purpose double -> text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

http://groups.google.com/group/comp....5f74496633fe3a

(mind the wrap)

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USA
http://www.iedu.com/DeSoto
 
Reply With Quote
 
Morris Dovey
Guest
Posts: n/a
 
      02-13-2008
Mark Bluemel wrote:

> My point was that the OP had clearly been trying to produce a textual
> representation.
>
> From context, I would not have expected the OP to understand why
> Santosh's suggestion was not compatible with his/her original goal.


I agree with you - that's why I wrote what you snipped:

>> A special-purpose double -> text conversion routine can provide a
>> fair amount of speed improvement.


My thinking is that if speed is critical to LilacSkin, then the
textual representation can be produced somewhat faster (perhaps
almost twice as fast) by substituting a user-written function for
the standard formatting routine.

Once that's been done, fwrite() may indeed be an appropriate
mechanism. I'm actually not sure whether Santosh mentally skipped
a step or was advocating binary output.

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USA
http://www.iedu.com/DeSoto
 
Reply With Quote
 
Mark Bluemel
Guest
Posts: n/a
 
      02-13-2008
Morris Dovey wrote:
> Mark Bluemel wrote:
>
>> How does dumping the data in binary form compare to writing a
>> tab-delimited textual representation?

>
> If the textual representation is being done by fprintf(), writing
> the unconverted data in binary form is _much_ faster - and, of
> course, may produce problems if there's a subsequent need to read
> the data on a different machine.


My point was that the OP had clearly been trying to produce a textual
representation.

From context, I would not have expected the OP to understand why
Santosh's suggestion was not compatible with his/her original goal.
 
Reply With Quote
 
Malcolm McLean
Guest
Posts: n/a
 
      02-13-2008
"LilacSkin" <(E-Mail Removed)> wrote in message news
> Hi,
>
> Currently, I write an signed long long in a file with the fprintf
> function:
>
> signed long long * pData = NULL;
> unsigned long long k = 0;
> unsigned long DataAvail = 0 ;
>
> pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
> (65536*16* / ;
>
> for ( k = 0; k < DataAvail/8; k ++ ) {
> fprintf ( pFilec, "%lli\t", pData[k]);
> }
>
> The problem is that function is very very slow !
>
> In several forums, I saw that fwrite function is better, but I don't
> know how to use it.
> Can you help me, please ?
>

Replace %lli with %llx and measure the speedup.
Is hexadecimal output acceptable to you?

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Easiest way to read a XML doc from file, reformat it and write it indented to a second file? Raymond Schanks Java 0 07-15-2010 02:52 PM
Any way to make pages faster? Terry Olsen ASP .Net 6 08-01-2005 11:26 PM
XSLT is way faster using Java 5 Matt Stephens Java 0 02-06-2005 09:16 PM
TURNING CRAZY, is there a way to write it in a different way? whats wrong francisco lopez Javascript 2 12-31-2004 11:15 PM
Faster way to write binary using fstream?? Jon Hyland C++ 4 10-04-2004 11:55 PM



Advertisments