Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > Need help reading file data

Reply
Thread Tools

Need help reading file data

 
 
Tom
Guest
Posts: n/a
 
      09-30-2007
I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
.. . .
.. . .
.. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");

for(i=0;i<=499;i++)
{
fscanf(fp,"%f",&(a[i]));

for(j=1;j<=10002;j++)
{
fscanf(fp,"%f",&b);
}

Dosomethingwith(float a, 500);
}
---------------------------------------------------------------------------------------------

 
Reply With Quote
 
 
 
 
Thad Smith
Guest
Posts: n/a
 
      10-01-2007
Tom wrote:
> I have a datafile which has a form of
>
> 10.10 10.20 10.30 ...
> 20.10 20.20 20.30 ...
> 30.10 30.20 30.30 ...
> . . .
> . . .
> . . .
>
> it has 500 rows and 10002 colums.
>
> I tried to read the whole data in a single variable.
> My pc shows error. (Windows XP, notebook)
>
> All I need is to read each column data and do some integration.
> and so on for every colum.
>
> my naive code is following but, it is too slow.
> (it is also limited to read first column, but have to expand to read
> every column)
>
> ---------------------------------------------------------------------------------------------
> FILE *fp;
> float b;
> int i,j;
> float a[500];
>
> fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");
>
> for(i=0;i<=499;i++)
> {
> fscanf(fp,"%f",&(a[i]));
>
> for(j=1;j<=10002;j++)
> {
> fscanf(fp,"%f",&b);
> }
>
> Dosomethingwith(float a, 500);
> }
> ---------------------------------------------------------------------------------------------


If the data is in strict character columns you may be able to seek to
the exact positions of the file where each datum is. I believe it is
not guaranteed for text files, but usually works.

Another approach is to transpose the original data file before processing.

--
Thad
 
Reply With Quote
 
 
 
 
Charlie Gordon
Guest
Posts: n/a
 
      10-01-2007
"Tom" <(E-Mail Removed)> a écrit dans le message de news:
(E-Mail Removed). com...
>I have a datafile which has a form of
>
> 10.10 10.20 10.30 ...
> 20.10 20.20 20.30 ...
> 30.10 30.20 30.30 ...
> . . .
> . . .
> . . .
>
> it has 500 rows and 10002 colums.
>
> I tried to read the whole data in a single variable.
> My pc shows error. (Windows XP, notebook)


It should not: the amount of data is roughly 5 million values, which can be
loaded in memory as float or double for a total of 20 or 40 megabytes. If
you are running XP, you must have a lot more RAM than that. It could also
be a limitation of the C compiler you are using. Try cygwin and gcc.

> All I need is to read each column data and do some integration.
> and so on for every colum.
>
> my naive code is following but, it is too slow.
> (it is also limited to read first column, but have to expand to read
> every column)
>
> ---------------------------------------------------------------------------------------------
> FILE *fp;
> float b;
> int i,j;
> float a[500];
>
> fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");
>
> for(i=0;i<=499;i++)


Better use the actual number of elements:

for (i = 0; i < 500; i++)

> {
> fscanf(fp,"%f",&(a[i]));
>
> for(j=1;j<=10002;j++)
> {
> fscanf(fp,"%f",&b);
> }


try removing this second loop and just scan for the '\n':

{
int c;
while ((c = getc(fp)) != EOF && c != '\n')
continue;
}

This should make the program faster.

>
> Dosomethingwith(float a, 500);
> }
> ---------------------------------------------------------------------------------------------


--
Chqrlie.


 
Reply With Quote
 
Barry Schwarz
Guest
Posts: n/a
 
      10-10-2007
On Sun, 30 Sep 2007 17:49:44 -0000, Tom <(E-Mail Removed)> wrote:

>I have a datafile which has a form of
>
>10.10 10.20 10.30 ...
>20.10 20.20 20.30 ...
>30.10 30.20 30.30 ...
>. . .
>. . .
>. . .
>
>it has 500 rows and 10002 colums.
>
>I tried to read the whole data in a single variable.
>My pc shows error. (Windows XP, notebook)


What error? Compile time or run time?

>
>All I need is to read each column data and do some integration.
>and so on for every colum.
>
>my naive code is following but, it is too slow.
>(it is also limited to read first column, but have to expand to read
>every column)
>
>---------------------------------------------------------------------------------------------
> FILE *fp;
> float b;
> int i,j;
> float a[500];
>
> fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");


You should check that fopen succeeded.

>
> for(i=0;i<=499;i++)
> {
> fscanf(fp,"%f",&(a[i]));


You should check that fscanf succeeded. You have now read the first
column of a record.

>
> for(j=1;j<=10002;j++)


This loop will try to read the next 10002 columns of the record.
Unfortunately, there are only 10001 columns left to be read. At the
end of this loop you will have read the first column of the next
record.

You might want to try using fgets and strtod to insure you never lose
synchronization.

> {
> fscanf(fp,"%f",&b);
> }
>
> Dosomethingwith(float a, 500);


It seems like you should be using a[i] and b here.

> }
>---------------------------------------------------------------------------------------------



Remove del for email
 
Reply With Quote
 
alex
Guest
Posts: n/a
 
      10-10-2007
On Oct 1, 1:49 am, Tom <(E-Mail Removed)> wrote:
> I have a datafile which has a form of
>
> 10.10 10.20 10.30 ...
> 20.10 20.20 20.30 ...
> 30.10 30.20 30.30 ...
> . . .
> . . .
> . . .
>
> it has 500 rows and 10002 colums.
>
> I tried to read the whole data in a single variable.
> My pc shows error. (Windows XP, notebook)
>
> All I need is to read each column data and do some integration.
> and so on for every colum.
>
> my naive code is following but, it is too slow.
> (it is also limited to read first column, but have to expand to read
> every column)
>
> ---------------------------------------------------------------------------------------------
> FILE *fp;
> float b;
> int i,j;
> float a[500];
>
> fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");
>
> for(i=0;i<=499;i++)
> {
> fscanf(fp,"%f",&(a[i]));
>
> for(j=1;j<=10002;j++)
> {
> fscanf(fp,"%f",&b);
> }
>
> Dosomethingwith(float a, 500);
> }
> ---------------------------------------------------------------------------------------------


Since only numeric data presents, my suggestion is first convert this
text file into binary, and then seek to right
position as needed.

for example,
position(i, j) = i * 10002 * sizeof(float) + j * sizeof(float)
(i, j) is (row, col) index, starting from 0

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
help reading data from a file Zoe Phoenix Ruby 5 03-11-2009 03:24 PM
Reading a file and resuming reading. Karim Ali Python 2 05-25-2007 02:04 PM
UnauthorizedAccessException when reading XML files (no problem when reading other file-types) blabla120@gmx.net ASP .Net 0 09-15-2006 02:08 PM
reading the DB vs. reading a text file...performance preference? Darrel ASP .Net 3 11-11-2004 02:27 PM
reading output file data as input data Stephen Moon Perl Misc 5 02-29-2004 02:16 PM



Advertisments