Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > maximum file size of FOPEN

Reply
Thread Tools

maximum file size of FOPEN

 
 
uremae
Guest
Posts: n/a
 
      05-19-2005
I tried to open some large files in my computer. (ram 512MB)

1. is there limitation of file size FOPEN?

2. if I have a file that is larger than the maximum size, how can I
open the file?

 
Reply With Quote
 
 
 
 
Walter Roberson
Guest
Posts: n/a
 
      05-19-2005
In article <(E-Mail Removed) .com>,
uremae <(E-Mail Removed)> wrote:
>I tried to open some large files in my computer. (ram 512MB)


>1. is there limitation of file size FOPEN?


No, not unless there is something unusual and system-specific.

There could, for example, -potentially- be issues with
opening a device or pseudo-device, if the device driver for some
reason decided it needed to take a snapshot into memory
(e.g., opening /dev/core might take a snapshot of kernel state,)
Any such behaviour would be outside the bounds of C: such
behaviour does not happen for plain files.


>2. if I have a file that is larger than the maximum size, how can I
>open the file?


Any such matter would be system specific. fopen() is all that
C itself provides.

It isn't usually a problem to *open* an existing large file: the
problem is usually in *reading* the large file. Some systems
are only able to read to about the 2 gigabyte mark, or are
allowed to read indefinitely but cannot position (ftell/fseek)
beyond 2 gigabytes without using system-specific calls.
--
"Never install telephone wiring during a lightning storm." -- Linksys
 
Reply With Quote
 
 
 
 
Malcolm
Guest
Posts: n/a
 
      05-19-2005

"uremae" <(E-Mail Removed)> wrote
>
> 1. is there limitation of file size FOPEN?
>
> 2. if I have a file that is larger than the maximum size, how can I
> open the file?
>

There's ususally some limit, though it might be so large that there is no
chance of exceeding it.
If fopen() fails to open a huge file, or the read functions won't allow you
to read it all, the best solution is to look for lower-level system specific
calls. Sometimes you might be able to redesign your files so that they fall
within the limit.

However make sure that the size of the file really is the problem, not that
there is some hardware problem with the disk drive, or that the file is
corrupt in some way.


 
Reply With Quote
 
Walter Roberson
Guest
Posts: n/a
 
      05-19-2005
In article <d6ivtm$hv4$(E-Mail Removed)-infra.bt.com>,
Malcolm <(E-Mail Removed)> wrote:

>"uremae" <(E-Mail Removed)> wrote


>> 1. is there limitation of file size FOPEN?


>There's ususally some limit, though it might be so large that there is no
>chance of exceeding it.


Could you give us an example of a system with such a limit, Malcolm ?

I have certainly run into systems whose filesize was limited
(with different limits for different filesystem types), but I
cannot think of anything in C or POSIX.1 or any implementation
that I have -encountered- that would fopen() for opening a
[plain] file for read once the file had made it on to the
filesystem.

Ability to read all of a large file is questionable, though:
the internal file position counter can overflow (e.g., if one
is NFS or SMB'ing over the contents of a file which resides on
a remote filesystem that supports much larger files than the local
system expects.)

And ability to position/ reposition in a large file is quite
questionable seeing as fseek() is limited to taking a 'long' for the
offset...
--
Beware of bugs in the above code; I have only proved it correct,
not tried it. -- Donald Knuth
 
Reply With Quote
 
Mark McIntyre
Guest
Posts: n/a
 
      05-19-2005
On 19 May 2005 21:57:05 GMT, in comp.lang.c ,
http://www.velocityreviews.com/forums/(E-Mail Removed)-cnrc.gc.ca (Walter Roberson) wrote:

>In article <d6ivtm$hv4$(E-Mail Removed)-infra.bt.com>,
>Malcolm <(E-Mail Removed)> wrote:
>
>>"uremae" <(E-Mail Removed)> wrote

>
>>> 1. is there limitation of file size FOPEN?

>
>>There's ususally some limit, though it might be so large that there is no
>>chance of exceeding it.

>
>Could you give us an example of a system with such a limit, Malcolm ?
>
>I have certainly run into systems whose filesize was limited
>(with different limits for different filesystem types), but I
>cannot think of anything in C or POSIX.1 or any implementation
>that I have -encountered- that would fopen() for opening a
>[plain] file for read once the file had made it on to the
>filesystem.


The file could exist on network attached storage which supports larger
file sizes than the local OS - f'rexample you could have Win95
locally, and be mapping a W2k3 or *nix drive.
..


--
Mark McIntyre
CLC FAQ <http://www.eskimo.com/~scs/C-faq/top.html>
CLC readme: <http://www.ungerhu.com/jxh/clc.welcome.txt>
 
Reply With Quote
 
uremae
Guest
Posts: n/a
 
      05-20-2005
then how can I read the large file that is larger than gigabye with
what system-specific calls in linux or unix.

 
Reply With Quote
 
Walter Roberson
Guest
Posts: n/a
 
      05-20-2005
In article <(E-Mail Removed)>,
Mark McIntyre <(E-Mail Removed)> wrote:
>On 19 May 2005 21:57:05 GMT, in comp.lang.c ,
>(E-Mail Removed)-cnrc.gc.ca (Walter Roberson) wrote:


>>In article <d6ivtm$hv4$(E-Mail Removed)-infra.bt.com>,
>>Malcolm <(E-Mail Removed)> wrote:


>>>"uremae" <(E-Mail Removed)> wrote


>>>> 1. is there limitation of file size FOPEN?


>>>There's ususally some limit, though it might be so large that there is no
>>>chance of exceeding it.


>>Could you give us an example of a system with such a limit, Malcolm ?


>The file could exist on network attached storage which supports larger
>file sizes than the local OS


I see that Malcolm was correct. Checking the opengroup.org fopen() man
page, I see that one of the defined error returns is

[EOVERFLOW]
The named file is a regular file and the size of the file
cannot be represented correctly in an object of type off_t.
--
"Never install telephone wiring during a lightning storm." -- Linksys
 
Reply With Quote
 
Walter Roberson
Guest
Posts: n/a
 
      05-20-2005
In article <(E-Mail Removed). com>,
uremae <(E-Mail Removed)> wrote:
>then how can I read the large file that is larger than gigabye with
>what system-specific calls in linux or unix.


You are guaranteed up to (2 gigabytes - 1) in any conforming
unix system (provided the filesystem supports files that large.)

There is no standard unix interface for reading files whose
size cannot be represented as off_t .

In Linux, if your version of Linux has Large Files System support,
then you can open() using the O_LARGEFILE flag.
--
"Mathematics? I speak it like a native." -- Spike Milligan
 
Reply With Quote
 
William Ahern
Guest
Posts: n/a
 
      05-20-2005
uremae <(E-Mail Removed)> wrote:
> then how can I read the large file that is larger than gigabye with
> what system-specific calls in linux or unix.


The folks over in comp.os.linux.development.apps and comp.unix.programmer
would be more helpful since this is a really platform dependent thing.

On more recent Linux platforms you would typically define _FILE_OFFSET_BITS
to 64 before you include _any_ standard or system header. This would then
automagically select the 64 bit interface allowing for > 2GB file access for
those interfaces which normally wouldn't be able to handle files that large.

For example:

gcc -D_FILE_OFFSET_BITS=64 -o main main.c

- Bill
 
Reply With Quote
 
Peter Nilsson
Guest
Posts: n/a
 
      05-20-2005
Walter Roberson wrote:
> Ability to read all of a large file is questionable, though:
> the internal file position counter can overflow (e.g., if one
> is NFS or SMB'ing over the contents of a file which resides on
> a remote filesystem that supports much larger files than the local
> system expects.)


AFAICS, the counter can't overflow per se (this implies undefined
behaviour). Rather, an attempt to get the current file position can
fail.

> And ability to position/ reposition in a large file is quite
> questionable seeing as fseek() is limited to taking a 'long'
> for the offset...


You are allowed to perform multiple seeks and keep your own file
position data.

--
Peter

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
fopen() with full path affecting subsequent fopen calls Michel Rouzic C Programming 4 04-28-2008 04:48 PM
Creating the maximum number of menus and maximum number of stills rossco DVD Video 2 11-24-2005 09:33 PM
What is up with fopen??? FOpen and local directories Nonee HTML 2 10-25-2005 09:18 PM
The number name 'System.Web.UI.WebControls' contains more than the maximum number of prefixes. The maximum is 3. mayur ASP .Net 2 07-02-2004 10:35 AM
Usoing fopen etc is there a way to find file size Angus Comber C Programming 6 02-10-2004 01:41 AM



Advertisments