Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > File Size Limit Exceeded - How to handle work around this?

Reply
Thread Tools

File Size Limit Exceeded - How to handle work around this?

 
 
eastcoastguyz
Guest
Posts: n/a
 
      12-13-2006
I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.

Is this a compiler issue? I would like to see a C code example of how
to increase the limit or make it unlimited (if that is a wise thing to
do).

Thanks in advance!

 
Reply With Quote
 
 
 
 
Walter Roberson
Guest
Posts: n/a
 
      12-13-2006
In article <(E-Mail Removed) om>,
eastcoastguyz <(E-Mail Removed)> wrote:
>I wrote a simple program to continue to create a very large file (on
>purpose), and even though there is plenty of disk space on that device
>the program aborted with the error message "File Size Limit Exceeded".
>The file size was 2147483647. I checked ulimit -a and its set to
>unlimited.


>Is this a compiler issue?


Probably not.

>I would like to see a C code example of how
>to increase the limit or make it unlimited (if that is a wise thing to
>do).


It is probably an operating system limitation (or a disk quota
limitation). ulimit (which is not part of C) with -a set to
unlimited, just means that the operating system will allow you to
write files as big as is supported by that particular file system.

Whether your operating system supports larger files at all would
be OS specific, as would be any special means to create such files.
This is a matter that should be taken to a resource that deals
with your specific OS. It is -possible- that they will tell you
there to change some flags to your compiles, but that would be
for deep OS implementation reasons, not for reasons directly related
to standard C.
--
If you lie to the compiler, it will get its revenge. -- Henry Spencer
 
Reply With Quote
 
 
 
 
dcorbit@connx.com
Guest
Posts: n/a
 
      12-13-2006

eastcoastguyz wrote:
> I wrote a simple program to continue to create a very large file (on
> purpose), and even though there is plenty of disk space on that device
> the program aborted with the error message "File Size Limit Exceeded".
> The file size was 2147483647. I checked ulimit -a and its set to
> unlimited.
>
> Is this a compiler issue? I would like to see a C code example of how
> to increase the limit or make it unlimited (if that is a wise thing to
> do).
>
> Thanks in advance!


Write the file in blocks and maintain your own chain or use a database,
which will already have done that

 
Reply With Quote
 
eastcoastguyz
Guest
Posts: n/a
 
      12-13-2006
I forgot to mention, the OS is Linux with the latest version of CentOS
using 'cc'.

eastcoastguyz wrote:
> I wrote a simple program to continue to create a very large file (on
> purpose), and even though there is plenty of disk space on that device
> the program aborted with the error message "File Size Limit Exceeded".
> The file size was 2147483647. I checked ulimit -a and its set to
> unlimited.
>
> Is this a compiler issue? I would like to see a C code example of how
> to increase the limit or make it unlimited (if that is a wise thing to
> do).
>
> Thanks in advance!


 
Reply With Quote
 
Michael
Guest
Posts: n/a
 
      12-13-2006
> I wrote a simple program to continue to create a very large file (on
> purpose), and even though there is plenty of disk space on that device
> the program aborted with the error message "File Size Limit Exceeded".
> The file size was 2147483647. I checked ulimit -a and its set to
> unlimited.
>
> Is this a compiler issue? I would like to see a C code example of how
> to increase the limit or make it unlimited (if that is a wise thing to
> do).


It's been a long time, though, so this advice may be dated.

A long time ago, when the Linux 2.4 kernel was new (i.e., when 64-bit
file sizes were just starting to catch on), we used to have this same
problem. Back then, you had to compile with some special symbol
defined, LARGE_FILES or something like that. That caused the typical
file size types (offset_t and the like) to be 64 bit. You might find
something like that in the compiler documentation.

Of course, to narrow it to OS or C, you could have your program write
to stdout and redirect to a file. If it still doesn't work, you have
an OS problem. And if it isn't C, it could be your OS kernel, or your
shell, or your filesystem (e.g., NFS) that has some kind of limit in
it.

Michael

 
Reply With Quote
 
eastcoastguyz
Guest
Posts: n/a
 
      12-13-2006

Michael wrote:
> > I wrote a simple program to continue to create a very large file (on
> > purpose), and even though there is plenty of disk space on that device
> > the program aborted with the error message "File Size Limit Exceeded".
> > The file size was 2147483647. I checked ulimit -a and its set to
> > unlimited.
> >
> > Is this a compiler issue? I would like to see a C code example of how
> > to increase the limit or make it unlimited (if that is a wise thing to
> > do).

>
> It's been a long time, though, so this advice may be dated.
>
> A long time ago, when the Linux 2.4 kernel was new (i.e., when 64-bit
> file sizes were just starting to catch on), we used to have this same
> problem. Back then, you had to compile with some special symbol
> defined, LARGE_FILES or something like that. That caused the typical
> file size types (offset_t and the like) to be 64 bit. You might find
> something like that in the compiler documentation.
>
> Of course, to narrow it to OS or C, you could have your program write
> to stdout and redirect to a file. If it still doesn't work, you have
> an OS problem. And if it isn't C, it could be your OS kernel, or your
> shell, or your filesystem (e.g., NFS) that has some kind of limit in
> it.
>
> Michael


Thanks for the suggestion. I made another version of this C program,
where it writes to standard output and piped the output to a file. It
had no problem creating a file size over 3GB, exceeding the 2GB limit I
encountered.
This system is using gcc version 3.4.6 20060404 (Red Hat 3.4.6-3) which
is what came with the latest version of CentOS. The kernel is:
[root@localhost proc]# cat /proc/version
Linux version 2.6.9-42.0.3.EL (buildsvn@build-i386) (gcc version 3.4.6
20060404

 
Reply With Quote
 
loic-dev@gmx.net
Guest
Posts: n/a
 
      12-13-2006
Hello,

> > > I wrote a simple program to continue to create a very large file (on
> > > purpose), and even though there is plenty of disk space on that device
> > > the program aborted with the error message "File Size Limit Exceeded".
> > > The file size was 2147483647. I checked ulimit -a and its set to
> > > unlimited.


Try to compile your program with the following flags:
-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64

HTH,
Loic.

 
Reply With Quote
 
Kohn Emil Dan
Guest
Posts: n/a
 
      12-13-2006


On Wed, 13 Dec 2006, http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> Hello,
>
>>>> I wrote a simple program to continue to create a very large file (on
>>>> purpose), and even though there is plenty of disk space on that device
>>>> the program aborted with the error message "File Size Limit Exceeded".
>>>> The file size was 2147483647. I checked ulimit -a and its set to
>>>> unlimited.

>
> Try to compile your program with the following flags:
> -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64


Probably he meant:

-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64


 
Reply With Quote
 
eastcoastguyz
Guest
Posts: n/a
 
      12-13-2006

Kohn Emil Dan wrote:
> On Wed, 13 Dec 2006, (E-Mail Removed) wrote:
>
> > Hello,
> >
> >>>> I wrote a simple program to continue to create a very large file (on
> >>>> purpose), and even though there is plenty of disk space on that device
> >>>> the program aborted with the error message "File Size Limit Exceeded".
> >>>> The file size was 2147483647. I checked ulimit -a and its set to
> >>>> unlimited.

> >
> > Try to compile your program with the following flags:
> > -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64

>
> Probably he meant:
>
> -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64


Yes! This worked. Thanks to everyone who posted in response to my
request for help.

Does anyone know if it is required to do these compiler options
regardless of the hardware the same OS runs on? Thanks!

 
Reply With Quote
 
Nelu
Guest
Posts: n/a
 
      12-14-2006
eastcoastguyz wrote:
> Kohn Emil Dan wrote:
>> On Wed, 13 Dec 2006, (E-Mail Removed) wrote:
>>
>>> Hello,
>>>
>>>>>> I wrote a simple program to continue to create a very large file (on
>>>>>> purpose), and even though there is plenty of disk space on that device
>>>>>> the program aborted with the error message "File Size Limit Exceeded".
>>>>>> The file size was 2147483647. I checked ulimit -a and its set to
>>>>>> unlimited.
>>> Try to compile your program with the following flags:
>>> -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64

>> Probably he meant:
>>
>> -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

>
> Yes! This worked. Thanks to everyone who posted in response to my
> request for help.
>
> Does anyone know if it is required to do these compiler options
> regardless of the hardware the same OS runs on? Thanks!
>


<OT>
I think the only macro you need defined is _GNU_SOURCE which will
enable a number of macros and the ...LARGEFILE64... ones are
among them. I think this is the preferred way to do this. I
couldn't find _GNU_SOURCE in my header files on 64 bit Gentoo
</OT>
but it may be because I didn't pay too much attention. You
shouldn't have posted the question here and you should not
consider advices received from this group. The standard C doesn't
know about 64 vs 32 bit, files larger than 2 GB or specific
extensions. This is a question for a group that deals with the
combination of compiler/OS that you're using.

--
Ioan - Ciprian Tandau
tandau _at_ freeshell _dot_ org (hope it's not too late)
(... and that it still works...)
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Cisco IOS 2908 Exceeded Size Limit rdbrock Cisco 3 12-29-2008 07:42 AM
OverflowError: regular expression code size limit exceeded gdetre@princeton.edu Python 0 04-16-2008 01:59 AM
c program, file size limit, how to solve? 2G bytes limit. guru.slt@gmail.com C++ 1 06-27-2005 11:05 PM
file size limit exceeded Peter Ballard C Programming 9 08-02-2003 08:06 PM



Advertisments