Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > Signed and unsigned int

Reply
Thread Tools

Signed and unsigned int

 
 
jacob navia
Guest
Posts: n/a
 
      08-10-2004
A signed int can contain up to 2Gig, 2 147 483 648, to be exact.

Since The Mars rovers landed, I have been storing the
photographs in two directories, Spirit and Opportunity. I had
more than 18 000 files in a single directory. Without being
aware of it, I crossed the 2 147 483 648 border last week.

Nothing happens, if you do not attempt to read all files in the directory,
or even worst, copy them to another drive.

Worried by mysterious disk errors, I tried to save my work. When making
the copy however, there was an error when the copy arrived at an image
and could not read it.

Foolish me, I tried to copy the file again.

That was it: the disk started to make a periodic mechanical noise, and
it was GONE ALONG WITH ALL MY DATA IN THE DISK!!!!!!!!!!!!!!!!!!!!!

Why?

When a signed integer is increased beyond 2147483648, it becomes NEGATIVE.
This means that the system will issue a NONSENSE movemevent order to the
read heads, destroying the disk instantly.

I was lucky. I had a full backup of the mars data in my linux system. Ahh
Microsoft.
I fired up the linux machine running ext2 file system. I issued the order to
copy
the files, and started doing other things during the copy.

When the amount of data transmitted arrived at approx 2GB, I heared with
horror that
the disk started doing the SAME repeating mechanical noise and my linux
system
WAS GONE, I HAVE LOST several months of work without any means of getting
my data back.

Signed integer can contain up to 2147483648 bytes. Not a single byte more.

I have developed a compile time switch to check for overflows within
lcc-win32 and
posted here a message, several weeks ago. Nobody cared to answer.

**** !!!!!!!!!!!

C is a nice language in which to write file systems. But IT WOULD BE BETTER
TO
BE CAREFUL WITH THOSE "int" s OK?

You do not believe me?

Try it. Make several directories with several thousand files of 100-200K
each, until
you get more than 2GB.

But backup your drive first...

jacob






 
Reply With Quote
 
 
 
 
Christian Bau
Guest
Posts: n/a
 
      08-10-2004
In article <cfbegn$r34$(E-Mail Removed)>,
"jacob navia" <(E-Mail Removed)> wrote:

> Try it. Make several directories with several thousand files of 100-200K
> each, until
> you get more than 2GB.
>
> But backup your drive first...


A digital video camera records exactly 3.6 million bytes per seoncd.
That is 216 million bytes per minute, or 12,960,000,000 byte per hour. I
have directories on my harddisk with about 13 Gigabyte of contents, my
neighbour has about 30 Gigabyte in a single directory - just 2 1/2 hours
of digital video.

Both machines are Macs using the HFS+ filesystem, but I would think that
there is video editing software for Windows and Linux, and I am sure it
must be able to handle similar amounts of data.
 
Reply With Quote
 
 
 
 
Emmanuel Delahaye
Guest
Posts: n/a
 
      08-10-2004
jacob navia wrote on 10/08/04 :
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.


Of course you meant

"A signed long can contain at least to 2Gig, 2 147 483 648, to be
exact."

or

"On my machine, a signed int can contain up to 2Gig, 2 147 483 648, to
be exact."

--
Emmanuel
The C-FAQ: http://www.eskimo.com/~scs/C-faq/faq.html

"C is a sharp tool"

 
Reply With Quote
 
Eric Sosman
Guest
Posts: n/a
 
      08-10-2004
jacob navia wrote:
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.


"Exact" if you ignore the off-by-one error ...

> Since The Mars rovers landed, I have been storing the
> photographs in two directories, Spirit and Opportunity. I had
> more than 18 000 files in a single directory. Without being
> aware of it, I crossed the 2 147 483 648 border last week.
> [... and two different O/Ses trashed the data].
>
> C is a nice language in which to write file systems. But IT WOULD BE BETTER
> TO
> BE CAREFUL WITH THOSE "int" s OK?


Jacob, while I'm genuinely sorry for your data loss I think
you may have fingered the wrong culprit. It seems unlikely that
any file system nowadays would have a 31-bit limit built in; disks
have been larger than 2GB for a long time now. In fact, disks have
been larger than 4GB for a long time; you'd probably need to look
long and hard to find a drive that small today. And if the file
systems using these larger drives had been limited by 31-bit or
even 32-bit byte counts, plenty of people other than you would
have noticed ...

> You do not believe me?
>
> Try it. Make several directories with several thousand files of 100-200K
> each, until
> you get more than 2GB.


Well, I'm using files of 50-100MB instead, but I hope you'll
accept the (slightly edited) evidence anyhow:

Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c14t40d0s6 356770888 37071528 316131656 11% (withheld)
/dev/dsk/c14t40d1s6 356770888 37140512 316062672 11%
/dev/dsk/c13t40d0s6 356770888 37150672 316052512 11%
/dev/dsk/c16t40d0s6 356770888 37206864 315996320 11%
/dev/dsk/c13t40d1s6 356770888 37356592 315846592 11%
/dev/dsk/c15t40d0s6 356770888 37412760 315790424 11%
/dev/dsk/c16t40d1s6 356770888 53911776 299291408 16%
/dev/dsk/c15t40d1s6 356770888 54248872 298954312 16%

I'd bet that the 31-bit limit (the number in your sad tale
is too coincidental to ignore) has nothing to do with the file
system. It may, perhaps, have something to do with the utility
you were using to mass-copy all that data.

My condolences.

--
http://www.velocityreviews.com/forums/(E-Mail Removed)

 
Reply With Quote
 
Gordon Burditt
Guest
Posts: n/a
 
      08-10-2004
>A signed int can contain up to 2Gig, 2 147 483 648, to be exact.

I know of no system where this is the limit. A 32-bit signed int
cannot contain 2 147 483 648. It can contain 2 147 483 647. It
is not guaranteed that a signed int has more than 16 bits, nor is
there anything preventing it from having 47 or 128 bits.

>Since The Mars rovers landed, I have been storing the
>photographs in two directories, Spirit and Opportunity. I had
>more than 18 000 files in a single directory. Without being
>aware of it, I crossed the 2 147 483 648 border last week.
>
>Nothing happens, if you do not attempt to read all files in the directory,
>or even worst, copy them to another drive.


Most of the problems I have heard of involve having a SINGLE FILE
containing more than 2GB, or 4GB. I hope there is not much software
in use where a whole disk drive is constrained by size limits of
2GB or 4GB. I don't understand why there is a limit on reading a
bunch of files in a directory (unless you were copying them into a
single file, e.g. with tar or cpio) rather than a limit on individual
file size or disk partition size.

A reasonable OS will not let you write a file beyond the limit,
rather than corrupting the file system, whatever that limit is,
even if it's something huge like 512YB.

>Worried by mysterious disk errors, I tried to save my work. When making
>the copy however, there was an error when the copy arrived at an image
>and could not read it.
>
>Foolish me, I tried to copy the file again.
>
>That was it: the disk started to make a periodic mechanical noise, and
>it was GONE ALONG WITH ALL MY DATA IN THE DISK!!!!!!!!!!!!!!!!!!!!!


I'm sorry about your lost files but I'm not entirely convinced that
copying too much at once is the reason you lost them.

>Why?
>
>When a signed integer is increased beyond 2147483648, it becomes NEGATIVE.
>This means that the system will issue a NONSENSE movemevent order to the
>read heads, destroying the disk instantly.


A reasonable OS will not issue such an order to the disk drive.
The driver should range-check the values it is computing and object
if the values are out of range. So, for that matter, should the
disk drive. Disk drives nowadays are much smarter than the old
mechanical floppy drives where you could keep stepping and slam the
heads into things.

>I was lucky. I had a full backup of the mars data in my linux system. Ahh
>Microsoft.
>I fired up the linux machine running ext2 file system. I issued the order to
>copy
>the files, and started doing other things during the copy.
>
>When the amount of data transmitted arrived at approx 2GB, I heared with
>horror that
>the disk started doing the SAME repeating mechanical noise and my linux
>system
>WAS GONE, I HAVE LOST several months of work without any means of getting
>my data back.
>
>Signed integer can contain up to 2147483648 bytes. Not a single byte more.
>
>I have developed a compile time switch to check for overflows within
>lcc-win32 and
>posted here a message, several weeks ago. Nobody cared to answer.
>
>**** !!!!!!!!!!!
>
>C is a nice language in which to write file systems. But IT WOULD BE BETTER
>TO
>BE CAREFUL WITH THOSE "int" s OK?
>
>You do not believe me?
>
>Try it. Make several directories with several thousand files of 100-200K
>each, until
>you get more than 2GB.


I routinely make single files that are greater than 4GB on FreeBSD.
(The size of a burnable DVD image is about 4.7GB if you pack things
to use the maximum capacity. I'm using DVDs to store DATA, not
video or audio. The number goes up if your DVD supports double-sided
or blue-laser DVDs) Yes, I've got some directory trees somewhat
like what you describe, only the total is 22GB (most of it downloaded
with a single multi-file FTP command). Yes, there's a backup of
it on another system as a single 22GB tar file. I need to split
it up into chunks small enough to burn on DVDs. (actually, I've
done this several times, but I don't like how the files end up
getting split - I want related files on the same disk. I do need
to check the program that does the splitting, but I think it's using
off_t's for holding file lengths (64 bits on FreeBSD)). After I do
that, the program that makes a DVD image is essentially a copy
program that puts all the files in one big file (with assorted
headers & stuff). I have had no problems.

I believe I have done this on Linux also, even old versions of Linux
(think TiVo's OS - was that built in 1996?) only supporting LBA32
and not LBA48 (meaning Linux won't support DRIVES larger than 137GB
or so). Modern versions of Linux don't have that limit.

Gordon L. Burditt
 
Reply With Quote
 
Keith Thompson
Guest
Posts: n/a
 
      08-10-2004
"jacob navia" <(E-Mail Removed)> writes:
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.


Typically 2 147 483 647, to be even more exact, but of course it can
be any value greater than or equal to 32767.

> Since The Mars rovers landed, I have been storing the
> photographs in two directories, Spirit and Opportunity. I had
> more than 18 000 files in a single directory. Without being
> aware of it, I crossed the 2 147 483 648 border last week.
>
> Nothing happens, if you do not attempt to read all files in the directory,
> or even worst, copy them to another drive.
>
> Worried by mysterious disk errors, I tried to save my work. When making
> the copy however, there was an error when the copy arrived at an image
> and could not read it.
>
> Foolish me, I tried to copy the file again.
>
> That was it: the disk started to make a periodic mechanical noise, and
> it was GONE ALONG WITH ALL MY DATA IN THE DISK!!!!!!!!!!!!!!!!!!!!!
>
> Why?


Because there's a bug in your OS's filesystem code. (Do you know, or
are you assuming, that that code was written in C?)

[...]
> I have developed a compile time switch to check for overflows within
> lcc-win32 and posted here a message, several weeks ago. Nobody cared
> to answer.


Would this have prevented your problem? Detecting overflow is only
part of the problem; the programmer has to decide what to do once the
overflow is detected. Using a compiler that traps overflows might
have caused whatever program you were using (or the OS itself) to
crash rather than continue with invalid data, which is a bit of an
improvement but not enough of one.

> C is a nice language in which to write file systems. But IT WOULD BE
> BETTER TO BE CAREFUL WITH THOSE "int" s OK?


"Be careful" is always good advice, but you have to know *how* to be
careful. If you want to count more than 2**31-1 of something, you
just need to use something bigger than a 32-bit signed type.
Detecting overflow is a good way to detect bugs, but it's seldom a
good way to correct them; it's rare to want to an overflow handler to
be executed during the normal execution of a program.

<OT>I'm surprised that a Linux ext2 filesystem would behave the way
you describe, but of course this isn't the place to discuss the
specifics.</OT>

--
Keith Thompson (The_Other_Keith) (E-Mail Removed) <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.
 
Reply With Quote
 
Keith Thompson
Guest
Posts: n/a
 
      08-10-2004
Christian Bau <(E-Mail Removed)> writes:
> In article <cfbegn$r34$(E-Mail Removed)>,
> "jacob navia" <(E-Mail Removed)> wrote:
>
> > Try it. Make several directories with several thousand files of 100-200K
> > each, until
> > you get more than 2GB.
> >
> > But backup your drive first...

>
> A digital video camera records exactly 3.6 million bytes per seoncd.
> That is 216 million bytes per minute, or 12,960,000,000 byte per hour. I
> have directories on my harddisk with about 13 Gigabyte of contents, my
> neighbour has about 30 Gigabyte in a single directory - just 2 1/2 hours
> of digital video.
>
> Both machines are Macs using the HFS+ filesystem, but I would think that
> there is video editing software for Windows and Linux, and I am sure it
> must be able to handle similar amounts of data.


<WAY_OT>
For Windows, FAT32 vs. NTFS might be significant.
</WAY_OT>

--
Keith Thompson (The_Other_Keith) (E-Mail Removed) <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.
 
Reply With Quote
 
Joe Wright
Guest
Posts: n/a
 
      08-11-2004
jacob navia wrote:
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.
>
> Since The Mars rovers landed, I have been storing the
> photographs in two directories, Spirit and Opportunity. I had
> more than 18 000 files in a single directory. Without being
> aware of it, I crossed the 2 147 483 648 border last week.
>
> Nothing happens, if you do not attempt to read all files in the directory,
> or even worst, copy them to another drive.
>
> Worried by mysterious disk errors, I tried to save my work. When making
> the copy however, there was an error when the copy arrived at an image
> and could not read it.
>
> Foolish me, I tried to copy the file again.
>
> That was it: the disk started to make a periodic mechanical noise, and
> it was GONE ALONG WITH ALL MY DATA IN THE DISK!!!!!!!!!!!!!!!!!!!!!
>
> Why?
>
> When a signed integer is increased beyond 2147483648, it becomes NEGATIVE.
> This means that the system will issue a NONSENSE movemevent order to the
> read heads, destroying the disk instantly.
>
> I was lucky. I had a full backup of the mars data in my linux system. Ahh
> Microsoft.
> I fired up the linux machine running ext2 file system. I issued the order to
> copy
> the files, and started doing other things during the copy.
>
> When the amount of data transmitted arrived at approx 2GB, I heared with
> horror that
> the disk started doing the SAME repeating mechanical noise and my linux
> system
> WAS GONE, I HAVE LOST several months of work without any means of getting
> my data back.
>
> Signed integer can contain up to 2147483648 bytes. Not a single byte more.
>
> I have developed a compile time switch to check for overflows within
> lcc-win32 and
> posted here a message, several weeks ago. Nobody cared to answer.
>
> **** !!!!!!!!!!!
>
> C is a nice language in which to write file systems. But IT WOULD BE BETTER
> TO
> BE CAREFUL WITH THOSE "int" s OK?
>
> You do not believe me?
>
> Try it. Make several directories with several thousand files of 100-200K
> each, until
> you get more than 2GB.
>
> But backup your drive first...
>
> jacob
>
>
>
>
>
>

The largest signed int on your system is 2147483647, not 2147483648.
Navia should know this. 2^31-1 is a magic number on virtually all
32-bit computer architectures.

01111111 11111111 11111111 11111111 is magic. Magic +1 is..

10000000 00000000 00000000 00000000 and too much magic.

Having said that, I'm surprised and saddened that both Windows and
Linux would actually fall over past the upper bound. I would have
expected something more graceful.
--
Joe Wright (E-Mail Removed)
"Everything should be made as simple as possible, but not simpler."
--- Albert Einstein ---
 
Reply With Quote
 
CBFalconer
Guest
Posts: n/a
 
      08-11-2004
jacob navia wrote:
>
> A signed int can contain up to 2Gig, 2 147 483 648, to be exact.


No, a signed int can contain values from INT_MIN through INT_MAX,
if you #include <limits.h>. Those values must be at least -32767
through 32767.

If you need values from 0 to 2,147,483,648, use an unsigned long
and check for overflow yourself.

--
"Churchill and Bush can both be considered wartime leaders, just
as Secretariat and Mr Ed were both horses." - James Rhodes.
"A man who is right every time is not likely to do very much."
- Francis Crick, co-discover of DNA


 
Reply With Quote
 
Keith Thompson
Guest
Posts: n/a
 
      08-11-2004
Joe Wright <(E-Mail Removed)> writes:
[...]
> Having said that, I'm surprised and saddened that both Windows and
> Linux would actually fall over past the upper bound. I would have
> expected something more graceful.


Note the "[OT]" tag.

I'd be very surprised if the failures Jacob encountered were caused by
something as simple as a signed 32-bit integer overflow in the
filesystem implementation. I'd be especially surprised by this for
the Linux system; for Windows, I could almost believe it for FAT32,
but not for NTFS. (Not that I have any particular expertise for any
of these, but I've used them.) The failure at 2 gigabytes is
suspicious, but I'd expect any of the relevant filesystems to handle
at least that much data in a partition, and I wouldn't expect any them
to have a limitation that depends on the total size of the files in a
single directory.

Perhaps there's a bug in some program he was using to copy the files
-- though it's still surprising that such a bug would trash the target
Linux system.

Never underestimate the destructive power of an unlucky coincidence.

--
Keith Thompson (The_Other_Keith) (E-Mail Removed) <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
(int) -> (unsigned) -> (int) or (unsigned) -> (int) -> (unsigned):I'll loose something? pozz C Programming 12 03-20-2011 11:32 PM
Convert 32 bit unsigned int to 16 bit signed int. Fore C++ 29 09-21-2008 05:55 AM
comparison between signed int and unsigned int Alex C Programming 3 04-26-2006 05:20 AM
unsigned int with signed long int (same width) - how is conversion done? G Fernandes C Programming 2 02-16-2005 10:55 AM
convert signed int to unsigned int Siemel Naran C++ 3 11-29-2004 08:22 AM



Advertisments