Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > text and binary files confusion

Reply
Thread Tools

text and binary files confusion

 
 
joelagnel@gmail.com
Guest
Posts: n/a
 
      03-13-2006
hi friends,

i've been having this confusion for about a year, i want to know the
exact difference between text and binary files.

using the fwrite function in c, i wrote 2 bytes of integers in binary
mode.
according to me, notepad opens files and each byte of the file
read, it converts that byte from ascii to its correct character and
displays
it on screen..
so that's what i did, i wrote 2 bytes (an integer) using fwrite and
since ascii
is 1 byte, i expected 2 characters to be displayed in notepad..
the first character displayed correctly but not the second.

to add to my confusion of text and binary, some FTP servers running
on Linux require html files to be uploaded in 'ascii mode' and binary
files in 'binary mode'.
Both are ordinary files consisting of a sequential series of bytes
after all, then
why a seperate mode?


Any insight into this confusion would be greatly appreciated.

thanks a lot for your time.

- joel

 
Reply With Quote
 
 
 
 
Marc Boyer
Guest
Posts: n/a
 
      03-13-2006
Le 13-03-2006, http://www.velocityreviews.com/forums/(E-Mail Removed) <(E-Mail Removed)> a écrit*:
> i've been having this confusion for about a year, i want to know the
> exact difference between text and binary files.


To my knowledge, the main difference is the interpretation
of the '\n' character.
In text mode, '\n' is an 'end of line' indication, mapped
into '\n', '\r\n' or '\r' depending of the file system
encoding of the end of line.
In binary mode, '\n' is '\n'.

> using the fwrite function in c, i wrote 2 bytes of integers in binary
> mode.
> according to me, notepad opens files and each byte of the file
> read, it converts that byte from ascii to its correct character and
> displays
> it on screen..


Assuming that your encoding is ASCII, yes.

> so that's what i did, i wrote 2 bytes (an integer) using fwrite and
> since ascii
> is 1 byte, i expected 2 characters to be displayed in notepad..
> the first character displayed correctly but not the second.


Perhaps it was not a 'displayable' value.

Marc Boyer
 
Reply With Quote
 
 
 
 
Richard G. Riley
Guest
Posts: n/a
 
      03-13-2006
On 2006-03-13, (E-Mail Removed) <(E-Mail Removed)> wrote:
> hi friends,
>
> i've been having this confusion for about a year, i want to know the
> exact difference between text and binary files.
>
> using the fwrite function in c, i wrote 2 bytes of integers in binary
> mode.
> according to me, notepad opens files and each byte of the file
> read, it converts that byte from ascii to its correct character and
> displays
> it on screen..
> so that's what i did, i wrote 2 bytes (an integer) using fwrite and
> since ascii
> is 1 byte, i expected 2 characters to be displayed in notepad..
> the first character displayed correctly but not the second.
>
> to add to my confusion of text and binary, some FTP servers running
> on Linux require html files to be uploaded in 'ascii mode' and binary
> files in 'binary mode'.
> Both are ordinary files consisting of a sequential series of bytes
> after all, then
> why a seperate mode?
>
>
> Any insight into this confusion would be greatly appreciated.
>
> thanks a lot for your time.
>
> - joel
>


Firstly, dont worry : this is something that trips a lot of people up.

I was about to pen a few lines and then decided not to because (a) I
could not think of an eloquent way of doing it and (b) like most things,
someone else has done it first. The secret of being a great engineer
is not howing how to do something, but knowing that there may be a
better way and knowing how to locate that better way :-;

Here:

http://en.wikipedia.org/wiki/Binary_and_text_files

There is one key part which might confuse you (not knowing your
familiarity with ascii text) and that is:

"Text files are files where most bytes (or short sequences of bytes)
represent ordinary readable characters such as letters,"

The short sequence of bytes is important : google up unicode and dbcs.

--
Debuggers : you know it makes sense.
http://heather.cs.ucdavis.edu/~matlo...g.html#tth_sEc
 
Reply With Quote
 
joelagnel@gmail.com
Guest
Posts: n/a
 
      03-17-2006
> Perhaps it was not a 'displayable' value.

the value was displayable im sure, because the ascii codes
i wrote in binary to the file were:
40H
and 41H
2 bytes.
and i expected it to display AB
but it displayed A@

That's what confuses me..

- joel

 
Reply With Quote
 
joelagnel@gmail.com
Guest
Posts: n/a
 
      03-17-2006
also how does notepad detect the encoding, after all its just
a sequence of bytes, there's nothing in the file that says im
encoded in ascii or ebcdic...

joel

 
Reply With Quote
 
mensanator@aol.com
Guest
Posts: n/a
 
      03-17-2006

(E-Mail Removed) wrote:
> > Perhaps it was not a 'displayable' value.

>
> the value was displayable im sure, because the ascii codes
> i wrote in binary to the file were:
> 40H
> and 41H
> 2 bytes.
> and i expected it to display AB


Wrong.

> but it displayed A@


That's what it's supposed to display. ASCII 'A' is 41H.
ASCII '@' is 40H.

>
> That's what confuses me..


But the computer is not confused. It does what you tell it.
When what you get is not what you expect, always consider
that your expectations may be wrong.

>
> - joel


 
Reply With Quote
 
mensanator@aol.com
Guest
Posts: n/a
 
      03-17-2006

(E-Mail Removed) wrote:
> also how does notepad detect the encoding, after all its just
> a sequence of bytes, there's nothing in the file that says im
> encoded in ascii or ebcdic...


Notepad assumes it's in ASCII. If you actually used EBCDIC
and opened it in notepad, you'd _really_ be confused.

>
> joel


 
Reply With Quote
 
Me
Guest
Posts: n/a
 
      03-17-2006
(E-Mail Removed) wrote:
> hi friends,
>
> i've been having this confusion for about a year, i want to know the
> exact difference between text and binary files.


As far as the C standard is concerned there are some things like not
being able to get the exact file size with binary files, file position
may be off with text files, there being a maximum line length for text
files, and each line in a text file must be outputted with '\n'. This
is a summary, check the standard for the real list. So basically
writing a file in text mode then opening it in binary mode isn't
guaranteed to even give you anything meaningful or work at all (imagine
an implementation that marks whether a file has a text or binary
attribute and a file is determined by both the filename and this
attribute).

On many implementations, the above doesn't apply and all you have to
worry about how the implementation stores the newline character. Since
you're on Windows, here is the convention for text files (treating the
text file as binary here):

BOM(optional)
line1 newline
....
lineN newline(optional)
EOF(optional)

the BOM is to handle unicode files, it can be one of:

0xEF 0xBB 0xBF (UTF-8 BOM)
0xFF 0xFE (UTF-16LE BOM)
0xFE 0xFF (UTF-16BE BOM)

If there is no BOM, then it's up to the software opening it to figure
out the encoding of the file somehow.

Newline is the '\r' '\n' sequence of characters.

Lines are composed of characters. For UTF-16, these characters are
either 2-bytes or 4-bytes depending if they're surrogate pairs. For
UTF-8, characters are 1, 2, 3, or 4 bytes. (and on top of all this, you
have to deal with an arbitrary number of combining characters). You
should read up on unicode, UTF-8, and UTF-16 because this whole issue
of characters and glyphs is confusing when somebody like me uses loose
language like this. If it's not a Unicode file, it most likely uses
some encoding set on the system. Generally white-people countries use
1-byte per character and non-white-people countries use multiple bytes
to encode characters.

EOF is the ASCII ctrl+Z code (0x19). You won't find this except when
opening an ancient DOS file off a floppy or something.


When opening a file in text-mode, most of this should be transparent to
you if your program and the C runtime were carefully designed. i.e. the
above should pretty much be a concern for the C runtime implementors or
programmers that want to handle all of this themselves.


Here's some homework for you:

On the C side, read 7.19, 7.24, and 7.25 in the C standard. Make sure
you know what the following do and how they fit together:

mbstate_t
fwide
fwrite
fputs
fputws
mbtowc
mbstowcs
setlocale
wcstombs
wctomb
mblen

On the windows side, read:

GetACP
MultiByteToWideChar
WideCharToMultiByte
http://blogs.msdn.com/oldnewthing/ar...08/389527.aspx
http://blogs.msdn.com/oldnewthing/ar...31/144893.aspx
http://blogs.msdn.com/oldnewthing/ar...29/457483.aspx
http://blogs.msdn.com/oldnewthing/ar.../24/95235.aspx
http://blogs.msdn.com/michkap/archiv...gory/8717.aspx

On the Unicode side, read:

http://www.unicode.org/faq/utf_bom.html
http://www.cl.cam.ac.uk/~mgk25/unicode.html
http://catch22.net/tuts/ (the articles about his text editor)
http://en.wikipedia.org/wiki/ISO/IEC_8859
http://en.wikipedia.org/wiki/Unicode

After all this, you should be way more advanced about files than most C
programmers.


> using the fwrite function in c, i wrote 2 bytes of integers in binary
> mode.
> according to me, notepad opens files and each byte of the file
> read, it converts that byte from ascii to its correct character and
> displays it on screen..
> so that's what i did, i wrote 2 bytes (an integer) using fwrite and
> since ascii
> is 1 byte, i expected 2 characters to be displayed in notepad..
> the first character displayed correctly but not the second.


Notepad likely uses the winapi function IsTextUnicode to determine the
encoding of the file. Windows supports the ASCII codepage but it is
very rarely used. Yours is most likely set to this:

http://en.wikipedia.org/wiki/Windows-1252

> to add to my confusion of text and binary, some FTP servers running
> on Linux require html files to be uploaded in 'ascii mode' and binary
> files in 'binary mode'.
> Both are ordinary files consisting of a sequential series of bytes
> after all, then
> why a seperate mode?


I don't know about FTP but I think they also allow EBCIDIC to be
transferred as well. I doubt you can send any arbitrary text file out
to another because it highly depends on the source and destination
character set of computers so obviously FTP can only send it out to
computers (non-lossy) that have some sort of mapping between each-other
and the FTP server is aware of this mapping.

 
Reply With Quote
 
Marc Boyer
Guest
Posts: n/a
 
      03-17-2006
Le 17-03-2006, (E-Mail Removed) <(E-Mail Removed)> a écrit*:
> also how does notepad detect the encoding,


Does it ?

> after all its just
> a sequence of bytes, there's nothing in the file that says im
> encoded in ascii or ebcdic...


No, but, in general, a 'familly of platform'
(like Win*, AIX*, AS*) uses the same encoding. That
is to says, I do not know any Win* running EBCDIC.
So, notepad can assume ASCII is used.

Nevertheless, nowadays, in non-english countries,
peoples are using iso-* encodings, UTF-8, perhaps
UTF-16 and others...
As french, I often have problem openning UTF-8
files with iso-latin* editors, and so on.

There are some heuristics used to guess the
encoding. Some editors (like [X]emacs) use some,
and it often works.

Marc Boyer
 
Reply With Quote
 
Richard Bos
Guest
Posts: n/a
 
      03-17-2006
Marc Boyer <(E-Mail Removed)> wrote:

> Le 17-03-2006, (E-Mail Removed) <(E-Mail Removed)> a écrit*:
> > also how does notepad detect the encoding,

>
> Does it ?
>
> > after all its just
> > a sequence of bytes, there's nothing in the file that says im
> > encoded in ascii or ebcdic...

>
> No, but, in general, a 'familly of platform'
> (like Win*, AIX*, AS*) uses the same encoding. That
> is to says, I do not know any Win* running EBCDIC.
> So, notepad can assume ASCII is used.


Not on newer versions of Windows, it can't. More would be off-topic,
except to say that
a. the detection used is easily writable, correctly, in ISO C and
b. _if_ the implementation uses UTF-16 for wchar_t, so is the rest of
the editor.

Richard
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Newbie: working with binary files/extract png from a binary file Jim Ruby 6 12-24-2013 08:09 AM
Advantages of Binary Files over Text files in Search and read utab C++ 3 11-28-2006 03:09 PM
Text files read multiple files into single file, and then recreate the multiple files googlinggoogler@hotmail.com Python 4 02-13-2005 05:44 PM
Question about reading a big binary file and write it into several text (ascii) files Albert Tu Python 2 01-25-2005 03:56 AM
distinguish between binary text and regular text zvika Perl Misc 2 12-12-2004 04:20 PM



Advertisments