Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > Any convenient and elegant way to do encoding conversion in C++?

Reply
Thread Tools

Any convenient and elegant way to do encoding conversion in C++?

 
 
Licheng Fang
Guest
Posts: n/a
 
      09-23-2006
I want to store Chinese in Unicode internally in my program, and give
output in UTF-8 or GBK format. After two days of searching and reading,
I still cannot find a simple and straightforward way to do the code
conversions. In particular, I want portability of the code across
platfroms (Windows and Linux), and I don't like having to refer the
user of my code to some third party libraries for compiling.

Some STL references point to the class "codecvt<>" for this task, but
it seems that I must rely on non-standard, third-party specializations
of this class. The STL itself doesn't implement the code conversions.
Another option I've read about is using GNU's "iconv", which is
implemented in C, and Glib provides a C++ wrapper of "iconv". Again,
re-compiling my source code can be a trouble if I relied heavily on
these libraries. Boost also seems to have some tools for code
conversion. Considering the huge size of the boost libraries, I would
have to pass that as an option.

These are the only possible ways I know of so far. I have to say that
my idea of how this task should be done is somewhat influenced by the
Python way, which is simple and elegant:

if 's' is a string in GBK.

unicode_s = s.decode('gbk')

and when I need to output in GBK I simply convert it back by

output = unicode_s.encode('gbk')

or, I can let the file object know what's the external coding:

import codecs
f = open('somefile', 'r', 'gbk')

I know it's not fair to expect the same things from two different
languages. I wonder, however, how can such a seemingly trivial task be
so infuriatingly complicated in C++.

 
Reply With Quote
 
 
 
 
=?ISO-8859-15?Q?Juli=E1n?= Albo
Guest
Posts: n/a
 
      09-23-2006
Licheng Fang wrote:

> I want to store Chinese in Unicode internally in my program, and give
> output in UTF-8 or GBK format. After two days of searching and reading,
> I still cannot find a simple and straightforward way to do the code
> conversions. In particular, I want portability of the code across
> platfroms (Windows and Linux), and I don't like having to refer the
> user of my code to some third party libraries for compiling.


Then you are in trouble. The windows apis, and the usual windows compilers,
uses a wchar_t type of 16 bits with utf16 encoding, and gcc and his
libraries in Linux use a wchar_t of 32 bits. So if you want the internals
of the program be the same on both platforms, and don't want to use third
party libraries, you must define your own wchar type and conversions to and
from utf8. And some platform dependent code to see what characters are
available in the fonts used.

The utf8 conversions are not hard, in http://www.unicode.org you have a
bunch of information.

--
Salu2
 
Reply With Quote
 
 
 
 
loufoque
Guest
Posts: n/a
 
      09-23-2006
Licheng Fang wrote :

> Some STL references point to the class "codecvt<>" for this task, but
> it seems that I must rely on non-standard, third-party specializations
> of this class. The STL itself doesn't implement the code conversions.


Indeed, it's not in the standard library.
That's why you need to use a third party library, like libiconv, unless
you want to write it yourself of course.

Basically, you just have to define mappings between one encoding and
Unicode. This is a very boring task.
 
Reply With Quote
 
Licheng Fang
Guest
Posts: n/a
 
      09-23-2006

Julián Albo wrote:
> Licheng Fang wrote:
>
> > I want to store Chinese in Unicode internally in my program, and give
> > output in UTF-8 or GBK format. After two days of searching and reading,
> > I still cannot find a simple and straightforward way to do the code
> > conversions. In particular, I want portability of the code across
> > platfroms (Windows and Linux), and I don't like having to refer the
> > user of my code to some third party libraries for compiling.

>
> Then you are in trouble. The windows apis, and the usual windows compilers,
> uses a wchar_t type of 16 bits with utf16 encoding, and gcc and his
> libraries in Linux use a wchar_t of 32 bits. So if you want the internals
> of the program be the same on both platforms, and don't want to use third
> party libraries, you must define your own wchar type and conversions to and
> from utf8. And some platform dependent code to see what characters are
> available in the fonts used.
>
> The utf8 conversions are not hard, in http://www.unicode.org you have a
> bunch of information.


Thanks very much.

I know it's simple to convert Unicode to UTF-8, but the input of my
code is mostly in GBK, which is a popular Chinese encoding. I have to
deal with that.

It seems I have to accept that there's no standard way to convert
encodings in C++. Let me re-state my goals:

1) use Unicode internally in my program, to facilitate my coding task
2) make it as convenient as possible for the users of my code to
compile it

And let me forget about Windows for now, and think about how I can make
it simple to re-compile my code on Linux. Given that there's no
standard way to do encoding convesions, my question is:

What is the most widely used encoding conversion approach on Linux? Is
that the "iconv" library? Is this library included by default on most
Linux platforms? How about the Glib wrappings of this library? Should I
use it?

 
Reply With Quote
 
AnonMail2005@gmail.com
Guest
Posts: n/a
 
      09-23-2006
Licheng Fang wrote:
> It seems I have to accept that there's no standard way to convert
> encodings in C++. Let me re-state my goals:
>
> 1) use Unicode internally in my program, to facilitate my coding task
> 2) make it as convenient as possible for the users of my code to
> compile it
>
> And let me forget about Windows for now, and think about how I can make
> it simple to re-compile my code on Linux. Given that there's no
> standard way to do encoding convesions, my question is:
>
> What is the most widely used encoding conversion approach on Linux? Is
> that the "iconv" library? Is this library included by default on most
> Linux platforms? How about the Glib wrappings of this library? Should I
> use it?


iconv is a very standard way to do this. It's a single C function
which, given proper inputs, will do everything you need. Forget
about C++ wrappers. It's just a C function. Learn how to
declare it correctly for a C++ program and you're done (check
the FAQ). Hopefully, the Glib wrapper you are speaking of
just does that.

I use iconv for encoding internal strings when creating XML
messages which are sent externally. The iconv call is
contained in a wrapper we created to interface with an XML
library. It is cross platform - Linux and Windows.

Notice the word "call" above is not plural. The key point is
that the system is designed so that it doesn't need to keep
track of how strings are encoded. The process of creating
something for external consumption encapsulates the
conversion.

You should carefully design you system to do the same,
otherwise your code will be riddled with conversion calls
and anything that contains data will need to keep track
of how that data is encoded.

Good luck.

 
Reply With Quote
 
=?ISO-8859-15?Q?Juli=E1n?= Albo
Guest
Posts: n/a
 
      09-23-2006
Licheng Fang wrote:

> It seems I have to accept that there's no standard way to convert
> encodings in C++. Let me re-state my goals:
> 1) use Unicode internally in my program, to facilitate my coding task
> 2) make it as convenient as possible for the users of my code to
> compile it


You have for example the mbtowc (multibyte to wide char) function and his
family in the C library, that I suppose will support your encoding if you
have a locale that uses it. You can handle the locale with the C style
functions in <locale.h> or the C++ <locale> ones.

The availability of locales an libraries in linux is off-topic in this
group, you can ask in some linux related newsgroup.

--
Salu2
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
convenient way to read text file multiple times without reopening it Tomas Mikula Java 7 11-15-2007 03:05 PM
Any elegant way of doing this? Morten Wennevik ASP .Net 4 11-08-2005 03:00 PM
Convenient way to get many hash elements at once. Ron M Ruby 1 10-17-2005 09:23 AM
Convenient way for Windows XP Internet Explorers to add my Javascript Bookmarklet? Robert Oschler Javascript 1 08-14-2005 11:10 AM
ThumbsPlus: Any convenient way to display JPEG level? (PeteCresswell) Digital Photography 3 08-09-2005 11:05 PM



Advertisments