Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Shift-JIS to UTF-8 conversion

Reply
Thread Tools

Shift-JIS to UTF-8 conversion

 
 
PyTJ
Guest
Posts: n/a
 
      05-19-2005
Hello everybody,

I need to convert a Japanese Shift-JIS CSV file to Unicode UTF-8.

My machine is a Windows 98 english computer with Python 2.3.4

Any hints?.

 
Reply With Quote
 
 
 
 
Jeff Epler
Guest
Posts: n/a
 
      05-19-2005
I think you do something like this (untested):

import codecs

def transcode(infile, outfile, incoding="shift-jis",
outcoding="utf-8"):
f = codecs.open(infile, "rb", incoding)
g = codecs.open(outfile, "wb", outcoding)

g.write(f.read())
# If the file is so large that it can't be read at once, do a loop which
# reads and writes smaller chunks
# while 1:
# block = f.read(4096000)
# if not block: break
# g.write(block)

f.close()
g.close()

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCjRzZJd01MZaTXX0RAg8YAJ4rQ8Fcpwi1AB2a/ZVdALGysct8jACfYdXm
in2aJ3xmdB0ncRZBWXmfMQs=
=bHjV
-----END PGP SIGNATURE-----

 
Reply With Quote
 
 
 
 
rbsharp@gmx.de
Guest
Posts: n/a
 
      05-20-2005
Hello,
I think the answer is basically correct but shift-jis is not a standard
part of
Python 2.3. You will either need to use Python 2.4 where the cjkcodes
are integrated or install them under Python 2.3. The link is
http://cjkpython.i18n.org/

You then also need:
import cjkcodecs.aliases

Richard

Jeff Epler wrote:
> I think you do something like this (untested):
>
> import codecs
>
> def transcode(infile, outfile, incoding="shift-jis",
> outcoding="utf-8"):
> f = codecs.open(infile, "rb", incoding)
> g = codecs.open(outfile, "wb", outcoding)
>
> g.write(f.read())
> # If the file is so large that it can't be read at once, do a loop

which
> # reads and writes smaller chunks
> # while 1:
> # block = f.read(4096000)
> # if not block: break
> # g.write(block)
>
> f.close()
> g.close()
>
> Jeff


 
Reply With Quote
 
George Yoshida
Guest
Posts: n/a
 
      05-20-2005
PyTJ wrote:

> I need to convert a Japanese Shift-JIS CSV file to Unicode UTF-8.
>
> My machine is a Windows 98 english computer with Python 2.3.4
>
> Any hints?.
>


First, you need to install codecs to support japanese encodings.
Python 2.3.* does not support SJIS by default.

I'll give you two options.

- Japanese Codecs
http://www.python.jp/Zope/download/JapaneseCodecs

http://ftp.python.jp/pub/JapaneseCod...in32-py2.3.exe

- CJKCodecs
http://cjkpython.i18n.org/
http://download.berlios.de/cjkpython...in32-py2.3.exe

If you only need Japanese support, Japanese Codecs might be handy.
On the other hand, CJKCodecs can handle much broader encodings.
Aside from that, starting from 2.4, Python ships with CJKCodecs,
so I'd recomment CJKCodecs without reservations.

-- george
 
Reply With Quote
 
Jeff Epler
Guest
Posts: n/a
 
      05-23-2005
On Fri, May 20, 2005 at 12:16:15AM -0700, http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Hello, I think the answer is basically correct but shift-jis is not a
> standard part of Python 2.3.


Ah, I was fooled --- I tested on Python 2.3, but my packager must have
included the codecs you went on to mention.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFCkcJCJd01MZaTXX0RAhNUAKCLbSsAAzxXe9UIjMXd5A N/wKcfbQCeI9j0
lpU5Zu0BgAdD2hTFvKB8kJs=
=Tof0
-----END PGP SIGNATURE-----

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
PSD to XHTML Conversion Services and PSD to HTML CSS ConversionServices, PSD to Joomla, Drupal, Wordpress Conversion xhtml champs Python 0 06-21-2011 11:59 AM
PSD to XHTML Conversion Services and PSD to HTML CSS ConversionServices, PSD to Joomla, Drupal, Wordpress Conversion PSD to XHTML Conversion Services and PSD to HTML CSS Conversion Services, PSD to Joomla, Drupal, Wor VHDL 0 04-25-2011 06:43 AM
conversion operator and conversion ctor subramanian100in@yahoo.com, India C++ 2 09-15-2009 12:46 PM
Date conversion problem with OE importing saroxonline76@vodafone.it Firefox 0 07-12-2005 07:38 PM
Framework 1.0 to 1.1 conversion Luc Bisson ASP .Net 2 11-19-2003 02:40 AM



Advertisments