Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Program works great, except under less, cron or execl (Unicode?)

Reply
Thread Tools

Program works great, except under less, cron or execl (Unicode?)

 
 
Sam
Guest
Posts: n/a
 
      09-18-2008
I have a program which works great when run from the command line.

But when I run it combined with something else such as:
- piping it through less
- cron
- execl (i.e. calling it from another python program)

it gives me a unicode error

File "../myparser.py", line 261, in set_attributes
print "self.atd['Name'] is: ", self.atd['Name']
UnicodeEncodeError: 'ascii' codec can't encode character u'\xeb' in
position 7: ordinal not in range(12

I'd post the whole program here, except it involves weird Unicode
strings.

I could probably change the program to get it working under less/cron/
etc.

But I'd rather understand exactly what the issue is. Why does it work
fine when run directly from the command line, but not otherwise?
 
Reply With Quote
 
 
 
 
Diez B. Roggisch
Guest
Posts: n/a
 
      09-18-2008
Sam wrote:

> I have a program which works great when run from the command line.
>
> But when I run it combined with something else such as:
> - piping it through less
> - cron
> - execl (i.e. calling it from another python program)
>
> it gives me a unicode error
>
> File "../myparser.py", line 261, in set_attributes
> print "self.atd['Name'] is: ", self.atd['Name']
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xeb' in
> position 7: ordinal not in range(12
>
> I'd post the whole program here, except it involves weird Unicode
> strings.
>
> I could probably change the program to get it working under less/cron/
> etc.
>
> But I'd rather understand exactly what the issue is. Why does it work
> fine when run directly from the command line, but not otherwise?


Most probably because when to running directly inside a terminal, it gets
it's stdin/stdout as pipes - and python can't attempt to guess the proper
encoding on that, as it does on a terminal.

And thus, when you print unicode to the pipe, it can't decide which encoding
to use.

To circumvene this, try & wrap stdout into a codecs-module wrapper with a
proper encoding applied (e.g. utf-.

You might make that conditionally based on the sys.stdout.encoding-variable
being set or not, albeit I'm not 100% sure to what it actually gets set
when used in a subprocess. But this should give you the idea where to look.



Diez
 
Reply With Quote
 
 
 
 
Diez B. Roggisch
Guest
Posts: n/a
 
      09-18-2008
> Most probably because when to running directly inside a terminal, it gets

That was of course meant to be "not running directly inside a terminal".

> it's stdin/stdout as pipes - and python can't attempt to guess the proper
> encoding on that, as it does on a terminal.


Diez
 
Reply With Quote
 
Sam
Guest
Posts: n/a
 
      09-19-2008
Diez for the win...

sys.stdout.encoding does indeed have the proper value when called from
the command line of UTF-8.

But when piped into anything or called from anywhere it's None.

Just for completeness, here's my test program:
#!/usr/bin/env python
import sys
print sys.stdout.encoding

And here are the results:
$ ./encoding.py
UTF-8
$ ./encoding.py | cat
None

Really, really annoying!

So how can I set sys.stdout.encoding so it's UTF-8 when piped through
cat (or anything else).

I tried assigning to it, but no dice.

On Sep 18, 2:12*am, "Diez B. Roggisch" <(E-Mail Removed)> wrote:
> Sam wrote:
> > I have a program which works great when run from the command line.

>
> > But when I run it combined with something else such as:
> > - piping it through less
> > - cron
> > - execl (i.e. calling it from another python program)

>
> > it gives me a unicode error

>
> > *File "../myparser.py", line 261, in set_attributes
> > * * print "self.atd['Name'] is: ", self.atd['Name']
> > UnicodeEncodeError: 'ascii' codec can't encode character u'\xeb' in
> > position 7: ordinal not in range(12

>
> > I'd post the whole program here, except it involves weird Unicode
> > strings.

>
> > I could probably change the program to get it working under less/cron/
> > etc.

>
> > But I'd rather understand exactly what the issue is. *Why does it work
> > fine when run directly from the command line, but not otherwise?

>
> Most probably because when to running directly inside a terminal, it gets
> it's stdin/stdout as pipes - and python can't attempt to guess the proper
> encoding on that, as it does on a terminal.
>
> And thus, when you print unicode to the pipe, it can't decide which encoding
> to use.
>
> To circumvene this, try & wrap stdout into a codecs-module wrapper with a
> proper encoding applied (e.g. utf-.
>
> You might make that conditionally based on the sys.stdout.encoding-variable
> being set or not, albeit I'm not 100% sure to what it actually gets set
> when used in a subprocess. But this should give you the idea where to look.
>
> Diez


 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      09-24-2008
In message
<(E-Mail Removed)>, Sam
wrote:

> So how can I set sys.stdout.encoding so it's UTF-8 when piped through
> cat (or anything else).
>
> I tried assigning to it, but no dice.


You could try wrapping it in a file object that does explicit encoding
translation, using codecs.EncodedFile
<http://docs.python.org/lib/module-codecs.html>.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: What is the difference between 'except IOError as e:' and 'except MRAB Python 0 11-18-2009 02:52 AM
What is the difference between 'except IOError as e:' and 'except Peng Yu Python 1 11-18-2009 02:38 AM
try -> except -> else -> except? David House Python 2 07-06-2009 05:48 PM
who is simpler? try/except/else or try/except Fabio Z Tessitore Python 5 08-13-2007 12:52 AM
converting a nested try/except statement into try/except/else John Salerno Python 20 08-11-2006 02:48 PM



Advertisments