Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > xmlrpclib and carriagereturn (\r)

Reply
Thread Tools

xmlrpclib and carriagereturn (\r)

 
 
Jonathan Ballet
Guest
Posts: n/a
 
      03-17-2006
Hello,

I have developped a XMLRPC server, which runs under Gnu/Linux with
python2.3.
This server receives method calls from Windows client. The server got some
parameters which are string, which contains carriage return characters,
just after the line feed character; like "bla\n\rbla".


The problem is, xmlrpclib "eats" those carriage return characters when
loading the XMLRPC request, and replace it by "\n". So I got "bla\n\nbla".

When I sent back those parameters to others Windows clients (they are
doing some kind of synchronisation throught the XMLRPC server), I send
to them only "\n\n", which makes problems when rendering strings.



It seems that XMLRPC spec doesn't propose to eat carriage return
characters : (from http://www.xmlrpc.com/spec)
"""
What characters are allowed in strings? Non-printable characters?
Null characters? Can a "string" be used to hold an arbitrary chunk
of binary data?

Any characters are allowed in a string except < and &, which are
encoded as &lt; and &amp;. A string can be used to encode binary
data.
"""



Here is an example which described the problem :
$ python
Python 2.3.5 (#2, Sep 4 2005, 22:01:42)
[GCC 3.3.5 (Debian 1:3.3.5-13)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xmlrpclib
>>>
>>> data = """<?xml version="1.0" encoding="utf-8"?>\n<methodCall>\n

<methodName>testmethod</methodName>\n
<params>\n <param>\n <value>\n
<string>bla\n\rbla\n\r bla</string>\n </value>\n </param>\n
</params>\n</methodCall>"""
>>>
>>> xmlrpclib.loads(data)

(('bla\n\nbla\n\n\tbla',), u'testmethod')
>>>


... whereas I expected to have (('bla\n\rbla\n\r\tbla',), u'testmethod')


It seems to be a rather strange comportement from xmlrpclib. Is it known ?
So, what happens here ? How could I solve this problem ?

Thanks for any answers,
Jonathan


 
Reply With Quote
 
 
 
 
Fredrik Lundh
Guest
Posts: n/a
 
      03-18-2006
Jonathan Ballet wrote:

> The problem is, xmlrpclib "eats" those carriage return characters when
> loading the XMLRPC request, and replace it by "\n". So I got "bla\n\nbla".
>
> When I sent back those parameters to others Windows clients (they are
> doing some kind of synchronisation throught the XMLRPC server), I send
> to them only "\n\n", which makes problems when rendering strings.
>
> It seems that XMLRPC spec doesn't propose to eat carriage return
> characters : (from http://www.xmlrpc.com/spec)


XMLRPC is XML, and XML normalizes line feeds:

http://www.w3.org/TR/2004/REC-xml-20...#sec-line-ends

relying on non-standard line terminators in text is fragile and horridly out-
dated; if the data you're sending qualifies as text, send it as text, and do
the conversion at the end points (if at all necessary).

if it's not text, use the binary wrapper.

> I send to them only "\n\n", which makes problems when rendering strings.


how are you rendering things if you have problems treating a line feed
as a line feed? that's rather unusual; most Windows applications have
no problems handling line feeds properly, so I'm not sure why it has to
be a problem in your application...

</F>



 
Reply With Quote
 
 
 
 
Ben Cartwright
Guest
Posts: n/a
 
      03-18-2006
Jonathan Ballet wrote:
> The problem is, xmlrpclib "eats" those carriage return characters when
> loading the XMLRPC request, and replace it by "\n". So I got "bla\n\nbla".
>
> When I sent back those parameters to others Windows clients (they are
> doing some kind of synchronisation throught the XMLRPC server), I send
> to them only "\n\n", which makes problems when rendering strings.


Did you develop the Windows client, too? If so, the client-side fix is
trivial: replace \n with \r\n in all renderable strings. Or update
both the client and the server to encode the strings, also trivial
using the base64 module.

If not, and you're in the unfortunate position of being forced to
support buggy third-party clients, read on.

> It seems that XMLRPC spec doesn't propose to eat carriage return
> characters : (from http://www.xmlrpc.com/spec)

(snip)
> It seems to be a rather strange comportement from xmlrpclib. Is it known ?
> So, what happens here ? How could I solve this problem ?


The XMLRPC spec doesn't say anything about CRs one way or the other.
Newline handling is necessarily left to the XML parser implementation.
In Python's case, xmlrpclib uses the xml.parsers.expat module, which
reads universal newlines and writes Unix-style newlines (\n). There's
no option to disable this feature.

You could modify xmlrpclib to use a different parser, but it would be
much easier to just hack the XML response right before it's sent out.
I'm assuming you used the SimpleXMLRPCServer module. Example:

from SimpleXMLRPCServer import *

class MyServer(SimpleXMLRPCServer):
def _marshaled_dispatch(self, data, dm=None):
response = SimpleXMLRPCDispatcher._marshaled_dispatch(self,
data, dm)
return response.replace('\n', '\r\n')

server = MyServer(('localhost', 8000))
server.register_introspection_functions()
server.register_function(lambda x: x, 'echo')
server.serve_forever()

Hope that helps,
--Ben

 
Reply With Quote
 
Ben Cartwright
Guest
Posts: n/a
 
      03-18-2006
Jonathan Ballet wrote:
> The problem is, xmlrpclib "eats" those carriage return characters when
> loading the XMLRPC request, and replace it by "\n". So I got "bla\n\nbla".
>
> When I sent back those parameters to others Windows clients (they are
> doing some kind of synchronisation throught the XMLRPC server), I send
> to them only "\n\n", which makes problems when rendering strings.



Whoops, just realized we're talking about "\n\r" here, not "\r\n".
Most of my previous reply doesn't apply to your situation, then.

As far as Python's expat parser is concerned, "\n\r" is two newlines:
one Unix-style and one Mac-style. It correctly (per XML specs)
normalizes both to Unix-style.

Is "\n\r" being used as a newline by your Windows clients, or is it a
control code? If the former, I'd sure like to know why. If the
latter, then you're submitting binary data and you shouldn't be using
<string> to begin with. Try <base64>.

If worst comes to worst and you have to stick with sending "\n\r"
intact in a <string> param, you'll need to modify xmlrpclib to use a
different (and technically noncompliant) XML parser. Here's an ugly
hack to do that out of the box:

# In your server code:
import xmlrpclib
# This forces xmlrpclib to fall back on the obsolete xmllib module:
xmlrpclib.ExpatParser = None

xmllib doesn't normalize newlines, so it's noncompliant. But this is
actually what you want.

--Ben

 
Reply With Quote
 
Jonathan Ballet
Guest
Posts: n/a
 
      03-18-2006
Le Sat, 18 Mar 2006 02:17:36 -0800, Ben Cartwright a écrit*:

> Jonathan Ballet wrote:
>> The problem is, xmlrpclib "eats" those carriage return characters when
>> loading the XMLRPC request, and replace it by "\n". So I got "bla\n\nbla".
>>
>> When I sent back those parameters to others Windows clients (they are
>> doing some kind of synchronisation throught the XMLRPC server), I send
>> to them only "\n\n", which makes problems when rendering strings.

>


[Just replying to this message. See F. Lundh reply too]

>
> Whoops, just realized we're talking about "\n\r" here, not "\r\n".
> Most of my previous reply doesn't apply to your situation, then.
>
> As far as Python's expat parser is concerned, "\n\r" is two newlines:
> one Unix-style and one Mac-style. It correctly (per XML specs)
> normalizes both to Unix-style.
>
> Is "\n\r" being used as a newline by your Windows clients, or is it a
> control code? If the former, I'd sure like to know why.


We are developping the Windows client. I think my teammates keep \n\r,
which is the defaut line terminator they had when getting string from text
input, and because ".net framework does the right things for you, etc. etc."

> If the
> latter, then you're submitting binary data and you shouldn't be using
> <string> to begin with. Try <base64>.
>
> If worst comes to worst and you have to stick with sending "\n\r"
> intact in a <string> param, you'll need to modify xmlrpclib to use a
> different (and technically noncompliant) XML parser. Here's an ugly
> hack to do that out of the box:
>
> # In your server code:
> import xmlrpclib
> # This forces xmlrpclib to fall back on the obsolete xmllib module:
> xmlrpclib.ExpatParser = None
>
> xmllib doesn't normalize newlines, so it's noncompliant. But this is
> actually what you want.


Well, I thought to use something like that. Currently, we are stucking
with the kind ugly hack you sent in your previous message (replace("\n",
"\n\r")
However, now that I know that xmlrpclib handle line terminator correctly
(regarding XML spec. ), I will try to see if we can handle line feed
correctly in our Windows application.
I think it should be the better solution.

Anyway, thanks a lot for your answers (both of them

>
> --Ben


J.
 
Reply With Quote
 
Jonathan Ballet
Guest
Posts: n/a
 
      03-18-2006
Le Sat, 18 Mar 2006 08:54:49 +0100, Fredrik Lundh a écrit*:

> Jonathan Ballet wrote:
>

[snip]
>
> XMLRPC is XML, and XML normalizes line feeds:
>
> http://www.w3.org/TR/2004/REC-xml-20...#sec-line-ends
>
> relying on non-standard line terminators in text is fragile and horridly out-
> dated; if the data you're sending qualifies as text, send it as text, and do
> the conversion at the end points (if at all necessary).


Ah, you send me the right link. So xmlrpclib handle those line
terminators correctly. That's good.

>
> if it's not text, use the binary wrapper.
>
>> I send to them only "\n\n", which makes problems when rendering strings.

>
> how are you rendering things if you have problems treating a line feed
> as a line feed? that's rather unusual; most Windows applications have
> no problems handling line feeds properly, so I'm not sure why it has to
> be a problem in your application...
>
> </F>


The problem recently pointed out, so we were searching where those
carriage return disapeared.
However, if we can throw them away and render line feed as line feed, it
would be the best solution imho. I'll look into this.

Thanks for your answer,
J.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: illegal line end in character literal in escaped unicode CARRIAGERETURN / NEW LINE Joshua Cranmer Java 0 05-15-2009 02:50 PM
Re: illegal line end in character literal in escaped unicode CARRIAGERETURN / NEW LINE Lew Java 0 05-15-2009 02:38 PM
Re: illegal line end in character literal in escaped unicode CARRIAGERETURN / NEW LINE Mark Space Java 0 05-15-2009 02:35 PM
xmlrpclib and decoding entity references Chris Curvey Python 5 05-04-2005 10:33 PM
xmlrpclib and binary data as normal parameter strings Rune Froysa Python 3 04-20-2005 09:19 AM



Advertisments