Binary XML

I work in software research and development and we're going to be doing
some investigations into message traffic. This is for embedded systems.
What we're looking at right now is XML encoded messages and want to
look into binary or compressed XML, using network services (probably
IP). As such, we'd like to find some libraries that would aid our
investigation.

I'm busily doing web and newsgroup searches, including some of the
WBXML stuff, but thought I'd throw it out here in case someone had
suggestions on some candidates.

Features we'd like:

1. Open Source. We aren't creating any deliverables, so we aren't too
worried about license restrictions at this point. We're purely
interested in sending messages and gathering data.

2. We're working with Linux-based components, so something compatible
with that.

3. A Python API. Actually the tech lead is interested in that, I'm more
interested in a C API, but I'll try to keep him happy. Something
usuable from either would be ideal. Other scripting APIs would be
considered as well.


If anyone has libraries they've worked with, worked on, or reviewed
that look like they might be candidates, I'd be grateful for
suggestions. If this isn't the best newsgroup (seemed the most
promising general-interest group) redirection to a more appropriate
place would also be welcome.



Brian

Default User

Default User wrote:

> I work in software research and development and we're going to be doing
> some investigations into message traffic. This is for embedded systems.
> What we're looking at right now is XML encoded messages and want to
> look into binary or compressed XML, using network services (probably
> IP). As such, we'd like to find some libraries that would aid our
> investigation.

I have read your posting twice and still havent
understood what you want to do. Do you want to
send XML data across a network ?

You mention embedded systems. Such systems are
usually constrained in available memory. For
such systems with limited resources, I can
recommend an extension of the AWK scripting
language with focus XML processing:

http://home.vrweb.de/~juergen.kahrs/...of-an-XML-file

* Default User wrote in comp.text.xml:
^^^^^^^^^^^^
>If anyone has libraries they've worked with, worked on, or reviewed
>that look like they might be candidates, I'd be grateful for
>suggestions. If this isn't the best newsgroup (seemed the most
>promising general-interest group) redirection to a more appropriate
>place would also be welcome.

The XML-Dev mailing list and http://www.w3.org/XML/EXI/ fora might
be a better place.
--
Björn Höhrmann · private.php?do=newpm&u= · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

Jürgen Kahrs wrote:

> Default User wrote:
>
> > I work in software research and development and we're going to be
> > doing some investigations into message traffic. This is for
> > embedded systems. What we're looking at right now is XML encoded
> > messages and want to look into binary or compressed XML, using
> > network services (probably IP). As such, we'd like to find some
> > libraries that would aid our investigation.
>
> I have read your posting twice and still havent
> understood what you want to do. Do you want to
> send XML data across a network ?
>
> You mention embedded systems. Such systems are
> usually constrained in available memory.

This would be the sorts of processors typically used in some embedded
systems, such as flight control, connected via ethernet, and passing
data via network services.

Our hardware tends not to be as restricted in some aspects as other
embedded applications. We're not doing microwave ovens

The embedded part is not the most important part, think Linux boxes
connected in LAN configurations. The speed of message transmission and
the ability of test applications to receive said messages is what we
are investigating. We would not want to trade off speed for size or
memory usage.

> For
> such systems with limited resources, I can
> recommend an extension of the AWK scripting
> language with focus XML processing:
>
> http://home.vrweb.de/~juergen.kahrs/...l#Printing-an-
> outline-of-an-XML-file

I will take a look, thanks.




Brian

Bjoern Hoehrmann wrote:

> * Default User wrote in comp.text.xml:
> ^^^^^^^^^^^^
> > If anyone has libraries they've worked with, worked on, or reviewed
> > that look like they might be candidates, I'd be grateful for
> > suggestions. If this isn't the best newsgroup (seemed the most
> > promising general-interest group) redirection to a more appropriate
> > place would also be welcome.
>
> The XML-Dev mailing list and http://www.w3.org/XML/EXI/ fora might
> be a better place.

I'll check it out, thanks.



Brian

Default User wrote:
> look into binary or compressed XML

This has been proposed many times, and the debate is still in progress.
The W3C has a Working Group in progress which is continuing to debate
whether there is in fact any possiblity of reaching a consensus on what
"binary XML" actually means, or whether in fact it's a catch-all term
for a bunch of solutions which are best considered local/custom/internal
representations rather than something to be standardized.

See http://www.w3.org/XML/Binary/

Historically, after folks have investigated it, they've generally
reached the conclusion that simply running ordinary text-based XML
through a compressor such as the ZIP algorithm yields more compact
messages than an attempt at a binary-XML representation would (since it
compresses on larger scales, and compresses text content) -- and unlike
the binary XML proposals, it retains all the advantages of being easily
human-accessible/debuggable which are part of what makes XML attractive.
(And decompressing zip and the like is surprisingly cheap; compression
is the hard part.)

There are certainly uses for non-text representations of the XML infoset
-- but I don't think I've ever seen a good justification for using one
of these outside a specific application. In fact, trying to make it
general-purpose tends to fight against the compression you're hoping to
achieve.

Interesting question. I may yet be proven wrong. But I'm betting that
the characterization working group comes back to the same old
conclusion: standardized binary XML is an oxymoron.


--
() ASCII Ribbon Campaign | Joe Kesselman
/\ Stamp out HTML e-mail! | System architexture and kinetic poetry

Hi,

I am working on an open-source XML data binding implementation for
C++[1]. Some of our users expressed interest[2] in sending data over
the network in compact, binary form. As a result, I am working on
binary serialization/deserialization of the in-memory representation.
I already have serialization implemented and on a sample document
of about 1K we get about 400 bytes binary representation using
CDR streams (data representation used in CORBA). For comparison,
zip-compressing the same XML file results in about 600 bytes.

Also note that this is not a binary XML in a sense that no markup
information is stored (e.g., element/attribute names, etc.) but
rather a pure data that receiver is assumed to know the format
of. This saves additional space.


"Default User" <> writes:

> Features we'd like:
>
> 1. Open Source. We aren't creating any deliverables, so we aren't too
> worried about license restrictions at this point. We're purely
> interested in sending messages and gathering data.

Check.


> 2. We're working with Linux-based components, so something compatible
> with that.

Check.


> 3. A Python API. Actually the tech lead is interested in that, I'm more
> interested in a C API, but I'll try to keep him happy. Something
> usuable from either would be ideal. Other scripting APIs would be
> considered as well.

It provides neither at the moment but I would imagine it shouldn't
be to difficult to interface with the code from either C or Python.


[1] http://codesynthesis.com/products/xsd/

[2] http://codesynthesis.com/pipermail/x...ay/000338.html
http://codesynthesis.com/pipermail/x...ay/000343.html

hth,
-boris
--
Boris Kolpackov
Code Synthesis Tools CC
http://www.codesynthesis.com
Open-Source, Cross-Platform C++ XML Data Binding

> See http://www.w3.org/XML/Binary/

Actually, that group is no longer in existance. The new group that was
formed to take on this task is the 'Efficient XML Interchange Working
Group' (EXI for short). The URL for it is as follows:

http://www.w3.org/XML/EXI/

Regards,

Ed Day
Objective Systems, Inc.
http://www.obj-sys.com


Joe Kesselman wrote:
> Default User wrote:
> > look into binary or compressed XML
>
> This has been proposed many times, and the debate is still in progress.
> The W3C has a Working Group in progress which is continuing to debate
> whether there is in fact any possiblity of reaching a consensus on what
> "binary XML" actually means, or whether in fact it's a catch-all term
> for a bunch of solutions which are best considered local/custom/internal
> representations rather than something to be standardized.
>
> See http://www.w3.org/XML/Binary/
>
> Historically, after folks have investigated it, they've generally
> reached the conclusion that simply running ordinary text-based XML
> through a compressor such as the ZIP algorithm yields more compact
> messages than an attempt at a binary-XML representation would (since it
> compresses on larger scales, and compresses text content) -- and unlike
> the binary XML proposals, it retains all the advantages of being easily
> human-accessible/debuggable which are part of what makes XML attractive.
> (And decompressing zip and the like is surprisingly cheap; compression
> is the hard part.)
>
> There are certainly uses for non-text representations of the XML infoset
> -- but I don't think I've ever seen a good justification for using one
> of these outside a specific application. In fact, trying to make it
> general-purpose tends to fight against the compression you're hoping to
> achieve.
>
> Interesting question. I may yet be proven wrong. But I'm betting that
> the characterization working group comes back to the same old
> conclusion: standardized binary XML is an oxymoron.
>
>
> --
> () ASCII Ribbon Campaign | Joe Kesselman
> /\ Stamp out HTML e-mail! | System architexture and kinetic poetry

wrote:
> Actually, that group is no longer in existance. The new group that was
> formed to take on this task is the 'Efficient XML Interchange Working
> Group' (EXI for short).

Thanks; I'd missed that switch-over.

It will be interesting to see whether they can reach a consensus... and,
then, whether the rest of the XML community agrees to accept that
consensus.


--
() ASCII Ribbon Campaign | Joe Kesselman
/\ Stamp out HTML e-mail! | System architexture and kinetic poetry

Boris Kolpackov wrote:

> Hi,
>
> I am working on an open-source XML data binding implementation for
> C++[1]. Some of our users expressed interest[2] in sending data over
> the network in compact, binary form.


> [1] http://codesynthesis.com/products/xsd/
>
> [2] http://codesynthesis.com/pipermail/x...ay/000338.html
> http://codesynthesis.com/pipermail/x...ay/000343.html

I appreciate the information and I'll review these links. Thanks.




Brian