Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Cpp + Python: static data dynamic initialization in *nix shared lib?

Reply
Thread Tools

Cpp + Python: static data dynamic initialization in *nix shared lib?

 
 
Alf P. Steinbach /Usenet
Guest
Posts: n/a
 
      07-09-2010
[Cross-posted comp.lang.python and comp.lang.c++]

I lack experience with shared libraries in *nix and so I need to ask...

This is about "cppy", some support for writing Python extensions in C++ that I
just started on (some days ago almost known as "pynis" (not funny after all)).

For an extension module it seems that Python requires each routine to be defined
as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern "C"' and C++
linkage, using a routine declared as 'static' in a class as a C callback,
formally they're two different kinds, and I seem to recall that /some/ C++
compiler balks at that kind of mixing unless specially instructed to allow it.
Perhaps it was the Sun compiler?

Anyway, to be formally correct I cannot generate the required C routines via
templating, and I ended up using macros that the user must explicitly invoke,
like, here the Py doc's first extension module example recoded using cppy,


------------------------------------------------------------------
<code file="spam.cpp">
#include <progrock/cppx/devsupport/better_experience.h>
#include <progrock/cppy/Module.h>
using namespace progrock;

class Spam: public cppy::Module
{
public:
Spam(): cppy::Module( "spam" )
{
setDocString( L"blåbærsyltetøy er blått" );
}

PyObject* system( PyObject* args )
{
const char *command;
int sts;

if( !PyArg_ParseTuple( args, "s", &command ) )
{
return NULL;
}
sts = ::system( command );
return Py_BuildValue( "i", sts );
}
};

CPPY_MODULE_CROUTINE( Spam, system, "Execute a shell command" )

PyMODINIT_FUNC PyInit_spam()
{
return cppy::init< Spam >();
}
</code>
------------------------------------------------------------------


It works in Windows.

But here CPPY_MODULE_CROUTINE does three things:

A Defining the 'extern "C"' routine.
I cannot think of any problem here.

B Defining installation data for that routine.
Possible problem: initializing a static with address of routine?

C -> Adding that install data record into a linked list!
Possible problem: are dynamic initialization actions guaranteed
to be performed in *nix shared library?

Problem (C) is outside the realm of the C++ standard, since the C++ standard
doesn't support shared libraries, and I've never actually used *nix shared
libraries so I don't /know/...

Is such dynamic initialization guaranteed?

For completeness, the macro definition (the 0 in there is a list next-pointer):


<code>
#define CPPY_MODULE_CROUTINE_DEF( cppClassName, name ) \
extern "C" \
static PyObject* cppClassName##_##name( PyObject*, PyObject* args ) \
{ \
return :rogrock::cppy::module<cppClassName>().name( args ); \
}

#define CPPY_MODULE_CROUTINE_INSTALLDATA( cppClassName, name, docString ) \
static :rogrock::cppy::detail::ModuleRoutineDescriptor \
cppClassName##_##name##_descriptor = { \
0, \
#name, \
docString, \
&cppClassName##_##name \
}; \
\
static bool cppClassName##_##name##_descriptor_installed = \
:rogrock::cppy::detail::addToList< cppClassName >( \
cppClassName##_##name##_descriptor \
);

#define CPPY_MODULE_CROUTINE( cppClassName, name, docString ) \
CPPY_MODULE_CROUTINE_DEF( cppClassName, name ) \
CPPY_MODULE_CROUTINE_INSTALLDATA( cppClassName, name, docString )
</code>


TIA.,

- Alf

--
blog at <url: http://alfps.wordpress.com>
 
Reply With Quote
 
 
 
 
Ian Collins
Guest
Posts: n/a
 
      07-09-2010
On 07/10/10 03:52 AM, Alf P. Steinbach /Usenet wrote:
> [Cross-posted comp.lang.python and comp.lang.c++]
>
> I lack experience with shared libraries in *nix and so I need to ask...
>
> This is about "cppy", some support for writing Python extensions in C++
> that I just started on (some days ago almost known as "pynis" (not funny
> after all)).
>
> For an extension module it seems that Python requires each routine to be
> defined as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern
> "C"' and C++ linkage, using a routine declared as 'static' in a class as
> a C callback, formally they're two different kinds, and I seem to recall
> that /some/ C++ compiler balks at that kind of mixing unless specially
> instructed to allow it. Perhaps it was the Sun compiler?


Yes, it will (correctly) issue a warning.

As the is a bit OT, contact me directly and we can work through it. I
have had similar fun and games adding PHP modules!

--
Ian Collins
 
Reply With Quote
 
 
 
 
geremy condra
Guest
Posts: n/a
 
      07-09-2010
On Fri, Jul 9, 2010 at 5:22 PM, Ian Collins <(E-Mail Removed)> wrote:
> On 07/10/10 03:52 AM, Alf P. Steinbach /Usenet wrote:
>>
>> [Cross-posted comp.lang.python and comp.lang.c++]
>>
>> I lack experience with shared libraries in *nix and so I need to ask...
>>
>> This is about "cppy", some support for writing Python extensions in C++
>> that I just started on (some days ago almost known as "pynis" (not funny
>> after all)).
>>
>> For an extension module it seems that Python requires each routine to be
>> defined as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern
>> "C"' and C++ linkage, using a routine declared as 'static' in a class as
>> a C callback, formally they're two different kinds, and I seem to recall
>> that /some/ C++ compiler balks at that kind of mixing unless specially
>> instructed to allow it. Perhaps it was the Sun compiler?

>
> Yes, it will (correctly) issue a warning.
>
> As the is a bit OT, contact me directly and we can work through it. *I have
> had similar fun and games adding PHP modules!


I'd appreciate it if you'd either leave this on-list or cc me in on this, as
I'm working through a similar issue.

Geremy Condra
 
Reply With Quote
 
Alf P. Steinbach /Usenet
Guest
Posts: n/a
 
      07-09-2010
* Ian Collins, on 09.07.2010 23:22:
> On 07/10/10 03:52 AM, Alf P. Steinbach /Usenet wrote:
>> [Cross-posted comp.lang.python and comp.lang.c++]
>>
>> I lack experience with shared libraries in *nix and so I need to ask...
>>
>> This is about "cppy", some support for writing Python extensions in C++
>> that I just started on (some days ago almost known as "pynis" (not funny
>> after all)).
>>
>> For an extension module it seems that Python requires each routine to be
>> defined as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern
>> "C"' and C++ linkage, using a routine declared as 'static' in a class as
>> a C callback, formally they're two different kinds, and I seem to recall
>> that /some/ C++ compiler balks at that kind of mixing unless specially
>> instructed to allow it. Perhaps it was the Sun compiler?

>
> Yes, it will (correctly) issue a warning.
>
> As the is a bit OT, contact me directly and we can work through it. I
> have had similar fun and games adding PHP modules!


Thanks. I'm mailing you a zip with the code... <g>

The question, of course, whether it works in *nix.


Cheers,

- Alf

--
blog at <url: http://alfps.wordpress.com>
 
Reply With Quote
 
Alf P. Steinbach /Usenet
Guest
Posts: n/a
 
      07-13-2010
* geremy condra, on 09.07.2010 23:43:
> On Fri, Jul 9, 2010 at 5:22 PM, Ian Collins<(E-Mail Removed)> wrote:
>> On 07/10/10 03:52 AM, Alf P. Steinbach /Usenet wrote:
>>>
>>> [Cross-posted comp.lang.python and comp.lang.c++]
>>>
>>> I lack experience with shared libraries in *nix and so I need to ask...
>>>
>>> This is about "cppy", some support for writing Python extensions in C++
>>> that I just started on (some days ago almost known as "pynis" (not funny
>>> after all)).
>>>
>>> For an extension module it seems that Python requires each routine to be
>>> defined as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern
>>> "C"' and C++ linkage, using a routine declared as 'static' in a class as
>>> a C callback, formally they're two different kinds, and I seem to recall
>>> that /some/ C++ compiler balks at that kind of mixing unless specially
>>> instructed to allow it. Perhaps it was the Sun compiler?

>>
>> Yes, it will (correctly) issue a warning.
>>
>> As the is a bit OT, contact me directly and we can work through it. I have
>> had similar fun and games adding PHP modules!

>
> I'd appreciate it if you'd either leave this on-list or cc me in on this, as
> I'm working through a similar issue.


Well, we got no further, but I know of three solutions:

A) Punting: just say that the compiler has to support C++/C function type
mingling.
-> Perhaps the practical solution, but formally unsafe.

B) On the script side of things, delegate all calls to single Mother Of All
C func downcaller that supplies as extra arg an id of the C++ function.
-> Micro-level inefficient but easy to use and formally correct.

C) Let the user define the C linkage function wrappers via macros.
-> Efficient and formally correct but exposes ugly macro names.

I chose (C).

I believe Boost's Python binding uses (A), or perhaps (B).


Cheers,

- Alf

PS: You (the reader) may be wondering, why why why Yet Another Python/C++
binding? Well, because I had this great name for it, "pyni", unfortunately
already in use. But cppy is very different from Boost: Boost is large, cppy is
tiny; Boost has as main goal to expose arbitrary C++ code to Python, automating
argument conversion etc., while with cppy your Python design is exposed to C++
with no enforced arg conversions and such; Boost relies on canned magic,
difficult to subvert when it doesn't do what you want, while with cppy you are
(or, so far, I am) in control; and I suspect that the Boost Python binding,
relying on dynamic registries and stuff, is not all that efficient, while cppy
is as efficient as using the Python C API to create an extension. And besides,
cppy supports national characters in doc strings etc. And I'm Norwegian. So.

--
blog at <url: http://alfps.wordpress.com>
 
Reply With Quote
 
Jonathan Lee
Guest
Posts: n/a
 
      07-13-2010
> Problem (C) is outside the realm of the C++ standard, since the C++ standard
> doesn't support shared libraries, and I've never actually used *nix shared
> libraries so I don't /know/...
>
> Is such dynamic initialization guaranteed?
>


Not guaranteed, though I think there's a combination of dlopen options
and gcc command line parameters that invoke this behavior. See the
second page of

http://www.linuxjournal.com/article/3687

about auto-registration.

Personally, though, it never worked for me :/

--Jonathan
 
Reply With Quote
 
Robert Kern
Guest
Posts: n/a
 
      07-13-2010
On 7/13/10 2:34 AM, Alf P. Steinbach /Usenet wrote:

> PS: You (the reader) may be wondering, why why why Yet Another Python/C++
> binding? Well, because I had this great name for it, "pyni", unfortunately
> already in use. But cppy is very different from Boost: Boost is large, cppy is
> tiny; Boost has as main goal to expose arbitrary C++ code to Python, automating
> argument conversion etc., while with cppy your Python design is exposed to C++
> with no enforced arg conversions and such; Boost relies on canned magic,
> difficult to subvert when it doesn't do what you want, while with cppy you are
> (or, so far, I am) in control; and I suspect that the Boost Python binding,
> relying on dynamic registries and stuff, is not all that efficient, while cppy
> is as efficient as using the Python C API to create an extension. And besides,
> cppy supports national characters in doc strings etc. And I'm Norwegian. So.


Note that Boost is not the only C++ binding out there. You may want to take a
look at the old SCXX library, which appears to be similar in intent:

http://davidf.sjsoft.com/mirrors/mcmillan-inc/scxx.html

matplotlib uses it heavily, and their included copy may include some more recent
bugfixes and enhancements:

http://matplotlib.sourceforge.net/

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 
Reply With Quote
 
Alf P. Steinbach /Usenet
Guest
Posts: n/a
 
      07-13-2010
* Robert Kern, on 13.07.2010 17:16:
> On 7/13/10 2:34 AM, Alf P. Steinbach /Usenet wrote:
>
>> PS: You (the reader) may be wondering, why why why Yet Another Python/C++
>> binding? Well, because I had this great name for it, "pyni", unfortunately
>> already in use. But cppy is very different from Boost: Boost is large, cppy is
>> tiny; Boost has as main goal to expose arbitrary C++ code to Python, automating
>> argument conversion etc., while with cppy your Python design is exposed to C++
>> with no enforced arg conversions and such; Boost relies on canned magic,
>> difficult to subvert when it doesn't do what you want, while with cppy you are
>> (or, so far, I am) in control; and I suspect that the Boost Python binding,
>> relying on dynamic registries and stuff, is not all that efficient, while cppy
>> is as efficient as using the Python C API to create an extension. And besides,
>> cppy supports national characters in doc strings etc. And I'm Norwegian. So.

>
> Note that Boost is not the only C++ binding out there. You may want to
> take a look at the old SCXX library, which appears to be similar in intent:
>
> http://davidf.sjsoft.com/mirrors/mcmillan-inc/scxx.html
>
> matplotlib uses it heavily, and their included copy may include some
> more recent bugfixes and enhancements:
>
> http://matplotlib.sourceforge.net/


Thanks! It seems that SCXX does those things that I've been planning to do but
haven't got around to (wrapping standard Python types), while what it doesn't do
(abstracting away all those tables etc. and mapping Python calls to C++ calls)
is what I've been working on. Which could be a Very Nice combination except that
I'm assuming Py3, while SCXX seems to be Py2 only.


Cheers,

- Alf

--
blog at <url: http://alfps.wordpress.com>
 
Reply With Quote
 
Alf P. Steinbach /Usenet
Guest
Posts: n/a
 
      07-13-2010
* Jonathan Lee, on 13.07.2010 16:41:
>> Problem (C) is outside the realm of the C++ standard, since the C++ standard
>> doesn't support shared libraries, and I've never actually used *nix shared
>> libraries so I don't /know/...
>>
>> Is such dynamic initialization guaranteed?
>>

>
> Not guaranteed, though I think there's a combination of dlopen options
> and gcc command line parameters that invoke this behavior. See the
> second page of
>
> http://www.linuxjournal.com/article/3687
>
> about auto-registration.
>
> Personally, though, it never worked for me :/


Ah, well. Thanks for the info! OK, I'll just have to replace the
auto-registration with some C++ magic. For which I think I'll simply /require/
that the compiler supports mixing of C and C++ linkage, that is, that ...


<code language="Not quite standard C++!">
#include <iostream>

extern "C"
{
typedef int (*Callback)( int );
}

void foo( Callback f ) { std::cout << "foo!" << f( 42 ) << std::endl; }

int a( int ) { return 1; }
extern "C" int b( int ) { return 2; }

int main()
{
foo( a ); // Unholy Mix of C++ and C linkage, formally not OK.
foo( b ); // Should be OK with any compiler.
}
</code>


.... compiles, and works.


Cheers, & thanks,

- Alf

--
blog at <url: http://alfps.wordpress.com>
 
Reply With Quote
 
sturlamolden
Guest
Posts: n/a
 
      07-13-2010
On 9 Jul, 17:52, "Alf P. Steinbach /Usenet" <alf.p.steinbach
(E-Mail Removed)> wrote:

> For an extension module it seems that Python requires each routine to be defined
> as 'extern "C"'.


That is strange. PyMethodDef is just a jump table. So why should
'extern "C"' matter?

Good luck on re-inventing the wheel (you've probably heared about
Swig, SIP, Boost.Python, PyCXX, scipy.weave and Cython...)

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Cpp + Python: static data dynamic initialization in *nix shared lib? Alf P. Steinbach /Usenet C++ 10 07-13-2010 09:28 PM
Free online test in C, CPP / Placement papers / CPP,C Interview Questions www.hitechskill.com C++ 0 04-09-2006 10:53 AM
when i compile the cpp file(cmdargs.cpp) int main(int argc, wchar_t* argv[]) Vinu C++ 9 05-05-2005 04:11 AM
Method inlined in source1.cpp and called in source2.cpp Alex Vinokur C++ 7 11-15-2004 09:14 PM
What is better /standard for creating files. a cpp file with header or cpp and seperate file for header DrUg13 C++ 1 02-10-2004 09:20 AM



Advertisments