private data stashed in local/global execution context of PyEval_EvalCode disappears down the execution stack
 

i naively created execution context:
PyObject *execcontext = PyDict_New();
stuffed a handle in it:
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
int st= PyDict_SetItemString(res, "interp", ih);

and later on in a function for a module that i defined
expected to extract that handle:
PyObject *dict = PyEval_GetGlobals(); // or GetLocals
depending on where
// execcontext ended up in PyEval_EvalCode
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(PyCObject_Check(co));
but i get null co either way and the dict does not match execcontext
that is passed into PyEval_EvalCode.

any ideas how to fix this?

Re: private data stashed in local/global execution context ofPyEval_EvalCode disappears down the execution stack
 

En Tue, 06 Nov 2007 23:25:17 -0300, <sndive@gmail.com> escribió:

> i naively created execution context:
> PyObject *execcontext = PyDict_New();
> stuffed a handle in it:
> PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
> int st= PyDict_SetItemString(res, "interp", ih);

What's `res`?
One should make a lot of assumptions about your code because it's not
complete. Please post a minimal complete example showing your problem.

--
Gabriel Genellina

Re: private data stashed in local/global execution context of PyEval_EvalCode disappears down the execution stack
 

On Nov 9, 5:36 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> En Tue, 06 Nov 2007 23:25:17 -0300, <snd...@gmail.com> escribió:
>
> > i naively created execution context:
> > PyObject *execcontext = PyDict_New();
> > stuffed a handle in it:
> > PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
> > int st= PyDict_SetItemString(res, "interp", ih);
>
> What's `res`?

duh!
it should be execcontext:
int st= PyDict_SetItemString(execcontext, "interp", ih);
then i'd
PyEval_EvalCode(somecallable, execcontext, g_maindict);
where g_maindict is a dictionary from __main__ module
....

later as part of PyEval_EvalCode
a method in a module extending python is executed by python runtime
and i want to get "interp" assuming it would be in the global context:
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
but apparently PyEval_GetGlobals(); returns the dictionary
that has nothing to do with the execcontext passed into
PyEval_EvalCode higher up on the call stack
and i do not know how to get to the execcontext dictionary that
i passed in PyEval_EvalCode.
If each module has its own globals what the point of passing
global and local namespace into PyEval_EvalCode?

> One should make a lot of assumptions about your code because it's not
> complete. Please post a minimal complete example showing your problem.
>
It's a rather large program. My assumption was that just posting the
snippet around the call site and the callee pathetic attempt
to extract interp would be sufficient :(

Re: private data stashed in local/global execution context ofPyEval_EvalCode disappears down the execution stack
 

On Nov 13, 2007 3:18 PM, <sndive@gmail.com> wrote:
> On Nov 9, 5:36 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> > En Tue, 06 Nov 2007 23:25:17 -0300, <snd...@gmail.com> escribió:

>
> > One should make a lot of assumptions about your code because it's not
> > complete. Please post a minimal complete example showing your problem.
> >
> It's a rather large program. My assumption was that just posting the
> snippet around the call site and the callee pathetic attempt
> to extract interp would be sufficient :(
>

The creation of a minimal runnable sample is a fantastic way to find
any bugs in your code, and has the benefit of (if the bug is not in
your code) giving other people a simple way to recreate the bug. If I
were to check this (and I'm not, but I would if you'd posted runnable
code) I'll have to write the code myself from your textual
description. Then, if the code works, I'll have to post the code that
I wrote as well as my negative response, and go through several back
and forths trying to isolate any differences between what I wrote and
what you wrote but didn't show. That's way more work than I'm willing
to do to solve someone else's problem.

In my experience, creating a minimal sample that demonstrates the bug
will lead you to the actual bug more than half the time. That's a lot
of time (yours and other peoples) that can be saved if you do it.

Re: private data stashed in local/global execution context of PyEval_EvalCode disappears down the execution stack
 

On Nov 13, 3:31 pm, "Chris Mellon" <arka...@gmail.com> wrote:
> On Nov 13, 2007 3:18 PM, <snd...@gmail.com> wrote:
>
> > On Nov 9, 5:36 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> > > En Tue, 06 Nov 2007 23:25:17 -0300, <snd...@gmail.com> escribió:
>
> > > One should make a lot of assumptions about your code because it's not
> > > complete. Please post a minimal complete example showing your problem.
>
> > It's a rather large program. My assumption was that just posting the
> > snippet around the call site and the callee pathetic attempt
> > to extract interp would be sufficient :(
>
> The creation of a minimal runnable sample is a fantastic way to find
> any bugs in your code, and has the benefit of (if the bug is not in
> your code) giving other people a simple way to recreate the bug. If I
> were to check this (and I'm not, but I would if you'd posted runnable
> code) I'll have to write the code myself from your textual
> description. Then, if the code works, I'll have to post the code that
> I wrote as well as my negative response, and go through several back
> and forths trying to isolate any differences between what I wrote and
> what you wrote but didn't show. That's way more work than I'm willing
> to do to solve someone else's problem.
>
> In my experience, creating a minimal sample that demonstrates the bug
> will lead you to the actual bug more than half the time. That's a lot
> of time (yours and other peoples) that can be saved if you do it.

working on a smaller example. i could not get pyNode_root invoked yet
and
PyRun_String("import node\nprint node.root()\n",
Py_file_input,
exec, g_maindict);
triggers Py_CompileString
failure with
Traceback (most recent call last):
File "<string>", line 1, in ?
ImportError: __import__ not found

without RyRun_String call program runs but the pyNode_root is not
invoked

#undef _POSIX_C_SOURCE
#include <Python.h>

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif

PyObject *g_mainmod;
PyObject *g_maindict;
bool worked = false;
static
PyObject *
pyNode_root(PyObject *self, PyObject *args)
{
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(PyCObject_Check(co));
void *interp = PyCObject_AsVoidPtr(co);
assert(interp);
// ...
printf("root() worked\n");
worked=true;
return 0;
}

static PyMethodDef module_methods[] = {
/* no need to create pyNode from python programs
{"new", pyNode_new, METH_VARARGS,
PyDoc_STR("new() -> new Node object")},
*/
{"root", pyNode_root, METH_VARARGS,
PyDoc_STR("root('dm') -> wrapper for the rootnode")},

{NULL} /* Sentinel */
};

int
main()
{
Py_Initialize();

g_mainmod = PyImport_AddModule("__main__");
assert(g_mainmod);
g_maindict = PyModule_GetDict(g_mainmod);
assert(g_maindict);
Py_INCREF(g_maindict); // it was a borrowed reference

PyObject* m = Py_InitModule("node", module_methods);
if (m == NULL)
return 1;

PyObject *exec = PyDict_New();
void *handle = (void*)0xdeadc0ed;
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
int st= PyDict_SetItemString(exec, "interp", ih);
assert(!st);


PyRun_String("import node\nprint node.root()\n",
Py_file_input,
exec, g_maindict);

PyObject *mScriptHandle=
Py_CompileString("import node\nprint node.root()\n",
"comp", Py_file_input);
if(!mScriptHandle) {
PyErr_Print();
return 2;
}

PyObject *res = PyEval_EvalCode((PyCodeObject*)mScriptHandle,
exec, g_maindict);

assert(res=Py_None); // compiled as file_input
assert(worked);
}

Re: private data stashed in local/global execution context of PyEval_EvalCode disappears down the execution stack
 

On Nov 9, 7:36 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> En Tue, 06 Nov 2007 23:25:17 -0300, <snd...@gmail.com> escribió:
>
> > i naively created execution context:
> > PyObject *execcontext = PyDict_New();
> > stuffed a handle in it:
> > PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
> > int st= PyDict_SetItemString(res, "interp", ih);
>
> What's `res`?
> One should make a lot of assumptions about your code because it's not
> complete. Please post a minimal complete example showing your problem.
>
> --
> Gabriel Genellina

for some reason i can't get compilestring/run_string to work in my
example.
compile the code below
and from the command line type

import node
print node.root()

you'd get:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1218588544 (LWP 13539)]
0x08048e37 in pyNode_root (self=0x0, args=0xb759d02c) at main.cpp:17
17 assert(PyCObject_Check(co));
(gdb) p co
$1 = (PyObject *) 0x0

because interp is not on globals

#undef _POSIX_C_SOURCE
#include <Python.h>

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif

PyObject *g_mainmod;
PyObject *g_maindict;
bool worked = false;
static
PyObject *
pyNode_root(PyObject *self, PyObject *args)
{
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(PyCObject_Check(co));
void *interp = PyCObject_AsVoidPtr(co);
assert(interp);
// ...
printf("root() worked\n");
worked=true;
return 0;
}

static PyMethodDef module_methods[] = {
/* no need to create pyNode from python programs
{"new", pyNode_new, METH_VARARGS,
PyDoc_STR("new() -> new Node object")},
*/
{"root", pyNode_root, METH_VARARGS,
PyDoc_STR("root('dm') -> wrapper for the rootnode")},

{NULL} /* Sentinel */
};

int
main()
{
Py_Initialize();

g_mainmod = PyImport_AddModule("__main__");
assert(g_mainmod);
g_maindict = PyModule_GetDict(g_mainmod);
assert(g_maindict);
Py_INCREF(g_maindict); // it was a borrowed reference

PyObject* m = Py_InitModule("node", module_methods);
if (m == NULL)
return 1;

PyObject *exec = PyDict_New();
void *handle = (void*)0xdeadc0ed;
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
//Py_INCREF(ih);
int st= PyDict_SetItemString(exec, "interp", ih);
assert(!st);

PyRun_InteractiveLoop(stdin, "<stdin>");
}

Re: private data stashed in local/global execution context of PyEval_EvalCode disappears down the execution stack
 

On Nov 13, 3:31 pm, "Chris Mellon" <arka...@gmail.com> wrote:
> On Nov 13, 2007 3:18 PM, <snd...@gmail.com> wrote:
>
> > On Nov 9, 5:36 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> > > En Tue, 06 Nov 2007 23:25:17 -0300, <snd...@gmail.com> escribió:
>
> > > One should make a lot of assumptions about your code because it's not
> > > complete. Please post a minimal complete example showing your problem.
>
> > It's a rather large program. My assumption was that just posting the
> > snippet around the call site and the callee pathetic attempt
> > to extract interp would be sufficient :(
>
> The creation of a minimal runnable sample is a fantastic way to find
> any bugs in your code, and has the benefit of (if the bug is not in
> your code) giving other people a simple way to recreate the bug. If I
> were to check this (and I'm not, but I would if you'd posted runnable
> code) I'll have to write the code myself from your textual
> description. Then, if the code works, I'll have to post the code that
> I wrote as well as my negative response, and go through several back
> and forths trying to isolate any differences between what I wrote and
> what you wrote but didn't show. That's way more work than I'm willing
> to do to solve someone else's problem.
>
> In my experience, creating a minimal sample that demonstrates the bug
> will lead you to the actual bug more than half the time. That's a lot
> of time (yours and other peoples) that can be saved if you do it.


save as testimp.py:
import node

print "import test worked"

def runtest():
print "runtest()!!!"
res = node.root()

the program reproducing the problem:
#undef _POSIX_C_SOURCE
#include <Python.h>

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif

PyObject *g_mainmod;
PyObject *g_maindict;
bool worked = false;
static
PyObject *
pyNode_root(PyObject *self, PyObject *args)
{
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(PyCObject_Check(co));
void *interp = PyCObject_AsVoidPtr(co);
assert(interp);
// ...
printf("root() worked\n");
worked=true;
return 0;
}

static PyMethodDef module_methods[] = {
/* no need to create pyNode from python programs
{"new", pyNode_new, METH_VARARGS,
PyDoc_STR("new() -> new Node object")},
*/
{"root", pyNode_root, METH_VARARGS,
PyDoc_STR("root('dm') -> wrapper for the rootnode")},

{NULL} /* Sentinel */
};

int
main()
{
Py_Initialize();
PyRun_SimpleString("import sys\n"
"sys.path.append('.')\n"
"print ': ',sys.path\n"
);

g_mainmod = PyImport_AddModule("__main__");
assert(g_mainmod);
g_maindict = PyModule_GetDict(g_mainmod);
assert(g_maindict);
Py_INCREF(g_maindict); // it was a borrowed reference

PyObject* m = Py_InitModule("node", module_methods);
if (m == NULL)
return 1;

PyObject *eglobal = PyDict_New();
void *handle = (void*)0xdeadc0ed;
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
int st= PyDict_SetItemString(eglobal, "interp", ih);
assert(!st);

PyObject *import = PyImport_ImportModule("testimp");
if(!import) {
PyErr_Print();
return 3;
}
Py_INCREF(import);
PyModule_AddObject(g_mainmod, "testimp", import);

PyObject *mScriptHandle=
Py_CompileString("testimp.runtest()",
"comp", Py_eval_input);
if(!mScriptHandle) {
if (PyErr_Occurred()) {
PyErr_Print();
}
return 2;
}

PyObject *res = PyEval_EvalCode((PyCodeObject*)mScriptHandle,
eglobal, g_maindict);
if (PyErr_Occurred()) {
PyErr_Print();
}

Py_DECREF(res);
assert(worked);
}

Re: private data stashed in local/global execution context ofPyEval_EvalCode disappears down the execution stack
 

En Tue, 13 Nov 2007 19:59:56 -0300, <sndive@gmail.com> escribió:

> working on a smaller example. i could not get pyNode_root invoked yet
> and
> PyRun_String("import node\nprint node.root()\n",
> Py_file_input,
> exec, g_maindict);

The globals argument should contain (at least) a key "__builtins__"
pointing to the __builtin__ module; else, no builtin functions (like
__import__) will be found. PyEval_Globals (you used it somewhere) returns
a suitable globals argument.

--
Gabriel Genellina

Re: private data stashed in local/global execution context of PyEval_EvalCode disappears down the execution stack
 

On Nov 14, 12:57 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
wrote:
> En Tue, 13 Nov 2007 19:59:56 -0300, <snd...@gmail.com> escribió:
>
> > working on a smaller example. i could not get pyNode_root invoked yet
> > and
> > PyRun_String("import node\nprint node.root()\n",
> > Py_file_input,
> > exec, g_maindict);
>
> The globals argument should contain (at least) a key "__builtins__"
> pointing to the __builtin__ module; else, no builtin functions (like
> __import__) will be found. PyEval_Globals (you used it somewhere) returns
> a suitable globals argument.
>
thank you.
result is the same however:
pyt: main.cpp:17: PyObject* pyNode_root(PyObject*, PyObject*):
Assertion `co' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread -1218617216 (LWP 4807)]
0x00a54eff in raise () from /lib/tls/libc.so.6

with testimp.py:
import node

print "import test worked"

def runtest():
print "runtest()!!!"
res = node.root()

and the code
#undef _POSIX_C_SOURCE
#include <Python.h>

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif

PyObject *g_mainmod;
PyObject *g_maindict;
bool worked = false;
static
PyObject *
pyNode_root(PyObject *self, PyObject *args)
{
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(co);
assert(PyCObject_Check(co));
void *interp = PyCObject_AsVoidPtr(co);
assert(interp);
// ...
printf("root() worked\n");
worked=true;
return 0;
}

static PyMethodDef module_methods[] = {
/* no need to create pyNode from python programs
{"new", pyNode_new, METH_VARARGS,
PyDoc_STR("new() -> new Node object")},
*/
{"root", pyNode_root, METH_VARARGS,
PyDoc_STR("root('dm') -> wrapper for the rootnode")},

{NULL} /* Sentinel */
};

int
main()
{
Py_Initialize();
PyRun_SimpleString("import sys\n"
"sys.path.append('.')\n"
"print ': ',sys.path\n"
);

g_mainmod = PyImport_AddModule("__main__");
assert(g_mainmod);
g_maindict = PyModule_GetDict(g_mainmod);
assert(g_maindict);
Py_INCREF(g_maindict); // it was a borrowed reference

PyObject* m = Py_InitModule("node", module_methods);
if (m == NULL)
return 1;

PyObject *eglobal = PyDict_New();
void *handle = (void*)0xdeadc0ed;
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
int st= PyDict_SetItemString(eglobal, "interp", ih);
assert(!st);
PyObject *builtins = PyEval_GetBuiltins();
st= PyDict_SetItemString(eglobal, "__builtins__", builtins);
assert(!st);

PyObject *import = PyImport_ImportModule("testimp");
if(!import) {
PyErr_Print();
return 3;
}
Py_INCREF(import);
PyModule_AddObject(g_mainmod, "testimp", import);

PyObject *mScriptHandle=
Py_CompileString("testimp.runtest()",
"comp", Py_eval_input);
if(!mScriptHandle) {
if (PyErr_Occurred()) {
PyErr_Print();
}
return 2;
}

PyObject *res = PyEval_EvalCode((PyCodeObject*)mScriptHandle,
eglobal, g_maindict);
if (PyErr_Occurred()) {
PyErr_Print();
}

Py_DECREF(res);
assert(worked);
}

Re: private data stashed in local/global execution context ofPyEval_EvalCode disappears down the execution stack
 

En Wed, 14 Nov 2007 15:08:42 -0300, <sndive@gmail.com> escribió:

> thank you.
> result is the same however:
> pyt: main.cpp:17: PyObject* pyNode_root(PyObject*, PyObject*):
> Assertion `co' failed.

Well, "is the same" in the sense that the code still doesn't do what you
want... But the previous error is gone. (Now I regret having said the
reason it didn't work before: you took my words too literally).

I suggest you first try to write the program in pure Python, even with a
dummy Node. From what I can understand of what you are doing, it appears
you don't fully understand how import works, how modules work, and
namespaces in Python. The C API exposes the same concepts so it's easier
when one knows how to do things in pure Python first. Right now you have
two problems: "what" to do, and "how" to write that using the API. First
try to solve the "what", in Python, and only later move to the second
stage.

--
Gabriel Genellina