Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Strange effect with import

Reply
Thread Tools

Strange effect with import

 
 
Jens Thoms Toerring
Guest
Posts: n/a
 
      12-20-2012
Hi,

I hope that this isn't a stupid question, asked already a
hundred times, but I haven't found anything definitive on
the problem I got bitten by. I have two Python files like
this:

-------- S1.py ------
import random
import S2

class R( object ) :
r = random.random( )

if __name__ == "__main__" :
print R.r
S2.p( )

-------- S2.py ------
import S1

def p( ) :
print S1.R.r

and my expectation was that the static variable 'r' of class
R would be identical when accessed from S1.py and S2.py.
Unfortunately, that isn't the case, the output is different
(and R seems to get instantiated twice).

But when I define R in S2.py instead

-------- S1.py ------
import S2

print S2.R.r
S2.p( )

-------- S2.py ------
import random

class R( object ) :
r = random.random( )

def p( ) :
print R.r

or, alternatively, if I put the defintion of class R into
a third file which I then import from the other 2 files,
things suddenly start to work as expected/ Can someone
explain what's going one here? I found this a bit sur-
prising.

This is, of course, not my "real" code - it would be much
more sensible to pass the number to the function in the
second file as an argument - but is the smallest possinle
program I could come up with that demonstrate the prob-
lem. In my "real" code it's unfortunately not possible
to pass that number to whatever is going to use it in the
other file, I have to simulate a kind of global variable
shared between different files.

Best regards, Jens
--
\ Jens Thoms Toerring ___ http://www.velocityreviews.com/forums/(E-Mail Removed)
\__________________________ http://toerring.de
 
Reply With Quote
 
 
 
 
Dave Angel
Guest
Posts: n/a
 
      12-20-2012
On 12/20/2012 03:39 PM, Jens Thoms Toerring wrote:
> Hi,
>
> I hope that this isn't a stupid question, asked already a
> hundred times, but I haven't found anything definitive on
> the problem I got bitten by. I have two Python files like
> this:
>
> -------- S1.py ------
> import random
> import S2
>
> class R( object ) :
> r = random.random( )
>
> if __name__ == "__main__" :
> print R.r
> S2.p( )
>
> -------- S2.py ------
> import S1


You have a big problem right here. You have two modules importing each
other. Any time you have direct or indirect mutual imports, you have
the potential for trouble.

That trouble gets much worse since you are actually running one of these
as a script. Presumably you're running S1.py as a script. The script's
module object is NOT the same one as the other module S2 gets by
importing S1. Don't do that.

Move the common code into a third module, and import that one from both
places. Then it'll only exist once.



> def p( ) :
> print S1.R.r
>
> and my expectation was that the static variable 'r' of class
> R would be identical when accessed from S1.py and S2.py.
> Unfortunately, that isn't the case, the output is different
> (and R seems to get instantiated twice).
>
> But when I define R in S2.py instead
>
> -------- S1.py ------
> import S2
>
> print S2.R.r
> S2.p( )
>
> -------- S2.py ------
> import random
>
> class R( object ) :
> r = random.random( )
>
> def p( ) :
> print R.r
>
> or, alternatively, if I put the defintion of class R into
> a third file which I then import from the other 2 files,
> things suddenly start to work as expected/ Can someone
> explain what's going one here? I found this a bit sur-
> prising.
>
> This is, of course, not my "real" code - it would be much
> more sensible to pass the number to the function in the
> second file as an argument - but is the smallest possinle
> program I could come up with that demonstrate the prob-
> lem. In my "real" code it's unfortunately not possible
> to pass that number to whatever is going to use it in the
> other file, I have to simulate a kind of global variable
> shared between different files.
>
> Best regards, Jens



--

DaveA

 
Reply With Quote
 
 
 
 
Peter Otten
Guest
Posts: n/a
 
      12-20-2012
Jens Thoms Toerring wrote:

> Hi,
>
> I hope that this isn't a stupid question, asked already a
> hundred times, but I haven't found anything definitive on
> the problem I got bitten by. I have two Python files like
> this:
>
> -------- S1.py ------
> import random
> import S2
>
> class R( object ) :
> r = random.random( )
>
> if __name__ == "__main__" :
> print R.r
> S2.p( )
>
> -------- S2.py ------
> import S1
>
> def p( ) :
> print S1.R.r
>
> and my expectation was that the static variable 'r' of class
> R would be identical when accessed from S1.py and S2.py.
> Unfortunately, that isn't the case, the output is different
> (and R seems to get instantiated twice).
>
> But when I define R in S2.py instead
>
> -------- S1.py ------
> import S2
>
> print S2.R.r
> S2.p( )
>
> -------- S2.py ------
> import random
>
> class R( object ) :
> r = random.random( )
>
> def p( ) :
> print R.r
>
> or, alternatively, if I put the defintion of class R into
> a third file which I then import from the other 2 files,
> things suddenly start to work as expected/


That's the correct approach.

> Can someone
> explain what's going one here? I found this a bit sur-
> prising.


You should never import your program's main module anywhere else in the
program. When Python imports a module it looks it up by the module's name in
the sys.modules cache. For the main script that name will be "__main__"
regardless of the file's actual name, so a subsequent "import S2" will
result in a cache miss and a new module instance.

Similar problems occur when there is a PYTHONPATH pointing into a package
and you have both

import package.module

and

import module

Again you will end up with two module instances, one called
"package.module", the other just "module".

> This is, of course, not my "real" code - it would be much
> more sensible to pass the number to the function in the
> second file as an argument - but is the smallest possinle
> program I could come up with that demonstrate the prob-
> lem. In my "real" code it's unfortunately not possible
> to pass that number to whatever is going to use it in the
> other file, I have to simulate a kind of global variable
> shared between different files.




 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      12-20-2012
On Thu, 20 Dec 2012 20:39:19 +0000, Jens Thoms Toerring wrote:

> Hi,
>
> I hope that this isn't a stupid question, asked already a
> hundred times, but I haven't found anything definitive on the problem I
> got bitten by. I have two Python files like this:
>
> -------- S1.py ------
> import random
> import S2
>
> class R( object ) :
> r = random.random( )
>
> if __name__ == "__main__" :
> print R.r
> S2.p( )
>
> -------- S2.py ------
> import S1
>
> def p( ) :
> print S1.R.r
>
> and my expectation was that the static variable 'r' of class R


The terminology we prefer here is "class attribute", not "static
variable". Attributes are always assigned in dynamic storage, whether
they are per-instance or on the class.



> would be
> identical when accessed from S1.py and S2.py. Unfortunately, that isn't
> the case, the output is different (and R seems to get instantiated
> twice).


You don't instantiate R at all. You only ever refer to the class object,
you never instantiate it to create an instance. What you are actually
seeing is a side-effect of the way Python modules are imported:

- Python modules are instances that are instantiated at import
time, and then cached by module name;

- the module name is *usually* the file name (sans .py extension),
except when you are running it as a script, in which case it
gets set to the special value "__main__" instead.

So the end result is that you actually end up with THREE module objects,
__main__, S2 and S1, even though there are only two module *files*. Both
__main__ and S1 are instantiated from the same source code and contain
the same objects: both have a class called R, with fully-qualified names
__main__.R and S1.R, but they are separate objects.


[...]
> or, alternatively, if I put the defintion of class R into a third file
> which I then import from the other 2 files, things suddenly start to
> work as expected/ Can someone explain what's going one here? I found
> this a bit surprising.


You have a combination of two tricky situations:

* A circular import: module S1 imports S2, and S2 imports S1.

* A .py file, S1.py, being used as both an importable module
and a runnable script.

Circular imports are usually hard to get rid at the best of time.
Combined with the second factor, they can lead to perplexing errors, as
you have just found out.


> This is, of course, not my "real" code - it would be much more sensible
> to pass the number to the function in the second file as an argument -
> but is the smallest possinle program I could come up with that
> demonstrate the problem.


And let me say sincerely, thank you for doing so! You would be amazed how
many people do not make any effort to simplify their problem before
asking for help.


> In my "real" code it's unfortunately not
> possible to pass that number to whatever is going to use it in the
> other file, I have to simulate a kind of global variable
> shared between different files.


Well, I find that hard to believe. "Not convenient"? I could believe
that. "Difficult"? Maybe. "Tricky"? I could even believe that. But "not
possible"? No, I don't believe that it is impossible to pass variables
around as method arguments.



--
Steven
 
Reply With Quote
 
Jens Thoms Toerring
Guest
Posts: n/a
 
      12-20-2012
Thanks a lot to all three of you: that helped me understand
the errors of my ways! You just saved me a few more hours
of head-scratching

A few replies to the questions and comments by Steven:

Steven D'Aprano <(E-Mail Removed)> wrote:
> On Thu, 20 Dec 2012 20:39:19 +0000, Jens Thoms Toerring wrote:
> > and my expectation was that the static variable 'r' of class R


> The terminology we prefer here is "class attribute", not "static
> variable". Attributes are always assigned in dynamic storage, whether
> they are per-instance or on the class.


I'm comimg from C/C++ and that's were my terminology is from,
I know I still have to learn a lot more about Python

<good advice snipped>

> > In my "real" code it's unfortunately not
> > possible to pass that number to whatever is going to use it in the
> > other file, I have to simulate a kind of global variable
> > shared between different files.


> Well, I find that hard to believe. "Not convenient"? I could believe
> that. "Difficult"? Maybe. "Tricky"? I could even believe that. But "not
> possible"? No, I don't believe that it is impossible to pass variables
> around as method arguments.


You are rather likely right and I probably should have written:
"I don't see any way to pass that variable to the object that
is supposed to use it". Perhaps you have an idea how it could
be done correctly when I explain the complete picture: I'm
writing a TCP server, based on SocketServer:

server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)

where ReqHandler is the name of a class derived from
SocketServer.BaseRequestHandler

class ReqHandler(SocketServer.BaseRequestHandler):
...

A new instance of this class is gernerated for each connection
request to the server. In the call that creates the server I can
only specify the name of the class but no arguments to be passed
to it on instantiation - at least I found nothing in the docu-
mentation. On the other hand I need to get some information into
this class and thus the only idea I came up with was to use some
kind of global variable for the purpose. Perhaps there's a much
better way to do that but I haven't found one yet. Or perhaps it
is an omission in the design of SocketServer or (more likely) my
mis-understanding of the documentation (as I wrote I'm relatively
new to Python).
Thnak you and best regards, Jens
--
\ Jens Thoms Toerring ___ (E-Mail Removed)
\__________________________ http://toerring.de
 
Reply With Quote
 
Terry Reedy
Guest
Posts: n/a
 
      12-21-2012
On 12/20/2012 5:52 PM, Jens Thoms Toerring wrote:

> You are rather likely right and I probably should have written:
> "I don't see any way to pass that variable to the object that
> is supposed to use it". Perhaps you have an idea how it could
> be done correctly when I explain the complete picture: I'm
> writing a TCP server, based on SocketServer:
>
> server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)
>
> where ReqHandler is the name of a class derived from
> SocketServer.BaseRequestHandler


You misunderstood the doc. You pass the class, not the name of the class.
From 21.19.4.1. socketserver.TCPServer Example
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)

MyTCPHandler is the actual class. What gets 'passed' at the C level in
CPython is a reference that class that TCPServer can use to call it, but
conceptually, at the Python level, think of it as the class. In the
code, you enter the name without quotes and that expression evaluates to
the (reference to the) class that gets passed.

If the signature required the name, the example would have had
'MyTCPHandler', with the quotes, to pass the name as a string.

Very few builtin functions require names as strings. open('filename'),
somebytes.encode(encoding='encoding-name', errors =
'error-handler-name') are two that come to mind. Notice that these are
situations where requiring a non-string object would be inconvenient at
best.


> class ReqHandler(SocketServer.BaseRequestHandler):
> ...
>
> A new instance of this class is gernerated for each connection
> request to the server. In the call that creates the server I can
> only specify the name of the class but no arguments to be passed


Code those arguments directly into the handle method of your version of
MyTCPhandler. Or if you need to override multiple methods and use the
same values in multiple methods, override __init__ and add self.x =
x-value statements.

--
Terry Jan Reedy

 
Reply With Quote
 
Hans Mulder
Guest
Posts: n/a
 
      12-21-2012
On 20/12/12 23:52:24, Jens Thoms Toerring wrote:
> I'm writing a TCP server, based on SocketServer:
>
> server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)
>
> where ReqHandler is the name of a class derived from
> SocketServer.BaseRequestHandler
>
> class ReqHandler(SocketServer.BaseRequestHandler):
> ...
>
> A new instance of this class is gernerated for each connection
> request to the server. In the call that creates the server I can
> only specify the name of the class but no arguments to be passed
> to it on instantiation - at least I found nothing in the docu-
> mentation.


What happens if instead of a class you pass a function that
takes the same arguments as the SocketServer.BaseRequestHandler
constructor and returns a new instance of your ReqHandler?

That's not quite what the documentaion clls for, but I'd hope
it's close enough.


Maybe something like this:

class ReqHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server, ham, spam)
super(SocketServer, self).__init__(
self, request, client_address, server)
self.ham = ham
self.spam = spam
....

And later:

import functools

server = SocketServer.TCPServer((192.168.1.10, 12345),
functools.partial(ReqHandler, ham="hello", spam=42))

> On the other hand I need to get some information into
> this class and thus the only idea I came up with was to use some
> kind of global variable for the purpose. Perhaps there's a much
> better way to do that but I haven't found one yet. Or perhaps it
> is an omission in the design of SocketServer


I think you could call it a weakness in the design of SocketServer.

Life would be easier if it took as an optional third argument some
sequence that it would pass as extra arguments when it instantiates
the handler instance. Then you wouldn't have to play with functools
(or closures, or global variables) to solve your problems.

> or (more likely) my mis-understanding of the documentation
> (as I wrote I'm relatively new to Python).


From where I sit, it looks like the authors of the SocketServer
module didn't expect subclasses of BaseRequestHandler to need
extra attributes than their base class.

Or maybe they thought everybody knew functools.partial.


Hope this helps,

-- HansM

 
Reply With Quote
 
Jens Thoms Toerring
Guest
Posts: n/a
 
      12-21-2012
Terry Reedy <(E-Mail Removed)> wrote:
> > server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)
> >
> > where ReqHandler is the name of a class derived from
> > SocketServer.BaseRequestHandler


> You misunderstood the doc. You pass the class, not the name of the class.
> From 21.19.4.1. socketserver.TCPServer Example
> server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)


Yes, I meant "the class", but I'm a bit weak on nomenclature in
Python

> > A new instance of this class is gernerated for each connection
> > request to the server. In the call that creates the server I can
> > only specify the name of the class but no arguments to be passed


> Code those arguments directly into the handle method of your version of
> MyTCPhandler. Or if you need to override multiple methods and use the
> same values in multiple methods, override __init__ and add self.x =
> x-value statements.


Sorry, you lost me there: what means "code those arguments
directly into the handle method"? According to the documen-
tation (or at least to my understanding of it the handle()
method is suppose to accept just one argument, 'self'. And
even if I would change the method to accept more arguments
and that wouldnt blow up into my face, where would they be
coming from (and from where would I pass them)?

Best regards, Jens
--
\ Jens Thoms Toerring ___ (E-Mail Removed)
\__________________________ http://toerring.de
 
Reply With Quote
 
Jens Thoms Toerring
Guest
Posts: n/a
 
      12-21-2012
Hans Mulder <(E-Mail Removed)> wrote:
> What happens if instead of a class you pass a function that
> takes the same arguments as the SocketServer.BaseRequestHandler
> constructor and returns a new instance of your ReqHandler?


> That's not quite what the documentaion clls for, but I'd hope
> it's close enough.


Interesting idea - I'm not yet at a level of Python wizardry
that I would dare to do something that's not explicitely bles-
sed be the documentation

> Maybe something like this:


> class ReqHandler(SocketServer.BaseRequestHandler):
> def __init__(self, request, client_address, server, ham, spam)
> super(SocketServer, self).__init__(
> self, request, client_address, server)
> self.ham = ham
> self.spam = spam
> ....


> And later:


> import functools


> server = SocketServer.TCPServer((192.168.1.10, 12345),
> functools.partial(ReqHandler, ham="hello", spam=42))


Ok, that's still way over may head at the moment I will hhave
to read up on functools tomorrow, it's the first time I heard of
it but it looks quite interesting at a first glance.

Thank you for these ideas, I'll need a bit of time to figure out
these new concepts and I don't think I'm up to it tonight any-
more
Best regards. Jens
--
\ Jens Thoms Toerring ___ (E-Mail Removed)
\__________________________ http://toerring.de
 
Reply With Quote
 
Jens Thoms Toerring
Guest
Posts: n/a
 
      12-21-2012
Hans Mulder <(E-Mail Removed)> wrote:
> Maybe something like this:


> class ReqHandler(SocketServer.BaseRequestHandler):
> def __init__(self, request, client_address, server, ham, spam)
> super(SocketServer, self).__init__(
> self, request, client_address, server)
> self.ham = ham
> self.spam = spam
> ....


The only thing I had to change about this was to assign the
additional class variables before calling super() because in
the __init__() method of the base class my overloaded handle()
method is already called which needs those extra variables.

> And later:


> import functools


> server = SocketServer.TCPServer((192.168.1.10, 12345),
> functools.partial(ReqHandler, ham="hello", spam=42))


Thanks a lot, that's now all working perfectly well and I got
rid of those pesky global variables Probably the guys that
wrote the SocketServer module indeed didn't expect people as
dense as me to use their module and thus didn't mention that
passing additional information to a handler object can be done
this way...
Best regards, Jens
--
\ Jens Thoms Toerring ___ (E-Mail Removed)
\__________________________ http://toerring.de
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Import Javascript effect into html body Viabug123 HTML 1 04-21-2006 06:50 AM
strange effect with images? John Salerno HTML 6 02-10-2006 04:35 AM
Strange side effect with events =?Utf-8?B?R2lhbmNhcmxv?= ASP .Net 2 01-13-2005 03:03 PM
java package import, effect on performance? asrahma1111 Java 11 05-01-2004 01:27 PM
Strange effect with PipedOutputStream/PipedInputStream Henning Moll Java 4 11-03-2003 02:09 PM



Advertisments