Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Module/package hierarchy and its separation from file structure

Reply
Thread Tools

Module/package hierarchy and its separation from file structure

 
 
Ben Finney
Guest
Posts: n/a
 
      01-26-2008
Carl Banks <(E-Mail Removed)> writes:

> On Jan 25, 6:45 pm, Ben Finney <(E-Mail Removed)>
> wrote:
> > "Gabriel Genellina" <(E-Mail Removed)> writes:
> > > You can also put, in animal/__init__.py:
> > > from monkey import Monkey
> > > and now you can refer to it as org.lib.animal.Monkey, but keep the
> > > implementation of Monkey class and all related stuff into
> > > .../animal/monkey.py

> >
> > This (as far as I can understand) is exactly the solution the
> > original poster desired to "shoot down", for reasons I still don't
> > understand.

>
> The solution is to modify the class's __module__ attribute as well as
> importing it, as I've already pointed out:
>
> from org.lib.animal.monkey import Monkey
> Monkey.__module__ = 'org.lib.animal'


Thanks, that makes it clear.

> This should be enough to satisfy the OP's requirements, at least for
> classes, without softening the one-to-one module-to-file
> relationship, or using "hacks".
>
> In fact, I'd say this is good practice.


I've not seen that before, but it seems an elegant way to address what
the OP is asking for.

--
\ "Madness is rare in individuals, but in groups, parties, |
`\ nations and ages it is the rule." -- Friedrich Nietzsche |
_o__) |
Ben Finney
 
Reply With Quote
 
 
 
 
Peter Schuller
Guest
Posts: n/a
 
      01-29-2008
> You can also put, in animal/__init__.py:
> from monkey import Monkey
> and now you can refer to it as org.lib.animal.Monkey, but keep the
> implementation of Monkey class and all related stuff into
> .../animal/monkey.py


The problem is that we are now back to the identity problem. The class
won't actually *BE* org.lib.animal.Monkey. Perhaps manipulating
__module__ is enough; perhaps not (for example, what about
sys.modules?). Looks like I'll just live with putting more than I
would like in the same file.

--
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <(E-Mail Removed)>'
Key retrieval: Send an E-Mail to http://www.velocityreviews.com/forums/(E-Mail Removed)
E-Mail: (E-Mail Removed) Web: http://www.scode.org

 
Reply With Quote
 
 
 
 
Peter Schuller
Guest
Posts: n/a
 
      01-29-2008
> You can reassign the class's module:
>
> from org.lib.animal.monkey import Monkey
> Monkey.__module__ = 'org.lib.animal'
>
>
> (Which, I must admit, is not a bad idea in some cases.)


Is there a sense whether this is truly a supported way of doing this,
in terms of not running into various unintended side-effects? One
example would be sys.modules that I mentioned in the previous
post. Another, possibly related, might be interaction with the import
keyword and its implementation.

I will probably have to read up more on the semantics of __import__
and related machinery.

--
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <(E-Mail Removed)>'
Key retrieval: Send an E-Mail to (E-Mail Removed)
E-Mail: (E-Mail Removed) Web: http://www.scode.org

 
Reply With Quote
 
Carl Banks
Guest
Posts: n/a
 
      01-29-2008
On Jan 29, 7:48 am, Peter Schuller <(E-Mail Removed)>
wrote:
> > You can also put, in animal/__init__.py:
> > from monkey import Monkey
> > and now you can refer to it as org.lib.animal.Monkey, but keep the
> > implementation of Monkey class and all related stuff into
> > .../animal/monkey.py

>
> The problem is that we are now back to the identity problem. The class
> won't actually *BE* org.lib.animal.Monkey.


The usage is the same; it works in all cases once you redefine
__module__. Who cares what it really is?


> Perhaps manipulating
> __module__ is enough; perhaps not (for example, what about
> sys.modules?).


It's enough. It satisfies the criteria you listed. sys.modules has
nothing to do with it. Monkey is a class, not a module.

If you set __module__, the only remaining discernable difference is
that the global variables accessed from the Monkey class will be in
org.lib.animal.monkey instead of org.lib.animal. This has no ill
effects when unpickling or instantiating the class from
org.lib.animal.

> Looks like I'll just live with putting more than I
> would like in the same file.


Whatever. ISTM you came here looking for a particular means and not a
particular end. Python already has the power to meet your stated
needs, but you won't use that solution because it's "hacky".
Apparently all you really wanted was the loosened file structure in
the first place.


Carl Banks
 
Reply With Quote
 
Robert Kern
Guest
Posts: n/a
 
      01-29-2008
Carl Banks wrote:
> On Jan 29, 7:48 am, Peter Schuller <(E-Mail Removed)>
> wrote:
>>> You can also put, in animal/__init__.py:
>>> from monkey import Monkey
>>> and now you can refer to it as org.lib.animal.Monkey, but keep the
>>> implementation of Monkey class and all related stuff into
>>> .../animal/monkey.py

>> The problem is that we are now back to the identity problem. The class
>> won't actually *BE* org.lib.animal.Monkey.

>
> The usage is the same; it works in all cases once you redefine
> __module__. Who cares what it really is?


The inspect module.

[animals]$ ls
animals
[animals]$ rm animals/*.pyc
[animals]$ ls
animals
[animals]$ ls animals
__init__.py monkey.py
[animals]$ cat animals/monkey.py
class Monkey(object):
pass
[animals]$ cat animals/__init__.py
from animals.monkey import Monkey
Monkey.__module__ = 'animals'
[animals]$ python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from animals import Monkey
>>> import inspect
>>> inspect.getsource(Monkey)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/inspect.py",
line 629, in getsource
lines, lnum = getsourcelines(object)
File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/inspect.py",
line 618, in getsourcelines
lines, lnum = findsource(object)
File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/inspect.py",
line 494, in findsource
raise IOError('could not find class definition')
IOError: could not find class definition
>>>


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 
Reply With Quote
 
Peter Schuller
Guest
Posts: n/a
 
      01-30-2008
>> The problem is that we are now back to the identity problem. The class
>> won't actually *BE* org.lib.animal.Monkey.

>
> The usage is the same; it works in all cases once you redefine
> __module__. Who cares what it really is?


The cases I listed were just examples. My point was that I wanted it
to *be* the right class, to avoid unintended consequences. If I knew
what all those possible consequences were, there would not be a
problem to begin with.

The other follow-up to your E-Mail points out a possible problem for
example. I would not have come up with that, but that does not mean
the effect does not exist. And committing to a solution that "seems to
work", only to break massively for some particular use case in the
future, is exactly why I don't want a "hack" for a solution.

I don't know Python internals enough to state of believe with any
authority wither, let's say, stomping __module__ and hacking
sys.modules would be enough to *truly* do it correctly in a proper way
such that it is entirely transparent. This is why I care about whether
it truly changes the real identity of the class; it's not about
satisfying my particular list of examples (because they *were* just
examples).

> Whatever. ISTM you came here looking for a particular means and not a
> particular end.


My particular preferred end is to be able to separate file hierarchy
from module hierarchy without causing unforseen consequences. This was
the stated goal all along.

> Python already has the power to meet your stated
> needs, but you won't use that solution because it's "hacky".
> Apparently all you really wanted was the loosened file structure in
> the first place.


Yes, or failing that an alternative that mitigates the problem. And it
*is* hacky, in my opinion, if things break as a result of it (such as
the other poster's inspect example).

--
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <(E-Mail Removed)>'
Key retrieval: Send an E-Mail to (E-Mail Removed)
E-Mail: (E-Mail Removed) Web: http://www.scode.org

 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      01-30-2008
On Tue, 29 Jan 2008 13:44:33 -0600, Robert Kern wrote:

> Carl Banks wrote:
>> On Jan 29, 7:48 am, Peter Schuller <(E-Mail Removed)> wrote:
>>>> You can also put, in animal/__init__.py:
>>>> from monkey import Monkey
>>>> and now you can refer to it as org.lib.animal.Monkey, but keep the
>>>> implementation of Monkey class and all related stuff into
>>>> .../animal/monkey.py
>>> The problem is that we are now back to the identity problem. The class
>>> won't actually *BE* org.lib.animal.Monkey.

>>
>> The usage is the same; it works in all cases once you redefine
>> __module__. Who cares what it really is?

>
> The inspect module.


[snip example]

I call that a bug in the inspect module. In fact, looking at the source
for the findsource() function, I can see no fewer than two bugs, just in
the way it handles classes:

(1) it assumes that the only way to create a class is with a class
statement, which is wrong; and

(2) it assumes that the first occurrence of "class <name>" must be the
correct definition, which is also wrong.


It isn't hard to break the inspect module. Here's an example:


>>> import broken
>>> import inspect
>>> lines, lineno = inspect.findsource(broken.Parrot)
>>> lines[lineno]

'class Parrot which will be defined later.\n'
>>>
>>> lines, lineno = inspect.findsource(broken.Wensleydale)
>>> lines[lineno]

'class Wensleydale: # THIS IS GONE\n'

Here's the source of broken.py:


$ cat broken.py
"""Here is a doc string, where I happen to discuss the
class Parrot which will be defined later.
"""
class Parrot:
pass

class Wensleydale: # THIS IS GONE
pass

del Wensleydale
class Wensleydale(object): # but this exists
pass



It isn't often that I would come right out and say that part of the
Python standard library is buggy, but this is one of those cases.


--
Steven
 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      01-30-2008
On Tue, 29 Jan 2008 06:48:59 -0600, Peter Schuller wrote:

>> You can also put, in animal/__init__.py:
>> from monkey import Monkey
>> and now you can refer to it as org.lib.animal.Monkey, but keep the
>> implementation of Monkey class and all related stuff into
>> .../animal/monkey.py

>
> The problem is that we are now back to the identity problem. The class
> won't actually *BE* org.lib.animal.Monkey.


It what sense will it not be? Why do you care so much about where the
source code for Monkey is defined? If you actually want to read the
source, you might need to follow the chain from "animal", see that Monkey
is imported from "monkey", and go look at that. But the rest of the time,
why would you care?

There is a very good reason to care *in practice*: if there is code out
there that assumes that the source code from Monkey is in the file it was
found in. In practice, you might be stuck needing to work around that.
But that's not a good reason to care *in principle*. In principle, the
actual location of the source code should be an implementation detail of
which we care nothing. It's possible that the source for Monkey doesn't
exist *anywhere*.

It is important to deal with buggy tools. But the correct way to do so is
to fix the bugs, not to throw away perfectly good abstractions.



--
Steven
 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      01-30-2008
On 30 ene, 12:00, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:

> I call that a bug in the inspect module. In fact, looking at the source
> for the findsource() function, I can see no fewer than two bugs, just in
> the way it handles classes:
>
> (1) it assumes that the only way to create a class is with a class
> statement, which is wrong; and
>
> (2) it assumes that the first occurrence of "class <name>" must be the
> correct definition, which is also wrong.


Yes, it's broken. But I'm afraid that's the only available thing to
do.
Python stores filename and line number information in code objects
(only). If you have a reference to any code object (a method, a
function, a traceback...) inspect can use it to retrieve that
information.
Once a class is defined, there is no code object attached to it. (The
class statement is executed when the module is loaded and initialized,
but that code object is discarded afterwards because it's not required
anymore).
If you *know* that a certain method is defined in a class, you can use
it to find the real module. But in general, there is nothing to start
with.
I'm eagerly waiting for someone to come and say I'm wrong...

--
Gabriel Genellina
 
Reply With Quote
 
Carl Banks
Guest
Posts: n/a
 
      01-30-2008
On Jan 30, 4:31 am, Peter Schuller <(E-Mail Removed)>
wrote:
> I don't know Python internals enough to state of believe with any
> authority wither, let's say, stomping __module__ and hacking
> sys.modules would be enough to *truly* do it correctly in a proper way
> such that it is entirely transparent. This is why I care about whether
> it truly changes the real identity of the class; it's not about
> satisfying my particular list of examples (because they *were* just
> examples).


Well, all I will say is that many people on this list, myself
included, do know Python internals, and we use the method we've been
suggesting here, without problems.

I think you're slipping to a level of paranoia that's more harmful
that helpful now.


The ironic thing is, breaking the one-to-one module-to-file
relationship is more likely to have "unintended consequences", by a
very large margin. Python has always been one-to-one module-to-file
(excepting modules built into the interpretter), and many codes and
tools have come to depend on it.


Carl Banks
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Its a bird, its a plane, its.. um, an Attribute based System? thunk Ruby 14 04-03-2010 10:08 AM
Its a bird, its a plane, its.. um, an Attribute based System? thunk Ruby 0 04-01-2010 10:25 PM
Its a bird, its a plane, no ummm, its a Ruide thunk Ruby 1 03-30-2010 11:10 AM
What is the structure hierarchy here? Doru Roman ASP .Net Datagrid Control 3 02-14-2006 09:15 PM
Input form for creating hierarchy structure Marco Alting ASP General 1 08-14-2003 08:19 PM



Advertisments