«

I'm having a hard time getting the hang of Python's package/module
scheme. I'd like to find out what's considered best practice when
dealing with the scenario illustrated below.

The quick description of the problem is: how can I have two nested
modules, spam.ham and spam.ham.eggs?

Suppose I have a module (I'm not even sure this is the right word)
called spam.ham, so I start out with the following file structure:

spam/
|-- ham.py
`-- __init__.py

With this arrangement, the line

import spam.ham

....in client code works as expected.

But now suppose that I want to factor out some code in spam/ham.py
to a helper module. (The reason behind factoring out this new
module is to "declutter" spam/ham.py, and improve its readibility.)
My instinct (from my Perl past) is to put this factored-out code
in a file spam/ham/eggs.py, i.e. to create the "nested" module
spam.ham.eggs, which requires expanding the tree as follows

spam/
|-- ham/
| |-- eggs.py
| `-- __init__.py
|-- ham.py
`-- __init__.py

....and adding the following spam/ham.py to

# spam/ham.py
from . import eggs

This doesn't work so well, because now spam/ham.py is not read.
It seems that adding the spam/ham directory, or maybe adding the
file spam/ham/__init__.py, causes spam/ham.py to be overlooked.


Clearly, I'm not playing this game right...

What is considered "best practice" for the use case sketched above?
Should I, e.g. rename the directory spam/ham something like spam/ham_
and refer to the helper module as spam.ham_.eggs? Or is some other
convention preferred?

I consulted PEP 8, but besides recommending "short, all-lowercase
names" for modules, it gives little guidance on the situation
described above.

TIA!

kynn

On Tue, Sep 1, 2009 at 11:58 AM, kj<> wrote:
>
>
>
> I'm having a hard time getting the hang of Python's package/module
> scheme. *I'd like to find out what's considered best practice when
> dealing with the scenario illustrated below.
>
> The quick description of the problem is: how can I have two nested
> modules, spam.ham and spam.ham.eggs?
>
> Suppose I have a module (I'm not even sure this is the right word)
> called spam.ham, so I start out with the following file structure:
>
> *spam/
> *|-- ham.py
> *`-- __init__.py
>
> With this arrangement, the line
>
> import spam.ham
>
> ...in client code works as expected.
>
> But now suppose that I want to factor out some code in spam/ham.py
> to a helper module. *(The reason behind factoring out this new
> module is to "declutter" spam/ham.py, and improve its readibility.)
> My instinct (from my Perl past) is to put this factored-out code
> in a file spam/ham/eggs.py, i.e. to create the "nested" module
> spam.ham.eggs, which requires expanding the tree as follows
>
> *spam/
> *|-- ham/
> *| * |-- eggs.py
> *| * `-- __init__.py
> *|-- ham.py
> *`-- __init__.py
>
> ...and adding the following spam/ham.py to
>
> # spam/ham.py
> from . import eggs
>
> This doesn't work so well, because now spam/ham.py is not read.
> It seems that adding the spam/ham directory, or maybe adding the
> file spam/ham/__init__.py, causes spam/ham.py to be overlooked.
>
>
> Clearly, I'm not playing this game right...
>
> What is considered "best practice" for the use case sketched above?
> Should I, e.g. rename the directory spam/ham something like spam/ham_
> and refer to the helper module as spam.ham_.eggs? *Or is some other
> convention preferred?
>
> I consulted PEP 8, but besides recommending "short, all-lowercase
> names" for modules, it gives little guidance on the situation
> described above.
>

Take everything in ham.py and stick it in ham/__init__.py instead.
This will give you the behavior you're looking for (don't import
spam.ham.__init__. Everything in __init__.py is loaded into spam.ham)
> TIA!
>
> kynn
> --
> http://mail.python.org/mailman/listinfo/python-list
>

In <h7jga8$ijj$> kj <> writes:

>I'm having a hard time getting the hang of Python's package/module
>scheme. I'd like to find out what's considered best practice when
>dealing with the scenario illustrated below.

>The quick description of the problem is: how can I have two nested
>modules, spam.ham and spam.ham.eggs?

Following up my own post...

From inspecting the directory structure of some of the standard
Python modules I infer the following rules:

1. the source for "leaf" modules lives in files named after them
(e.g. if x.y.z is a "leaf" module, its source code is in x/y/z.py)

2. the source for "non-leaf" modules lives in files named __init__.py
(e.g. if x.y is a "non-leaf" module, its source code lives in
the file x/y/__init__.py)

In the examples above, the module x.y is a "non-leaf" module because
there is a module x.y.z.

I.e. the "leaf"-ness of a module depends solely on whether other
modules deeper in the hierarchy are present.

An implication of all this is that if now I wanted to create a new
module x.y.z.w, this means that the previously "leaf"-module x.y.z
would become "non-leaf". In other words, I'd have to:

1. create the new directory x/y/z
2. *rename* the file x/y/z.py to x/y/z/__init__.py
3. create the file x/y/z/w.py to hold the source for the new x.y.z.w
module

Is the above correct? (BTW, to my Perl-pickled brain, step 2 above
is the one that causes most distress... But I think I can cope.)

kynn

On Sep 1, 8:58*am, kj <no.em...@please.post> wrote:
> I'm having a hard time getting the hang of Python's package/module
> scheme. *I'd like to find out what's considered best practice when
> dealing with the scenario illustrated below.
>
> The quick description of the problem is: how can I have two nested
> modules, spam.ham and spam.ham.eggs?
>
> Suppose I have a module (I'm not even sure this is the right word)
> called spam.ham, so I start out with the following file structure:
>
> * spam/
> * |-- ham.py
> * `-- __init__.py
>
> With this arrangement, the line
>
> import spam.ham
>
> ...in client code works as expected.
>
> But now suppose that I want to factor out some code in spam/ham.py
> to a helper module. *(The reason behind factoring out this new
> module is to "declutter" spam/ham.py, and improve its readibility.)
> My instinct (from my Perl past) is to put this factored-out code
> in a file spam/ham/eggs.py, i.e. to create the "nested" module
> spam.ham.eggs, which requires expanding the tree as follows
>
> * spam/
> * |-- ham/
> * | * |-- eggs.py
> * | * `-- __init__.py
> * |-- ham.py
> * `-- __init__.py
>
> ...and adding the following spam/ham.py to
>
> # spam/ham.py
> from . import eggs
>
> This doesn't work so well, because now spam/ham.py is not read.
> It seems that adding the spam/ham directory, or maybe adding the
> file spam/ham/__init__.py, causes spam/ham.py to be overlooked.

There's a hint here.

You can move the contents of ham.py into ham/__init__.py, and it'll
work the way you want, maybe with only a minor change or two.


> Clearly, I'm not playing this game right...
>
> What is considered "best practice" for the use case sketched above?
> Should I, e.g. rename the directory spam/ham something like spam/ham_
> and refer to the helper module as spam.ham_.eggs? *Or is some other
> convention preferred?

The way you were intending is often the approach people use. I doubt
there are any detailed consensus recommendations about this in the
Python community.


> I consulted PEP 8, but besides recommending "short, all-lowercase
> names" for modules, it gives little guidance on the situation
> described above.

Of course it doesn't, PEP 8 is a style guide, not a project
organization guide.


Carl Banks

kj wrote:
> In <h7jga8$ijj$> kj <> writes:
>
>
>>I'm having a hard time getting the hang of Python's package/module
>>scheme. I'd like to find out what's considered best practice when
>>dealing with the scenario illustrated below.
>
>
>>The quick description of the problem is: how can I have two nested
>>modules, spam.ham and spam.ham.eggs?
>
>
> Following up my own post...
>
>>From inspecting the directory structure of some of the standard
> Python modules I infer the following rules:
>
> 1. the source for "leaf" modules lives in files named after them
> (e.g. if x.y.z is a "leaf" module, its source code is in x/y/z.py)
>
> 2. the source for "non-leaf" modules lives in files named __init__.py
> (e.g. if x.y is a "non-leaf" module, its source code lives in
> the file x/y/__init__.py)
>
> In the examples above, the module x.y is a "non-leaf" module because
> there is a module x.y.z.
>
> I.e. the "leaf"-ness of a module depends solely on whether other
> modules deeper in the hierarchy are present.
>
> An implication of all this is that if now I wanted to create a new
> module x.y.z.w, this means that the previously "leaf"-module x.y.z
> would become "non-leaf". In other words, I'd have to:
>
> 1. create the new directory x/y/z
> 2. *rename* the file x/y/z.py to x/y/z/__init__.py
> 3. create the file x/y/z/w.py to hold the source for the new x.y.z.w
> module
>
> Is the above correct? (BTW, to my Perl-pickled brain, step 2 above
> is the one that causes most distress... But I think I can cope.)
>
> kynn

Looking at the layout of the (most excellent!-) xlrd package, the bulk
of the code is in the __init__.py file, and other supporting code is the
same directory, accessed in __init__ with normal imports.

I also am unclear on when it's best to have supporting files in the same
directory versus a subdirectory... perhaps it is that flat is better
than nested?

~Ethan~

kj wrote:
>
>
> But now suppose that I want to factor out some code in spam/ham.py
> to a helper module. (The reason behind factoring out this new
> module is to "declutter" spam/ham.py, and improve its readibility.)
> My instinct (from my Perl past) is to put this factored-out code
> in a file spam/ham/eggs.py, i.e. to create the "nested" module
> spam.ham.eggs, which requires expanding the tree as follows
>
> spam/
> |-- ham/
> | |-- eggs.py
> | `-- __init__.py
> |-- ham.py
> `-- __init__.py
>
> ...and adding the following spam/ham.py to
>
> # spam/ham.py
> from . import eggs
>
> This doesn't work so well, because now spam/ham.py is not read.
> It seems that adding the spam/ham directory, or maybe adding the
> file spam/ham/__init__.py, causes spam/ham.py to be overlooked.
>
>
> Clearly, I'm not playing this game right...

One way is the __init__.py answer you have already. Another is to put
eggs.py (or _eggs.py to indicate that it is private) in spam and skip
the ham directory.

spam/
__init__.py
ham.py
_eggs.py

tjr

> An implication of all this is that if now I wanted to create a new
> module x.y.z.w, this means that the previously "leaf"-module x.y.z
> would become "non-leaf". In other words, I'd have to:
>
> 1. create the new directory x/y/z
> 2. *rename* the file x/y/z.py to x/y/z/__init__.py
> 3. create the file x/y/z/w.py to hold the source for the new x.y.z.w
> module

With regard to point 2 -- would it be possible to just move z.py into
x/y/z, and put 'from z import *' into x/y/z/__init__.py, for the same
effect? Or is that not a good idea?

On Tue, 01 Sep 2009 09:53:08 -0700, kj <> wrote:

> In <h7jga8$ijj$> kj <> writes:
>
>> I'm having a hard time getting the hang of Python's package/module
>> scheme. I'd like to find out what's considered best practice when
>> dealing with the scenario illustrated below.
>
>> The quick description of the problem is: how can I have two nested
>> modules, spam.ham and spam.ham.eggs?
>
> Following up my own post...
>
>> From inspecting the directory structure of some of the standard
> Python modules I infer the following rules:
>
> 1. the source for "leaf" modules lives in files named after them
> (e.g. if x.y.z is a "leaf" module, its source code is in x/y/z.py)
>
> 2. the source for "non-leaf" modules lives in files named __init__.py
> (e.g. if x.y is a "non-leaf" module, its source code lives in
> the file x/y/__init__.py)
>
> In the examples above, the module x.y is a "non-leaf" module because
> there is a module x.y.z.
>
> I.e. the "leaf"-ness of a module depends solely on whether other
> modules deeper in the hierarchy are present.
>
> An implication of all this is that if now I wanted to create a new
> module x.y.z.w, this means that the previously "leaf"-module x.y.z
> would become "non-leaf". In other words, I'd have to:
>
> 1. create the new directory x/y/z
> 2. *rename* the file x/y/z.py to x/y/z/__init__.py
> 3. create the file x/y/z/w.py to hold the source for the new x.y.z.w
> module
>
> Is the above correct? (BTW, to my Perl-pickled brain, step 2 above
> is the one that causes most distress... But I think I can cope.)
>
> kynn



--
Rami Chowdhury
"Never attribute to malice that which can be attributed to stupidity" --
Hanlon's Razor
408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD)

In <mailman.809.1251832870.2854.python-> "Rami Chowdhury" <> writes:

>> An implication of all this is that if now I wanted to create a new
>> module x.y.z.w, this means that the previously "leaf"-module x.y.z
>> would become "non-leaf". In other words, I'd have to:
>>
>> 1. create the new directory x/y/z
>> 2. *rename* the file x/y/z.py to x/y/z/__init__.py
>> 3. create the file x/y/z/w.py to hold the source for the new x.y.z.w
>> module

>With regard to point 2 -- would it be possible to just move z.py into
>x/y/z, and put 'from z import *' into x/y/z/__init__.py, for the same
>effect? Or is that not a good idea?

I think in this case what you would need is something like 'from
...z import *', but this does not work either; one gets the error
"SyntaxError: 'import *' not allowed with 'from .'".

A third solution would be to change step 2 above to this:

2. create the *symlink* x/y/z/__init__.py --> ../z.py

This certainly would make things clearer-looking to me. Having
all these files called the same thing, __init__.py, but having
completely different meaning, makes my head hurt... In contrast,
the __init__.py *symlinks* would be there just to keep python happy;
the real content is where I'd expect it.

I ran a simple test of this idea and it works, but there may be
some subtle reasons for not doing it... Or maybe it's not a good
idea simply because it is not the Python way, and I'd agree that
this would be reason enough to avoid it.

kynn