How to organize Python files in a (relatively) big project
 

Hello there,

I have been programming python for a little while, now. But as I am
beginning to do more complex stuff, I am running into small organization
problems.
It is possible that what I want to obtain is not possible, but I would
like the advice of more experienced python programmers.

I am writing a relatively complex program in python that has now around
40 files.
At first, I had all of my files in one single directory, but now, with
the increasing number of files, it is becoming hard to browse my directory.
So, I would want to be able to divide the files between 8 directory,
according to their purpose. The problem is that it breaks the 'import's
between my files. And besides,AFAIK, there is no easy way to import a
file that is not in a subdirectory of the current file (I suppose I
could adjust the os.path in every file, but that seems not very elegant
to me).
I thought about turning the whole into a package. But I have to change
every 'import module_name' into
'from package_name.sub_directory_name import module_name'
which is a little bit time consuming and not very flexible (if I change
my mind about the subdirectory a file belongs to, I will have to track
again every import to correct them)

So, basically, here is the point: is there an 'elegant' way to keep my
files in separate directory and still be able to import between them
with simple 'import filename'?

And if not, what would be the standard way to organize numerous python
files belonging to the same project?

Thank you,

TokiDoki

Re: How to organize Python files in a (relatively) big project
 

I have this problem myself, and as I am a recent Python convert my
aproach may not be conventional - but as it is working for me I thought
I would share.

First off you need to split your code into logical objects. In my case
I have an obvious client and server object, But within each I have
individual stand alone major objects and so my directory structure
reflects this...

Prog Dir
-- MainApp.py
Bin
__init__.py
--- main.py
--- lib.py
Client
__init__.py
--- main.py
Browser 1
__init__.py
--- main.py
--- B1_file1.py
--- B1_file2.py
Browser 2
__init__.py
--- main.py
--- B2_file1.py
--- B2_file2.py
Server
__init__.py
--- main.py
Server Object 1
__init__.py
--- main.py
--- SO1_file1.py
--- SO1_file2.py
Server Object 2
__init__.py
--- main.py
--- SO2_file1.py
--- SO2_file2.py


Each main.py file is used to call each downstream main.py file be it as
a thread or once only run. The beauty of this idea is that if I want
to completely change say Server Object 2, I can do whatever I like
within the downstream directory as the only outside call is via an
external reference to the main.py file.

If you have a mind, you could have a library file of common apps held
within the bin directory that you can make available to all downstream
modules.

As I say - it only works if you can logically split your program into
independent blocks.

If there is a more Pythonesque way of doing this I be grateful if
someone could share.

HTH

Re: How to organize Python files in a (relatively) big project
 

TokiDoki wrote:

> At first, I had all of my files in one single directory, but now, with
> the increasing number of files, it is becoming hard to browse my
> directory. So, I would want to be able to divide the files between 8
> directory, according to their purpose. The problem is that it breaks
> the 'import's between my files. And besides,AFAIK, there is no
> easy way to import a
> file that is not in a subdirectory of the current file

Remember that the directory where you start the toplevel script is always
included in the sys.path. This means that you can have your structure like
this:

main.py
|
| - - pkg1
| - - pkg2
| - - pkg3

Files in any package can import other packages. The usual way is to do "import
pkgN" and then dereference. Within each package, you will have a __init__.py
which will define the package API (that is, will define those symbols that you
can access from outside the package).

Typically, you only want to import *packages*, not submodules. In other words,
try to not do stuff like "from pkg1.submodule3 import Foo", because this breaks
encapsulation (if you reorganize the structure of pkg1, code will break). So
you'd do "import pgk1" and later "pkg1.Foo", assuming that pkg1.__init__.py
does something like "from submodule3 import Foo". My preferred way is to have
__init__.py just do "from submodules import *", and then each submodule defines
__all__ to specify which are its public symbols. This allow for more
encapsulation (each submodule is able to change what it exports in a
self-contained way, you don't need to modify __init__ as well).

Moreover, the dependence graph between packages shouldn't have loops. For
instance, if pkg3 uses pkg1, pkg1 shouldn't use pkg3. It makes sense to think
of pkg1 as the moral equivalent of a library, which pkg3 uses.
--
Giovanni Bajo

Re: How to organize Python files in a (relatively) big project
 

If I understand you right you need a concept in which you can put the files
of your project where you want, i.e. restructure the nesting of directories
storing your scripts without the problem of breaking the import statements.
This will give you not a solution to any problem you maybe have with
managing a large package but is a problem in itself, so I see that you have
a huge number of options to choose from:

1. best is you get a better structure of the project in your mind, so you
don't need frequent changes by resorting your modules into a various
directories (flat is better than nested). Set a size limit of a single
Python script it must reach before you split it into two separate files, so
that the number of your files decreases to a number you can handle with ease
(this depends on how good you are at it).

2. create e.g. an HTML file from which you can view, edit, execute the
scripts you have. In this file you create your nested structure and have
this way the overview you need. The Python files remain in one directory
storing the entire project. This allows you to keep together what belongs
together, but to work with it as the single files were spread over many
categories (I mean directories :).

3. create a Python script which automatically creates an HTML file giving
you the overview of all your scripts from walking the directories your
project files are stored below and put in each of your scripts a function
which on call returns the actual full path of passed name of the file to
import and updates the Python PATH, so that it can be found (e.g
updatePATH('nameOfTheModuleToImport')

4. start to choose from many tools for project management which will keep
you busy for a while and away from the actual work on your project ...

None of these above can solve the supposed actual problem, that the project
grows beyond the capacity to manage it in mind. I know about this limitation
and I have seen programmers failing on projects with increasing size of
them. Usually the dependencies between the single modules in large projects
get more and more complicated. Each next added module making the project
size larger must work with the entire already existing ones and after a
given size of project is reached it becomes really hard (and for some just
impossible) to add a new module to it making the necessary modifications to
existing ones in order to gain the new functionality. Such modifications
result usually in breaks in other parts of the project and in my eyes no
management system can really help in anticipating it at that point. Every
programmer seems to have a limit of a project size he can manage and in my
opinion no tool for structuring it or getting a better overview is of real
help here. The core of the problem is lack of skills required for managing
large projects and it seems, that not everyone who managed to learn to
program is able to acquire this skills necessary to increase the size of a
project over the by his intellectual capacity given limits. That is why
programmer need to work in teams and have a very good manager able to
coordinate their efforts in order to succeed in creating large applications.
In really large projects it is sometimes necessary to work half a year or
more full time on the existing library of code before becoming capable of
writing a single line of additional one.

Publish your code, let others work with it, try to get responses on it and
you will probably get much more out of this than by asking for ways of
managing it.

I am curious if others on this newsgroup agree with what I said above, so I
would be glad to hear about it.

Claudio



"TokiDoki" <neuromancer@wanadoo.fr> schrieb im Newsbeitrag
news:dj492l$3hf$1@apollon.grec.isp.9tel.net...
> Hello there,
>
> I have been programming python for a little while, now. But as I am
> beginning to do more complex stuff, I am running into small organization
> problems.
> It is possible that what I want to obtain is not possible, but I would
> like the advice of more experienced python programmers.
>
> I am writing a relatively complex program in python that has now around
> 40 files.
> At first, I had all of my files in one single directory, but now, with
> the increasing number of files, it is becoming hard to browse my
directory.
> So, I would want to be able to divide the files between 8 directory,
> according to their purpose. The problem is that it breaks the 'import's
> between my files. And besides,AFAIK, there is no easy way to import a
> file that is not in a subdirectory of the current file (I suppose I
> could adjust the os.path in every file, but that seems not very elegant
> to me).
> I thought about turning the whole into a package. But I have to change
> every 'import module_name' into
> 'from package_name.sub_directory_name import module_name'
> which is a little bit time consuming and not very flexible (if I change
> my mind about the subdirectory a file belongs to, I will have to track
> again every import to correct them)
>
> So, basically, here is the point: is there an 'elegant' way to keep my
> files in separate directory and still be able to import between them
> with simple 'import filename'?
>
> And if not, what would be the standard way to organize numerous python
> files belonging to the same project?
>
> Thank you,
>
> TokiDoki

Re: How to organize Python files in a (relatively) big project
 

Maybe looking at the todays thread ["dynamical" importing] can be helpful
also here.

Claudio
P.S. Below a copy of one of the responses:
:
Joerg Schuster wrote:

> I need to import modules from user defined paths. I.e. I want to do
> something like:
>
> module_dir = sys.argv[1]
>
> my_path = os.path.join(module_dir, 'bin', 'my_module')
>
> from my_path import my_object
>
> Obviously, it doesn't work this way. How would it work?

some alternatives:

- if you want the modules to remain imported:

try:
sys.path.insert(0, os.path.join(module_dir, "bin"))
module = __import__("my_module")
finally:
del sys.path[0]
object = module.my_object

- if you're only interested in the object:

namespace = {}
execfile(os.path.join(module_dir, "bin", "my_module" + ".py"),
namespace)
object = namespace["my_object"]

</F>


"TokiDoki" <neuromancer@wanadoo.fr> schrieb im Newsbeitrag
news:dj492l$3hf$1@apollon.grec.isp.9tel.net...
> Hello there,
>
> I have been programming python for a little while, now. But as I am
> beginning to do more complex stuff, I am running into small organization
> problems.
> It is possible that what I want to obtain is not possible, but I would
> like the advice of more experienced python programmers.
>
> I am writing a relatively complex program in python that has now around
> 40 files.
> At first, I had all of my files in one single directory, but now, with
> the increasing number of files, it is becoming hard to browse my
directory.
> So, I would want to be able to divide the files between 8 directory,
> according to their purpose. The problem is that it breaks the 'import's
> between my files. And besides,AFAIK, there is no easy way to import a
> file that is not in a subdirectory of the current file (I suppose I
> could adjust the os.path in every file, but that seems not very elegant
> to me).
> I thought about turning the whole into a package. But I have to change
> every 'import module_name' into
> 'from package_name.sub_directory_name import module_name'
> which is a little bit time consuming and not very flexible (if I change
> my mind about the subdirectory a file belongs to, I will have to track
> again every import to correct them)
>
> So, basically, here is the point: is there an 'elegant' way to keep my
> files in separate directory and still be able to import between them
> with simple 'import filename'?
>
> And if not, what would be the standard way to organize numerous python
> files belonging to the same project?
>
> Thank you,
>
> TokiDoki

Re: How to organize Python files in a (relatively) big project
 

I, too have often come up against the inconvenience of creating
libraries that I want to reuse but that do not want to incorporate into
the Python library. I came up with this Python library addition to
automagically add the directory of where a module resides that I want
to import to the system path, so that I can simply add the module by
name without needing to worry about anything else.

The large advantage of this is that I can now import from adjacent
directories rather than subordinate ones. For best results in large
projects with deep tree structure, the paths of the importing script
and the imported module should have at least 3 levels in common. I do
not consider typing 3 levels of pathing as a hint to be a great
inconvenience.

It ain't perfect, it ain't Zen, but for me this has worked from Python
2.2 and up with very few problems for projects up to 1000 files.
Improvements, anyone?

The Eternal Squire

#----------------------relative.py---------------------
# Place in Python##\Lib
#
import sys
from inspect import getfile
from os.path import abspath


backslash = '\\'



def caller ():

'name of the calling script'

frame = sys._getframe(1)
if not frame: return ''
frame = frame.f_back
if not frame: return ''
result = getfile (frame)
if result[-4:].lower () in ['.pyc', '.pyo']: result = filename[:-4]
+ '.py'
return abspath (result)



def append (path):

'append relative path to system module search path'


if not path: return

try:
calling_script = caller ()

top_path = path.split (backslash)[0]
where = calling_script.find (top_path + backslash)

if where < 0: return
result = calling_script [:where] + path

if result not in sys.path: sys.path.append (result)
return result


except:
return ''
#---------------------end of relative.py---------------#

#---------------------"A\B\C\more levels\importer.py"-----------------#
#prototypical code, do not try to execute

import relative
relative.append ("\A\B\C\subpath") # no file extension!
from module import whatsit # module resides under subpath
whatsit ()

Re: How to organize Python files in a (relatively) big project
 

Giovanni Bajo napisa³(a):

> Remember that the directory where you start the toplevel script is always
> included in the sys.path. This means that you can have your structure like
> this:
>
> main.py
> |
> | - - pkg1
> | - - pkg2
> | - - pkg3
>
> Files in any package can import other packages. The usual way is to do "import
> pkgN" and then dereference. Within each package, you will have a __init__.py
> which will define the package API (that is, will define those symbols that you
> can access from outside the package).

How to install this structure eg. on Linux? What layout do you
recommend? It's tempting to use /opt hierarchy for installation target
(as it gives relatively much freedom within application directory), but
many administrators are reluctant to use this hierarchy and prefer more
standarized targets, such as /usr or /usr/local.

--
Jarek Zgoda
http://jpa.berlios.de/

Re: How to organize Python files in a (relatively) big project
 

Micah Elliott napisa³(a):

>>How to install this structure eg. on Linux? What layout do you
>>recommend? It's tempting to use /opt hierarchy for installation
>>target (as it gives relatively much freedom within application
>>directory), but many administrators are reluctant to use this
>>hierarchy and prefer more standarized targets, such as /usr or
>>/usr/local.
>
> Read about (and use) the Python-provided distutils, and let it do the
> work for you. In particular,
> <http://www.python.org/doc/2.4.2/inst/alt-install-windows.html>
> discusses installation location. The file name is a misnomer; it's
> equally applicable to linux.

I think that installing *application* (not misc. library) modules in
site-packages is at least stupid idea, as it makes a special filesystem
inside other, much more standarized filesystem (Windows, FHS, etc.). Why
not to do this usual way: libraries to $prefix/lib/$appname-$version,
binary to $prefix/bin, shared files to $prefix/share/$appname, etc --
and appropriately on Windows.

--
Jarek Zgoda
http://jpa.berlios.de/

Re: How to organize Python files in a (relatively) big project
 

On Oct 19, Jarek Zgoda wrote:
> How to install this structure eg. on Linux? What layout do you
> recommend? It's tempting to use /opt hierarchy for installation
> target (as it gives relatively much freedom within application
> directory), but many administrators are reluctant to use this
> hierarchy and prefer more standarized targets, such as /usr or
> /usr/local.

Read about (and use) the Python-provided distutils, and let it do the
work for you. In particular,
<http://www.python.org/doc/2.4.2/inst/alt-install-windows.html>
discusses installation location. The file name is a misnomer; it's
equally applicable to linux.

It allows users to easily specify the installation prefix. The
default should be the value of "sys.prefix".

Note that your modules/packages will be installed within your python
installation under its site-packages/ directory.

--
_ _ ___
|V|icah |- lliott http://micah.elliott.name mde@micah.elliott.name
" " """

Re: How to organize Python files in a (relatively) big project
 

Jarek Zgoda wrote:

> How to install this structure eg. on Linux? What layout do you
> recommend? It's tempting to use /opt hierarchy for installation target
> (as it gives relatively much freedom within application directory),
> but
> many administrators are reluctant to use this hierarchy and prefer
> more standarized targets, such as /usr or /usr/local.


I am not very experienced with Linux, but if you don't use something like
PyInstaller, you could still install the main tree somewhere like
/usr/local/myapp/ and then generate a simple script for /usr/local/bin which
adds /usr/local/myapp to sys.path[0], and "import main" to boot the
application.

I'm not sure I have answered your question though :)
--
Giovanni Bajo