How to arrange many files of C source code
 

Up to now I have usually written C modules as stand alone programs and
stored them in separate directories. This has been OK but is
inflexible, making interaction between them and having multiple
generations of a given piece of code awkward. Have you guys any
suggestions on how to organise C source code for use when developing
small to medium size projects?

This may be a generic query and maybe you use a certain tried and
tested scheme for every C program you write but if it helps to explain
the context it is as follows.

I would like to adopt a scheme that I can use for all C code from now
on. The model I have in mind is to have some apps (by which I mean
things such as user-facing programs) and some utilities. The utils
should be usable by multiple apps. At some point it may even be useful
for the apps to call each other. That is not a current requirement,
only a query in my mind as to whether there is some reason that apps
and utilities need to be distinguished.

Different generations of a given app or utility may need to be
available at the same time. In other words, I may need to keep a
certain piece of code around even though there is a newer version of
the same code.

At some point - might as well be now - I think I will need to use some
source code management application: GIT, Bazaar, Subversion or
similar. I might use it initially for local working but later to allow
collaboration. I guess that the choice of SCM affects how source code
is laid out. For example, it would be simpler to manage all files in a
given folder rather than selected ones.

I would like to be able to treat each app or util as an entity and
move or copy it around without having to also copy anything other than
the utils it depends on.

Ideally, compilation would treat the source code directories as read-
only and generate all of its output in a place relevant to the target
architecture; I would like to be able to build the same code for
different operating systems.

The above leads to questions over how best to include headers from
other directories and how to specify what to link together. I would
like to avoid use of environment variables as far as possible. Builds
should take the latest versions of a utility by default and only
choose older ones if told explicitly to do so. Makefiles are great if
building under Unix and I would prefer to use them to specify where
referenced files sit. Not sure what I would use if building under
Windows which I might have to do in some cases.

Sorry that's a long list. Hopefully it is a fairly standard set of
requirements!

James

Re: How to arrange many files of C source code
 

On 3/2/2013 06:17, James Harris wrote:
>
> Sorry that's a long list. Hopefully it is a fairly standard set of
> requirements!
>

If you wish to target, let's say, Linux, Mac OS X, Windows, well
Makefiles can work for all three. You can cross-compile for Windows
using MinGW on Linux, though the resulting programs cannot be debugged
with WinDbg (but can be debugged with a Windows-aware gdb). If you
compile for Windows using, let's say, the Windows driver kit (which is
capable of compiling user-mode programs, too), then you'll probably use
a different build system (the WDK's) than for the other targets.

Why do you wish to avoid environment variables? All three of these
operating systems have them, all three have a "path" where programs are
searched-for... Perhaps you wish for a 'make install' process to
install into a common path for each OS?

If you already have things organized a certain way and are running into
complications with using 'make', there might be a better forum than
comp.lang.c for finding assistance. If you're simply wondering about a
suggested organizational structure, it's pretty subjective... Here's
something I would find pretty and useful:

./
./.git/
./ReadMe.txt
./Makefile
./License.txt
./other_project_stuff
./inc/
./inc/portable.h
./inc/common.h
./inc/app1/
./inc/app1/foo.h
./inc/util1/
./inc/util1/bar.h
./src/
./src/app1/
./src/app1/foo.c
./src/app1/main.c
./src/utils1/
./src/utils1/bar.c
./src/utils1/main.c
./bin/
./bin/app1
./bin/util1

Straying from the forum's topic and with regards to MinGW, here's a
Makefile for building Windows programs with MinGW on Linux:


http://git.zytor.com/?p=users/sha0/s...s/mdiskchk_win

And here is a handy web-page for building programs with the WDK, in case
you choose that route:

http://www.wd-3.com/archive/WinBuild.htm

--
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily," -- Richard Harter

Re: How to arrange many files of C source code
 

On Saturday, March 2, 2013 11:17:44 AM UTC, James Harris wrote:
>
> Have you guys any
> suggestions on how to organise C source code for use when developing
> small to medium size projects?
>
Separate files out into four groups: this program only: this platform only,this program only: any platform, this platform only: many programs, and any platform, many program.

Obviously you want as much code as possible in the files in the last group,and as little as possible in the first group.

Now give the files clean interfaces. Use static heavily to make as many files as possible idempotent. (This doesn't apply on a small embedded system).Don't make a file dependent on another just because it's calling a few trivial roll your own string functions.

Now give each file a unit test, labelled mysourcemain / main.

What you should end up with is programs with trivial driver files consisting of just a short main() that holds everything together.

Re: How to arrange many files of C source code
 

On Mar 2, 1:49*pm, Shao Miller <sha0.mil...@gmail.com> wrote:

....

> Why do you wish to avoid environment variables? *All three of these
> operating systems have them, all three have a "path" where programs are
> searched-for...

As a rule, use of environment variables introduces a dependency on
something which is outside the makefile. If they are set incorrectly
the build will fail or, worse, will silently build incorrectly.

I wouldn't rule them out completely but I think they are overused.

> Perhaps you wish for a 'make install' process to
> install into a common path for each OS?
>
> If you already have things organized a certain way and are running into
> complications with using 'make', there might be a better forum than
> comp.lang.c for finding assistance. *If you're simply wondering about a
> suggested organizational structure, it's pretty subjective... *Here's
> something I would find pretty and useful:
>
> * *./
> * *./.git/
> * *./ReadMe.txt
> * *./Makefile
> * *./License.txt
> * *./other_project_stuff
> * *./inc/
> * *./inc/portable.h
> * *./inc/common.h
> * *./inc/app1/
> * *./inc/app1/foo.h
> * *./inc/util1/
> * *./inc/util1/bar.h
> * *./src/
> * *./src/app1/
> * *./src/app1/foo.c
> * *./src/app1/main.c
> * *./src/utils1/
> * *./src/utils1/bar.c
> * *./src/utils1/main.c
> * *./bin/
> * *./bin/app1
> * *./bin/util1

Thanks. That's the kind of suggestion I wanted but it leads to some
questions:
* What would be the process to build app1 - change to app1's directory
and run 'make'?
* How could app1's makefile refer to utils1 components?
* How would app1's makefile ensure utils1's components were up to
date?
* What if there were multiple generations of utils1?
* Why put the headers in separate inc folders?
* Does the structure mean there would be one GIT directory for all
those subdirectories? I was thinking that app1 (and utils1 for that
matter) would be a separate project.

James

Re: How to arrange many files of C source code
 

On 3/4/2013 08:17, James Harris wrote:
> On Mar 2, 1:49 pm, Shao Miller <sha0.mil...@gmail.com> wrote:
>
> ...
>
>> Why do you wish to avoid environment variables? All three of these
>> operating systems have them, all three have a "path" where programs are
>> searched-for...
>
> As a rule, use of environment variables introduces a dependency on
> something which is outside the makefile. If they are set incorrectly
> the build will fail or, worse, will silently build incorrectly.
>
> I wouldn't rule them out completely but I think they are overused.
>

I thought you were referring to using environment variables within your
programs, rather than for building. I think you'd mentioned that you
wanted the programs to be able to find one another; environment
variables seem like a fair way to accomplish that.

>> Perhaps you wish for a 'make install' process to
>> install into a common path for each OS?
>>
>> If you already have things organized a certain way and are running into
>> complications with using 'make', there might be a better forum than
>> comp.lang.c for finding assistance. If you're simply wondering about a
>> suggested organizational structure, it's pretty subjective... Here's
>> something I would find pretty and useful:
>>
>> ./
>> ./.git/
>> ./ReadMe.txt
>> ./Makefile
>> ./License.txt
>> ./other_project_stuff
>> ./inc/
>> ./inc/portable.h
>> ./inc/common.h
>> ./inc/app1/
>> ./inc/app1/foo.h
>> ./inc/util1/
>> ./inc/util1/bar.h
>> ./src/
>> ./src/app1/
>> ./src/app1/foo.c
>> ./src/app1/main.c
>> ./src/utils1/
>> ./src/utils1/bar.c
>> ./src/utils1/main.c
>> ./bin/
>> ./bin/app1
>> ./bin/util1
>
> Thanks. That's the kind of suggestion I wanted but it leads to some
> questions:
> * What would be the process to build app1 - change to app1's directory
> and run 'make'?

A top-level Makefile can invoke a Makefile in each subdir of src/

> * How could app1's makefile refer to utils1 components?
> * How would app1's makefile ensure utils1's components were up to
> date?
> * What if there were multiple generations of utils1?

These questions might meet with good answers in another forum. Perhaps
someone here might have a suggestion for a 'make'-related forum.

> * Why put the headers in separate inc folders?

To avoid filename collision in inc/

> * Does the structure mean there would be one GIT directory for all
> those subdirectories? I was thinking that app1 (and utils1 for that
> matter) would be a separate project.
>

You can track all projects together as a single Git repository, or Git
also allows you to have a top-level repository with sub-repositories,
such that each src/subproj/ could have its own src/subproj/.git/

--
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily," -- Richard Harter

Re: How to arrange many files of C source code
 

On Mar 4, 4:55*pm, Shao Miller <sha0.mil...@gmail.com> wrote:
> On 3/4/2013 08:17, James Harris wrote:
>
> > On Mar 2, 1:49 pm, Shao Miller <sha0.mil...@gmail.com> wrote:
>
> > ...
>
> >> Why do you wish to avoid environment variables? *All three of these
> >> operating systems have them, all three have a "path" where programs are
> >> searched-for...
>
> > As a rule, use of environment variables introduces a dependency on
> > something which is outside the makefile. If they are set incorrectly
> > the build will fail or, worse, will silently build incorrectly.
>
> > I wouldn't rule them out completely but I think they are overused.
>
> I thought you were referring to using environment variables within your
> programs, rather than for building. *I think you'd mentioned that you
> wanted the programs to be able to find one another; environment
> variables seem like a fair way to accomplish that.

Sorry - I tried to keep the description short. Let me go the other way
and illustrate by using something specific in the form of the
directory tree you suggested.

Say you wanted to include a header from another directory would you
write something along the lines or

#include "../../src/utils1/header.h"

or would it be better to write

#include "header.h"

and to add a compile switch along the lines of

--header-source ../../src/utils1

In either case how is it best to specify the referred-to directories?
Is the idea of relative subdirectories good or should they be absolute
or should they be relative to a site-chosen base directory? Is there a
better way to specify where the desired header should be looked for
bearing in mind that there may be multiple generations of a given
utility and by default the most recent one is wanted? So if the
directory tree includes

./src/utilA.1
./src/utilA.2

these would be two versions of utilA. I know that if using Unix I
could link ./src/utilA to whatever was the most recent version but
that wouldn't work on Windows and even on Unix I wondered if having
two names for the one directory would confuse a source code management
package.

I know that's quite a long description. I may be better to post
individual questions separately but even an overview of how others
organise their source code is a start so thanks for what you explained
earlier.

Of course, the above is just for compilation includes. There are the
same issues with telling the linker where to look for object modules.

> >> Perhaps you wish for a 'make install' process to
> >> install into a common path for each OS?
>
> >> If you already have things organized a certain way and are running into
> >> complications with using 'make', there might be a better forum than
> >> comp.lang.c for finding assistance. *If you're simply wondering about a
> >> suggested organizational structure, it's pretty subjective... *Here's
> >> something I would find pretty and useful:
>
> >> * * ./
> >> * * ./.git/
> >> * * ./ReadMe.txt
> >> * * ./Makefile
> >> * * ./License.txt
> >> * * ./other_project_stuff
> >> * * ./inc/
> >> * * ./inc/portable.h
> >> * * ./inc/common.h
> >> * * ./inc/app1/
> >> * * ./inc/app1/foo.h
> >> * * ./inc/util1/
> >> * * ./inc/util1/bar.h
> >> * * ./src/
> >> * * ./src/app1/
> >> * * ./src/app1/foo.c
> >> * * ./src/app1/main.c
> >> * * ./src/utils1/
> >> * * ./src/utils1/bar.c
> >> * * ./src/utils1/main.c
> >> * * ./bin/
> >> * * ./bin/app1
> >> * * ./bin/util1
>
> > Thanks. That's the kind of suggestion I wanted but it leads to some
> > questions:
> > * What would be the process to build app1 - change to app1's directory
> > and run 'make'?
>
> A top-level Makefile can invoke a Makefile in each subdir of src/

If using make I'm not sure I would want to build all packages at once.
For example, if I was working on application A I would only want to
make A and any utilities A depends on (which utilities might have
changed).

>
> > * How could app1's makefile refer to utils1 components?
> > * How would app1's makefile ensure utils1's components were up to
> > date?
> > * What if there were multiple generations of utils1?
>
> These questions might meet with good answers in another forum. *Perhaps
> someone here might have a suggestion for a 'make'-related forum.

Thanks. Yes, I'll be able to check up on make specifics that I haven't
used before. I was more interested in how C programmers organise their
source code directories etc so that one module can refer to modules in
other directories.

> > * Why put the headers in separate inc folders?
>
> To avoid filename collision in inc/

Wouldn't the headers for, say, util2 sit in the ./src/util2 directory?

James

Re: How to arrange many files of C source code
 

On Mar 4, 4:55*pm, Shao Miller <sha0.mil...@gmail.com> wrote:

> A top-level Makefile can invoke a Makefile in each subdir of src/

Isn't that what is warned about in http://miller.emu.id.au/pmiller/books/rmch/
"Recursive Make Considered Harmful" ?

Re: How to arrange many files of C source code
 

On 3/4/2013 16:54, Paul N wrote:
> On Mar 4, 4:55 pm, Shao Miller <sha0.mil...@gmail.com> wrote:
>
>> A top-level Makefile can invoke a Makefile in each subdir of src/
>
> Isn't that what is warned about in http://miller.emu.id.au/pmiller/books/rmch/
> "Recursive Make Considered Harmful" ?
>

It seems to be. That web-page says that its content is copyright 1997,
so I'm not sure if it's still relevant. As far as I know, one can do
dependency magic with a 'make' process, these days. I don't recall
suffering from recursive 'make', recently... But maybe other people do.

--
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily," -- Richard Harter

Re: How to arrange many files of C source code
 

On 3/4/2013 14:35, James Harris wrote:
> On Mar 4, 4:55 pm, Shao Miller <sha0.mil...@gmail.com> wrote:
>> On 3/4/2013 08:17, James Harris wrote:
>>
>>> On Mar 2, 1:49 pm, Shao Miller <sha0.mil...@gmail.com> wrote:
>>
>>> ...
>>
>>>> Why do you wish to avoid environment variables? All three of these
>>>> operating systems have them, all three have a "path" where programs are
>>>> searched-for...
>>
>>> As a rule, use of environment variables introduces a dependency on
>>> something which is outside the makefile. If they are set incorrectly
>>> the build will fail or, worse, will silently build incorrectly.
>>
>>> I wouldn't rule them out completely but I think they are overused.
>>
>> I thought you were referring to using environment variables within your
>> programs, rather than for building. I think you'd mentioned that you
>> wanted the programs to be able to find one another; environment
>> variables seem like a fair way to accomplish that.
>
> Sorry - I tried to keep the description short. Let me go the other way
> and illustrate by using something specific in the form of the
> directory tree you suggested.
>
> Say you wanted to include a header from another directory would you
> write something along the lines or
>
> #include "../../src/utils1/header.h"
>
> or would it be better to write
>
> #include "header.h"
>
> and to add a compile switch along the lines of
>
> --header-source ../../src/utils1
>

The latter. For each project, I'd tell the implementation to look for
#include "file.h" in inc/ and inc/thisproj/ If thisproj needed
knowledge of otherproj, thisproj's .c files would do #include
"otherproj/other.h" (which is under inc/).

> In either case how is it best to specify the referred-to directories?
> Is the idea of relative subdirectories good or should they be absolute
> or should they be relative to a site-chosen base directory? Is there a
> better way to specify where the desired header should be looked for
> bearing in mind that there may be multiple generations of a given
> utility and by default the most recent one is wanted? So if the
> directory tree includes
>
> ./src/utilA.1
> ./src/utilA.2
>
> these would be two versions of utilA. I know that if using Unix I
> could link ./src/utilA to whatever was the most recent version but
> that wouldn't work on Windows and even on Unix I wondered if having
> two names for the one directory would confuse a source code management
> package.
>

If those are directories, you could build for both. I'm not sure if
you're talking about directories or binaries, here.

> I know that's quite a long description. I may be better to post
> individual questions separately but even an overview of how others
> organise their source code is a start so thanks for what you explained
> earlier.
>

I hope it helps in some small way. :) It seems a bit "fringe" for
comp.lang.c.

> Of course, the above is just for compilation includes. There are the
> same issues with telling the linker where to look for object modules.
>

Well what you might do is have, somewhere on your build system:

./linux_x86/

and invoke 'make' from there, pointing it to the top-level Makefile in
your repository, then it can build:

./linux_x86/obj/app1/main.o
./linux_x86/obj/util1/main.o
./linux_x86/bin/app1
./linux_x86/bin/util1

That would work, for me.

Or, you could peek at what Apple's Bonjour does, which is different.
(To name a random example.)

>>>> Perhaps you wish for a 'make install' process to
>>>> install into a common path for each OS?
>>
>>>> If you already have things organized a certain way and are running into
>>>> complications with using 'make', there might be a better forum than
>>>> comp.lang.c for finding assistance. If you're simply wondering about a
>>>> suggested organizational structure, it's pretty subjective... Here's
>>>> something I would find pretty and useful:
>>
>>>> ./
>>>> ./.git/
>>>> ./ReadMe.txt
>>>> ./Makefile
>>>> ./License.txt
>>>> ./other_project_stuff
>>>> ./inc/
>>>> ./inc/portable.h
>>>> ./inc/common.h
>>>> ./inc/app1/
>>>> ./inc/app1/foo.h
>>>> ./inc/util1/
>>>> ./inc/util1/bar.h
>>>> ./src/
>>>> ./src/app1/
>>>> ./src/app1/foo.c
>>>> ./src/app1/main.c
>>>> ./src/utils1/
>>>> ./src/utils1/bar.c
>>>> ./src/utils1/main.c
>>>> ./bin/
>>>> ./bin/app1
>>>> ./bin/util1
>>
>>> Thanks. That's the kind of suggestion I wanted but it leads to some
>>> questions:
>>> * What would be the process to build app1 - change to app1's directory
>>> and run 'make'?
>>
>> A top-level Makefile can invoke a Makefile in each subdir of src/
>
> If using make I'm not sure I would want to build all packages at once.
> For example, if I was working on application A I would only want to
> make A and any utilities A depends on (which utilities might have
> changed).
>

That is why it might be handy to step inside a src/subdir/ and invoke
'make' from there, using src/subdir/Makefile I'd say the top-level
Makefile should be pretty "dumb".

>>
>>> * How could app1's makefile refer to utils1 components?
>>> * How would app1's makefile ensure utils1's components were up to
>>> date?
>>> * What if there were multiple generations of utils1?
>>
>> These questions might meet with good answers in another forum. Perhaps
>> someone here might have a suggestion for a 'make'-related forum.
>
> Thanks. Yes, I'll be able to check up on make specifics that I haven't
> used before. I was more interested in how C programmers organise their
> source code directories etc so that one module can refer to modules in
> other directories.
>

I think it's fairly subjective. There haven't been too many respondents
to the thread, so it might even be too far from C to interest others.

>>> * Why put the headers in separate inc folders?
>>
>> To avoid filename collision in inc/
>
> Wouldn't the headers for, say, util2 sit in the ./src/util2 directory?
>

I'd rather have them in inc/util2/ and every project looks in inc/ and
inc/thisproj/ for headers. However, I have seen projects that match
what you describe, too.

--
- Shao Miller
--
"Thank you for the kind words; those are the kind of words I like to hear.

Cheerily," -- Richard Harter

Re: How to arrange many files of C source code
 

On Mar 4, 11:55*pm, Shao Miller <sha0.mil...@gmail.com> wrote:

....

> > ... I was more interested in how C programmers organise their
> > source code directories etc so that one module can refer to modules in
> > other directories.
>
> I think it's fairly subjective. *There haven't been too many respondents
> to the thread, so it might even be too far from C to interest others.

It did seem odd at first. I thought there would have been a number of
suggestions over how best to organise C source code for multiple
projects. After all, we tend to build up loads of code after
programming for a few years, some of it utility code that is used in
different projects. The code has to be stored somewhere! Possibly
there were few other replies because people agreed with or had little
to add to the first suggestion made. Or perhaps there are no common
approaches that C programmers use. Or maybe folks don't address the
specific things I was asking about. At any rate, I appreciate the help
with this from you and Malcolm. It has given me some ideas for ways to
get started.

James