VPATH+: GNU make Enhanced VPATH Patch

VPATH+ is a patch to GNU make (3.63 and above) which modifies the functionality of VPATH/vpath. The shipped functionality is consistent with AT&T make's VPATH, so Roland is reluctant to change it for Rev 3.xx although he has promised to address the problem in Rev 4.

Until that time, I will try to maintain this patch for each new 3.xx revision of GNU make so that I (we) can have a VPATH/vpath which "does what you want", and is thus more useful (just because it's AT&T compatible doesn't mean it makes sense! :-)

You can obtain the patches for all the above versions of GNU make, along with a copy of this README, from ftp://ftp.wellfleet.com/netman/psmith/gmake/. You can also get them from this patch list.

A brief description of the problem with the shipped VPATH/vpath functionality follows, then a general discussion of my proposed solution, and finally details on how I implemented the solution in the enclosed patch.

NOTE:

If you use VPATH+, please send me email and I'll put you on my ANNOUNCE mailing list. This list is not accessible by the general public; only I can email to it (it's actually stored in BBDB :), so don't worry about excess traffic.

The only items I email to the list are bug announcements (hasn't been one since Jan, 1992) and new versions of VPATH+.

The Problem

Once GNU make finds a target file through VPATH/vpath it changes the target path to the VPATH/vpath name immediately, causing all dependencies of the target to be searched for in the VPATH directory to the exclusion of local files which might be newer.

For example, suppose you have a source tree with stable code built in it. You want to create a local directory, modify one .c file, and build a local copy of the binary to test without modifying anything in the tree. Although this seems like a perfect job for VPATH, this is exactly where the problem arises. To take a very simple example:

Suppose we have a stable tree:

    /tree/Makefile
      '' /bar.c
      '' /bar.o
      '' /foo
      '' /foo.c
      '' /foo.o

Where Makefile says:

    VPATH = /tree

    foo: foo.o bar.o

This will build just fine in /tree. Now suppose we want to test a change to bar.c. We go into our working directory and modify a copy of bar.c, so we have:

    /work/Makefile
      '' /bar.c

Now it's fairly obvious what we want to happen when we run make: we want a local copy of bar.o and foo built, but we don't want to have to re-compile foo.c needlessly; it hasn't changed. So after the make what we want is:

    /work/Makefile
      '' /bar.c
      '' /bar.o
      '' /foo

But this isn't what happens. What happens is this:

1. make examines target foo. It doesn't find it, so it searches VPATH and locates "/tree/foo". make then changes the name of the target to "/tree/foo".

   It's the nature of VPATH that it is only used on relative target pathnames; i.e., ones that don't begin with "/". Thus, we are now hosed :) ... watch:

2. make then examines the dependencies of foo. It sees foo.o and bar.o. Because the target name is now "/tree/foo", the dependencies are looked for in /tree as well.

3. /tree/foo.o and /tree/bar.o exist, and are newer than /tree/foo.c and /tree/bar.c, so there's no need to remake anything, and bingo! make says your product is up-to-date and quits.

Step 1 is an example of the problem I'm trying to solve. Because make expands the target to be the VPATH name *before* examining the dependencies, it looks for all further dependencies, and their dependencies, etc. in the same directory the target was found in, so it doesn't notice that there is a local copy of bar.c which is newer than /tree/bar.o and it doesn't even try to rebuild foo.

My Solution

My idea for the proper functioning of VPATH is this:

1. If the target does not exist locally then it is looked for on VPATH.

2. If the target is found on VPATH then it is marked as found and its mod time is obtained. The VPATH pathname of the file is stored, but the target name is *not* modified.

3. All dependencies of the target are examined using this method recursively.

4. make decides if it needs to rebuild the target:
   1. If the target does not have to be rebuilt, then the name of the target is modified to be the VPATH pathname. This modification is of course reflected in all dependency lists containing this target as well, so that all targets dependent on this file will use the VPATH pathname if they should need to be rebuilt.

   2. If the target must be rebuilt, then the VPATH pathname is thrown away and the target is rebuilt using the local name.

Simply put, if a target does not need to be rebuilt the VPATH version of the target will be used. If it does need to be rebuilt then it will be built locally (in the current working directory), using whatever dependencies were found (local or VPATH).

This seems to be exactly what I want. RMS, Roland and I have batted this around and RMS suggested one additional step which I have not implemented here because I don't need it. Briefly, RMS suggested a way in which to mark, on a per-directory basis, files which should be created *in the tree* rather than locally. I.e., all files which need to be rebuilt and which are found in a marked directory will be built in the marked directory, rather than in the current working directory. If you feel you have a need for this, let me know and we'll talk: so far I don't need it but I have a few ideas on how to implement such a thing if someone else needs it.

If you find problems with the above description please let me know.

Implementation Details

For those who care, this is how I modified GNU make to do the above:

• file.h: I modified struct file to contain another name field, called "hname". I also added another flag field called "ignore_vpath".

• file.c:lookup_file(): modified to hash on "hname" instead of the "name" field. This allows us to modify the name of the file in the hash table while still keeping the original name around.

• file.c:rename_file(): only modifies the "hname" fields, and leaves the "name" fields alone. I renamed the function to be rehash_file() to enforce that concept.

• remake.c:update_file_1(): if the target doesn't have to be rebuilt, changes the "name" field of the file to be the "hname" field (which is the filename found via VPATH, if one exists, and just "name" otherwise).

  If the target does have to be rebuilt, the "name" field of the target is not changed from what it was originally in the makefile, so the VPATH path is not used.

  Also in this case the "ignore_vpath" flag is set so that it's not searched for on VPATH again. This is necessary so that if the file doesn't exist after the rules to build it are run, it won't be looked for on VPATH again. Otherwise the VPATH name will be used as the built target: in addition to being incorrect behavior IMHO, this caused an infinite loop 'cause we renamed a file to itself (just as a precaution I added a check for this into file.c:file_hash_enter(), but it shouldn't happen anymore).

  I added a bunch of debugging statements here so that make -d tells you what is going on.

• remake.c:f_mtime(): calls rehash_file() instead of rename_file(). Also, because I didn't change the name of the file I had to remove the recursive call to file_mtime() or make would recurse infinitely. In it's place I put a call to name_mtime().

  After discussing that recursion with Roland it appears that removing it will cause an extra, potentially unneeded stat(2) call to be executed if a very obtuse, convoluted set of make rules is used. But removing the recursion was the easiest way for me to do it and it still works the same, and saving one stat() on a very obscure case wasn't important enough to me to get more complex :)

• implicit.c:pattern_search(): if we find an implicit file in VPATH, save the original name not the VPATH name.

• version.c: so that folks could tell which version they were using.

That's it! It has worked just fine for me in my development environment and in a number of tests I whipped up for different things. If you have any problems whatsoever with behavior which you feel is unexpected or nonintuitive W.R.T. VPATH+, please let me know...

Thanks, and happy making!