Metaprogramming is the ability of a program generate other programs, or even modify itself while running. This is the fifth in a series of posts exploring how to use metaprogramming in GNU make makefiles, discussing constructed include files.

Constructed Include Files

In previous posts we’ve discussed the concepts of evaluation and expansion, and ways to build lists of prerequisites from constructed variable names. In this post we’ll consider included makefiles, and in particular ways they can be automatically created.

Included Makefiles

Like many programming languages, makefiles have the ability to include other makefiles using make’s include directive. This is particularly useful for makefiles, since make reads a single makefile by default. If you want to break up and share makefile contents, the include directive allows you to do this:

include one.mk

The arguments to include are first expanded (so they can be variables, or functions such as wildcard) then split into words. Then the parsing of the current makefile is suspended and one by one in the order specified, the included makefiles are parsed. Afterwards the remainder of the current makefile is parsed. Of course, included makefiles can themselves include other makefiles.

Constructing Makefiles

Normally an included makefile is written by you, using your normal text editor. But we can imagine that a makefile can contain a rule that can be used to construct another makefile, perhaps based on variable values, then that file can be included. Using our example from previous posts, we could write a makefile like this:

foo_OBJECTS = baz.o biz.o
bar_OBJECTS = baz.o boz.o
LIBS = libwidget.a

all: foo bar

.PHONY: rules
rules:
        echo 'foo: foo.o $$(foo_OBJECTS) $$(LIBS)' >rules.mk
        echo 'bar: bar.o $$(bar_OBJECTS) $$(LIBS)' >>rules.mk

-include rules.mk

The result of running make rules will be a file rules.mk which contains:

foo: foo.o $(foo_OBJECTS) $(LIBS)
bar: bar.o $(bar_OBJECTS) $(LIBS)

This makefile is then included, defining a set of prerequisites for the foo and bar targets.

It’s important to notice that we use the -include operation here, which does not fail if the makefile doesn’t exist. If we used simple include then we couldn’t run make rules the first time because the include would fail while parsing the makefile, before make had a chance to run the rule and create it.

Automatically Rebuilding Makefiles

This is promising from a metaprogramming perspective: because we are including a complete makefile it can contain any make statements: variables, targets, prerequisites, even complete rules and pattern rules. And because we are building it from the contents of the makefile it can be created and controlled by the makefile itself.

The downside is that you need to invoke make rules to get the included makefile created, or updated when necessary. This isn’t so great.

However it turns out that this can be avoided, due to a feature of GNU make that will automatically attempt to rebuild any included makefiles. How does this work? After make has read in all makefiles, including all included makefiles, it will then attempt to rebuild the original makefile and all of the included makefiles using normal make rules. Basically, make pretends that you had invoked (in the above example) make Makefile rules.mk and tries to rebuild those files. If all the files were up to date and not rebuilt, make then proceeds to build the normal targets. If, on the other hand, any of the makefiles were rebuilt, then make will automatically re-invoke itself from scratch, thus parsing the new versions of those makefiles.

We can rewrite the above makefile to take advantage of this:

foo_OBJECTS = baz.o biz.o
bar_OBJECTS = baz.o boz.o
LIBS = libwidget.a

all: foo bar

rules.mk: Makefile
        echo 'foo: foo.o $$(foo_OBJECTS) $$(LIBS)' >$@
        echo 'bar: bar.o $$(bar_OBJECTS) $$(LIBS)' >>$@

-include rules.mk

Now that make knows how to build the included file rules.mk, we don’t have to build it ourselves. Running make when rules.mk doesn’t exist will cause it to be rebuilt, then the make process will be restarted from scratch to read in the new content.

Note that we’ve declared a prerequisite for rules.mk: the makefile itself. This means that whenever the makefile is edited, rules.mk will be considered out of date and rebuilt. One of the trickiest parts of using the rebuilt included makefiles feature is determining what the proper prerequisites are, such that the included file is rebuilt when necessary. It’s also important to ensure that the included file is not rebuilt when not necessary: as long as the included file is rebuilt the make process will be restarted. If the included file is always rebuilt, then the make process will be restarted forever in an infinite loop!

Creating Included Makefiles

If you examine the makefile above you’ll see that it’s actually longer than it would be if we just wrote out the prerequisites by hand directly in the makefile. To make this more efficient, you need a better way of constructing the included makefiles. One option is to use a loop so you only need to write the body one time regardless of the number of targets. Because we want to write make constructs we need to use make’s foreach loop, not the shell’s loop, something like this:

foo_OBJECTS = baz.o biz.o
bar_OBJECTS = baz.o boz.o
LIBS = libwidget.a

TARGETS = foo bar

all: $(TARGETS)

rules.mk: Makefile
        ( $(foreach T,$(TARGETS),echo '$T: $T.o $$($T_OBJECTS) $$(LIBS)';) ) >$@

-include rules.mk

Even this is awkward, and this is a simple situation. What if you wanted to create a complete rule, with a recipe? To do that with echo statements would be very difficult.

Another solution often used is to write a separate script, either a shell script or even written in a different interpreted language such as Perl or Python. This gives significantly more flexibility, but does require a separate script to be maintained.

Defined Variables

If you need to generate a complex included file containing rules, etc. rather than simply lists of prerequisites, you might consider using defined variables, which, unlike normal variables, can contain newlines. You might try something like this:

foo_TARGET = foo
foo_PREREQUISITES = bar.x biz.y
foo_RULE = ./build -o $$@ $$^

define RULE
$$($T_TARGET): $$($T_PREREQUISITES)
        $$($T_RULE)
endef

TARGETS = foo

rules.mk: Makefile
        ( $(foreach T,$(TARGETS),echo '$(RULE)';) ) >$@

-include rules.mk

Unfortunately this will not work, because newlines in the expansion of RULE will be interpreted by make as newlines in the recipe, and cause a new shell to be invoked rather than being passed to the shell. This will result in a syntax error (missing close quote) in the shell.

If you are using a “new enough” version of GNU make (4.0 or higher) you can use the file function to allow this to work properly:

rules.mk: Makefile
        $(file > $@,) $(foreach T,$(TARGETS),$(file >> $@,$(RULE))

-include rules.mk

The first instance of file clears the output file and the subsequent instances in the foreach loop will append the rules to the file. Since the expansion of the file function is the empty string, the actual recipe is empty and no shell will be invoked, making it more efficient as well.

Avoiding Makefile Recreation

One issue you’ll likely run into is that since makefiles are recreated before GNU make starts processing your build they will always be recreated–even when you don’t want them to be.  The most common situation is invoking make clean: you’re trying to delete all the files and get a clean workspace, so you certainly don’t want make to be rebuilding them!

The way to avoid this is to avoid including the rebuilt makefiles when you’re running the clean command goal.  The $(MAKECMDGOALS) variable contains the command goals for the current invocation of make.  So, suppose that you have a clean target and maybe some other specialized clean targets which all begin with clean-....  Then you can conditionalize the include in this way:

ifeq (,$(filter clean clean-%,$(MAKECMDGOALS)))
  -include rules.mk
endif

This filters all clean and clean-... targets out of $(MAKECMDGOALS); only if that list is empty (there are no clean goals) will the include operation be invoked.

There is one caveat: it does mean that you can’t use an all-in-one command such as make clean all; you must invoke these as two separate commands: make clean && make all. Given the prevalence of parallel build commands it’s probably not a good idea to provide both a clean and build target on the same command line, anyway.

Next …

Constructed include files are definitely the most powerful metaprogramming feature we’ve examined so far: using it you can dynamically create variables, prerequisites, even complete rules. However, the creation of these files can be awkward, especially in versions of GNU make that don’t support the file function. Also, ensuring the included file is rebuilt exactly when necessary is difficult. Finally, updating the included file requires make to completely re-execute which can be a performance hit if your makefiles are large and complex to parse. Next time we’ll examine an alternative to included makefiles which still allows completely arbitrary constructs to be created: the eval function.