Metaprogramming is the ability of a program generate other programs, or even modify itself while running. This is the fifth in a series of posts exploring how to use metaprogramming in GNU make makefiles, discussing constructed include files.
Constructed Include Files
In previous posts we’ve discussed the concepts of evaluation and expansion, and ways to build lists of prerequisites from constructed variable names. In this post we’ll consider included makefiles, and in particular ways they can be automatically created.
Like many programming languages, makefiles have the ability to include other makefiles using make’s
include directive. This is particularly useful for makefiles, since make reads a single makefile by default. If you want to break up and share makefile contents, the
include directive allows you to do this:
The arguments to include are first expanded (so they can be variables, or functions such as wildcard) then split into words. Then the parsing of the current makefile is suspended and one by one in the order specified, the included makefiles are parsed. Afterwards the remainder of the current makefile is parsed. Of course, included makefiles can themselves include other makefiles.
Normally an included makefile is written by you, using your normal text editor. But we can imagine that a makefile can contain a rule that can be used to construct another makefile, perhaps based on variable values, then that file can be included. Using our example from previous posts, we could write a makefile like this:
foo_OBJECTS = baz.o biz.o bar_OBJECTS = baz.o boz.o LIBS = libwidget.a all: foo bar .PHONY: rules rules: echo 'foo: foo.o $$(foo_OBJECTS) $$(LIBS)' >rules.mk echo 'bar: bar.o $$(bar_OBJECTS) $$(LIBS)' >>rules.mk -include rules.mk
The result of running
make rules will be a file
rules.mk which contains:
foo: foo.o $(foo_OBJECTS) $(LIBS) bar: bar.o $(bar_OBJECTS) $(LIBS)
This makefile is then included, defining a set of prerequisites for the
It’s important to notice that we use the
-include operation here, which does not fail if the makefile doesn’t exist. If we used simple
include then we couldn’t run
make rules the first time because the
include would fail while parsing the makefile, before make had a chance to run the rule and create it.
Automatically Rebuilding Makefiles
This is promising from a metaprogramming perspective: because we are including a complete makefile it can contain any make statements: variables, targets, prerequisites, even complete rules and pattern rules. And because we are building it from the contents of the makefile it can be created and controlled by the makefile itself.
The downside is that you need to invoke
make rules to get the included makefile created, or updated when necessary. This isn’t so great.
However it turns out that this can be avoided, due to a feature of GNU make that will automatically attempt to rebuild any included makefiles. How does this work? After make has read in all makefiles, including all included makefiles, it will then attempt to rebuild the original makefile and all of the included makefiles using normal make rules. Basically, make pretends that you had invoked (in the above example)
make Makefile rules.mk and tries to rebuild those files. If all the files were up to date and not rebuilt, make then proceeds to build the normal targets. If, on the other hand, any of the makefiles were rebuilt, then make will automatically re-invoke itself from scratch, thus parsing the new versions of those makefiles.
We can rewrite the above makefile to take advantage of this:
foo_OBJECTS = baz.o biz.o bar_OBJECTS = baz.o boz.o LIBS = libwidget.a all: foo bar rules.mk: Makefile echo 'foo: foo.o $$(foo_OBJECTS) $$(LIBS)' >$@ echo 'bar: bar.o $$(bar_OBJECTS) $$(LIBS)' >>$@ -include rules.mk
Now that make knows how to build the included file
rules.mk, we don’t have to build it ourselves. Running make when
rules.mk doesn’t exist will cause it to be rebuilt, then the make process will be restarted from scratch to read in the new content.
Note that we’ve declared a prerequisite for
rules.mk: the makefile itself. This means that whenever the makefile is edited,
rules.mk will be considered out of date and rebuilt. One of the trickiest parts of using the rebuilt included makefiles feature is determining what the proper prerequisites are, such that the included file is rebuilt when necessary. It’s also important to ensure that the included file is not rebuilt when not necessary: as long as the included file is rebuilt the make process will be restarted. If the included file is always rebuilt, then the make process will be restarted forever in an infinite loop!
Creating Included Makefiles
If you examine the makefile above you’ll see that it’s actually longer than it would be if we just wrote out the prerequisites by hand directly in the makefile. To make this more efficient, you need a better way of constructing the included makefiles. One option is to use a loop so you only need to write the body one time regardless of the number of targets. Because we want to write make constructs we need to use make’s
foreach loop, not the shell’s loop, something like this:
foo_OBJECTS = baz.o biz.o bar_OBJECTS = baz.o boz.o LIBS = libwidget.a TARGETS = foo bar all: $(TARGETS) rules.mk: Makefile ( $(foreach T,$(TARGETS),echo '$T: $T.o $$($T_OBJECTS) $$(LIBS)';) ) >$@ -include rules.mk
Even this is awkward, and this is a simple situation. What if you wanted to create a complete rule, with a recipe? To do that with echo statements would be very difficult.
Another solution often used is to write a separate script, either a shell script or even written in a different interpreted language such as Perl or Python. This gives significantly more flexibility, but does require a separate script to be maintained.
If you need to generate a complex included file containing rules, etc. rather than simply lists of prerequisites, you might consider using defined variables, which, unlike normal variables, can contain newlines. You might try something like this:
foo_TARGET = foo foo_PREREQUISITES = bar.x biz.y foo_RULE = ./build -o $$@ $$^ define RULE $$($T_TARGET): $$($T_PREREQUISITES) $$($T_RULE) endef TARGETS = foo rules.mk: Makefile ( $(foreach T,$(TARGETS),echo '$(RULE)';) ) >$@ -include rules.mk
Unfortunately this will not work, because newlines in the expansion of
RULE will be interpreted by make as newlines in the recipe, and cause a new shell to be invoked rather than being passed to the shell. This will result in a syntax error (missing close quote) in the shell.
If you are using a “new enough” version of GNU make (4.0 or higher) you can use the
file function to allow this to work properly:
rules.mk: Makefile $(file > $@,) $(foreach T,$(TARGETS),$(file >> $@,$(RULE)) -include rules.mk
The first instance of
file clears the output file and the subsequent instances in the
foreach loop will append the rules to the file. Since the expansion of the
file function is the empty string, the actual recipe is empty and no shell will be invoked, making it more efficient as well.
Avoiding Makefile Recreation
One issue you’ll likely run into is that since makefiles are recreated before GNU make starts processing your build they will always be recreated–even when you don’t want them to be. The most common situation is invoking
make clean: you’re trying to delete all the files and get a clean workspace, so you certainly don’t want make to be rebuilding them!
The way to avoid this is to avoid including the rebuilt makefiles when you’re running the
clean command goal. The
$(MAKECMDGOALS) variable contains the command goals for the current invocation of make. So, suppose that you have a
clean target and maybe some other specialized clean targets which all begin with
clean-.... Then you can conditionalize the include in this way:
ifeq (,$(filter clean clean-%,$(MAKECMDGOALS))) -include rules.mk endif
This filters all
clean-... targets out of
$(MAKECMDGOALS); only if that list is empty (there are no
clean goals) will the include operation be invoked.
There is one caveat: it does mean that you can’t use an all-in-one command such as
make clean all; you must invoke these as two separate commands:
make clean && make all. Given the prevalence of parallel build commands it’s probably not a good idea to provide both a clean and build target on the same command line, anyway.
Constructed include files are definitely the most powerful metaprogramming feature we’ve examined so far: using it you can dynamically create variables, prerequisites, even complete rules. However, the creation of these files can be awkward, especially in versions of GNU make that don’t support the
file function. Also, ensuring the included file is rebuilt exactly when necessary is difficult. Finally, updating the included file requires make to completely re-execute which can be a performance hit if your makefiles are large and complex to parse. Next time we’ll examine an alternative to included makefiles which still allows completely arbitrary constructs to be created: the