The Makefile VPATH

I want a Makefile rule to copy files that are not terminated by .md from the source to the website’ output directory.

So, the setup is this:

...
|-- web
|   |-- Makefile
|   |- assets
|   |   |-- navbar.html
|   |   |-- pandoc.css
|   |   \-- pandoc_minified.css
|   |-- index.html
|   \-- posts
|       \-- making_the_website
\-- website
    |-- about.md
    \-- posts
        |-- a.txt
        |-- bugs
        |   \-- 00_The_Makefile_VPATH
        \-- making_the_website
            |-- 00_motivation.md
            |-- 01_the-stack.md
            \-- rem_error_checking.py

The Makefile is located in web/Makefile, and the “sources” are under website/. So, to access the sources in the Makefile, I used the variable VPATH to specify the location of the sources. Then, I created a Pattern Rule to convert from Markdown to HTML.

VPATH = ../website

[....]

%.html : %.md
    mkdir -p $(dir $@)
    $(PANDOC) $(PFLAGS) $^ -o $@

There is no need to write website/%.md, because VPATH ensures that prerequisites are searched in website/. This rule states how to generate an HTML page (in make terms, the target) out of a single Markdown file with the same name (the prerequisite): first, ensure that the directory exists and then convert it from the Markdown file ($^) to an HTML one ($@).

So far, so good. Now I want to specify a rule that copies every file that is not a Markdown file. The mechanism for this is a Match-Anything Pattern Rule – basically, a match, like “%”, without any restrictions. So, I tried:

% : %
    cp -f $^ $@

Well, there are, at least, 2 problems with this rule:

• The output directory may not exist
• The prerequisite is the target, and that creates a circular dependency When that happens, make just says Circular Makefile <- Makefile dependency dropped, and the rule is never executed

The first problem is easily solved, by using the same strategy as in the rule for md->html conversion. But the second is more interesting, and has a unpredicted side-effect.

The second problem cames from an initial misunderstanding of the VPATH mechanism. make will not prepend VPATH to the prerequisites - if no rules match, it will then search for them in there. Also, the match-anything rule appears to have precedence over the VPATH directive. This means that, for example, when we try to find a match for a.html, the first rule is found - but it requires a.md; so, make searches for a rule for .md files - and finds this second rule, that matches. Then, make detects the possibility for recursion, and only after this it uses VPATH, and finds the needed file (and does nothing else). We want this order to be reversed: first, try to search on the source directory, and then, if that fails, try to use a rule.

The naïve solution to the circular Makefile problem is to try to limit the prerequisite matching:

% : $(INPUT_DIR)/%
    mkdir -p $(dir $@)
    cp -f $^ $@

Unfortunately, this doesn’t work for subdirectories, because of the way "%" (called the stem) matches. Imagine our target is posts/a.txt. Instead of matching

posts/a.txt : $(INPUT_DIR)/posts/a.txt

it expands to:

posts/a.txt : posts/$(INPUT_DIR)/a.txt

"%" matches only to the file name, and the directory part is prepended; if you actually read the manual section about pattern matching, it says that "%" matches filenames.

So, by now, we know:

• VPATH is not our friend
• We can’t (directly) rely on stem matching

What about not having a prerequisite? After all, all we need is a destination filename, and we can reconstruct the path afterwards. The only disadvantage this method has is that we lose the part of not doing anything if the target is newer than the prerequisites. So, a implementation could be:

% :
    mkdir -p $(dir $@)
    cp -f $(INPUT_DIR)/$@ $@

and effectively this solves the problem of circular Makefile.

However, a more serious problem still exists: make won’t execute this rule, thanks to VPATH. Continuing with the example of posts/a.txt, make will:

1. Search for rules that match a.txt
2. After finding the implicit (match-anything) rule, it tries to follow the prerequisites: which are none, in this case
3. For some reason, it decides to check if VPATH/posts/a.txt exists; and, because it exists, make says "No need to remake target posts/a.txt [...]"

What I think is happening is that make will only execute the rule if it does not find the target.

After VPATH giving so much trouble, why not just remove it? Without VPATH, the rule above works, but then the rule to convert Markdown stops working. The answer to this dillema is simple: restrict VPATH only to work with .md files. This can be done using the vpath directive. This directive, according to the manual,

allows you to specify a search path for a particular class of file names

So, changing from "VPATH $(INPUT_DIR)" to "vpath %.md $(INPUT_DIR)" just works! And, as a side effect, make will only copy other files if they do not exist in the output directory. To update to a newer version, we have to delete, and then run make.

Update 2017-12-26

The “solution” above was not good enough when dealing with assets: when tinkering with CSS, I had to manually delete and run make (or just make cleaner && make). To correct this, I used a trick explained in the manual section 8.9:

1. Define a multiline variable that is a template to the rule - in this case, copy the asset to the destination folder:

define ASSET_template = 
$(1) :: $(INPUT_DIR)/$(1)
        mkdir -p $(dir $$@) && cp -f $$< $$@
endef

2. Using a foreach loop, call this template to all assets and eval the resulting rule:

$(foreach asset,$(ASSETS),$(eval $(call ASSET_template,$(asset))))

After processing the Makefile, this generated rules are as strong as if I had written them explicitally (unlike implicit rules with pattern matching), and work as expected.

Update 2019-07-16

Finally found a decent solution! Static Pattern Rules allow us to make a special purpose rule, just for a known set of files:

$(ASSETS): % : $(INPUT_DIR)/%
        mkdir -p $(dir $@) && cp -f $< $@

No more mucking around macros! make is really a world in itself, so many neat little tricks…