Go Make

Makefile parsing and utilities in Go

Usage

Reading

The make.Parser is the primary way to read Makefiles.

f := os.Open("Makefile")
p := make.NewParser(f, nil)

m, err := p.ParseFile()

fmt.Println(m.Rules)

The more primitive make.Scanner and make.ScanTokens used by make.Parser can be used individually.

Using make.ScanTokens with a bufio.Scanner

f := os.Open("Makefile")
s := bufio.NewScanner(f)
s.Split(make.ScanTokens)

for s.Scan() {
  s.Bytes() // The current token byte slice i.e. []byte(":=")
  s.Text() // The current token as a string i.e. ":="
}

Using make.Scanner

f := os.Open("Makefile")
s := make.NewScanner(f, nil)

for pos, tok, lit := s.Scan(); tok != token.EOF; {
  fmt.Println(pos) // The position of tok
  fmt.Println(tok) // The current token.Token i.e. token.SIMPLE_ASSIGN
  fmt.Println(lit) // Literal tokens as a string i.e. "identifier"
}

if err := s.Err(); err != nil {
  fmt.Println(err)
}

Writing

Use make.Fprint to write ast nodes.

Note The AST in this project is a made-up, package-specific representation for Makefiles. It is not an official GNU Make or POSIX AST.

var file *ast.File

n, err := make.Fprint(os.Stdout, file)

The make.Writer can be used to incrementally write make syntax to an io.Writer.

buf := &bytes.Buffer{}
w := make.NewWriter(buf)

n, err := w.WriteRule(&ast.Rule{})

Builder

The builder package contains utilities for building AST nodes.

🚧 This API is not stable yet 🚧

f := builder.NewFile(1,
  file.WithRule(expr.Text("target1"),
    rule.WithVarRefTarget("FOO")
  ),
)

make.Fprint(os.Stdout, f)
// target1 ${FOO}:\n

Features

Syntax Support

Makefile syntax that is guaranteed to round-trip (parse and print without modification) is listed in ./testdata/roundtrip. Additional syntax is supported and may round-trip successfully, but no guarentees are provided until it is listed under ./testdata/roundtrip.

The table below covers the syntax in the GNU Make Quick Reference. Every status is verified by parsing the example, printing it, and comparing the result against the input, rather than by reading the code.

Status	Meaning
✅	Modeled by a dedicated AST node and round-trips
⚠️	Round-trips, but is stored as plain text or produces a structure that does not match make's own reading
(blank)	Not supported, see the remark for current behavior

Syntax	Example	Parser	Printer	Builder	Remarks
general
empty file		✅	✅
blank lines between elements	target:\n\n\ntarget2:	✅	✅		blank lines are recreated from stored positions
leading blank lines	\n\ntarget:	✅	✅
file of blank lines only	\n				prints as an empty file
no trailing newline	target: prereq	⚠️	⚠️		the printer always terminates the final line with \n
trailing whitespace	target: prereq \n				dropped by the scanner, so it is absent from printed output, with or without a trailing newline
CRLF line endings	target: prereq\r\n	✅	✅		the line ending is recorded once per file, a file that mixes endings is normalized to its dominant ending
escaped #	target: foo\#bar	✅	✅		kept verbatim, no unescaping is performed, and foo\\#bar escapes the backslash so the # starts a comment
line continuation, recipe	\techo x \\\n\t\ty	⚠️	⚠️		each physical line becomes its own Recipe
line continuation, prerequisites	target: b \\\n\tc	⚠️	⚠️		the \\ is a prerequisite and the next line becomes a recipe
line continuation, variable value	VAR := b \\\n\tc	⚠️	⚠️		the continued line becomes a BadObj, #139
comments
top-level comments	# comment text	✅	✅
comment groups	# comment text\n# more comment text	✅	✅
groups separated by a blank line	# a\n\n# b	✅	✅		each group is a separate CommentGroup
comments with no leading space	#comment text	✅	✅		Comment.Text keeps the text after # verbatim, so the original spacing round-trips
recipe comments	target:\n\trecipe # comment text	✅	✅		these are not make comments and are included in the recipe text
rule comments	target: prereq # comment text	✅	✅		recorded on Rule.Comment, which shares the target line rather than starting one
variable comments	VAR = val # comment text	✅	✅		recorded on Variable.Comment
comments after a word	target: prereq# comment text	✅	✅		make removes the comment before reading the line, so the prerequisite is prereq
conditional comments	ifeq (a,b) # comment text	✅	✅		recorded on the directive, so ifeq, ifdef, a bare else, and endif each keep the comment ending their own line
else if comments	else ifeq (a,b) # comment text	✅	✅		recorded on the condition rather than on ElseBlock.Comment, which holds the comment ending a bare else
conditional with no condition	ifeq # comment text				parse error, and the comment is consumed with the token the parser expected
comments in a define body	define B\ntext # text\nendef	⚠️	⚠️		not comments to make, every line of the body is a BadObj holding the text
rules
targets	target:, target :	✅	✅	✅
multiple targets	target1 target2:	✅	✅	✅	the builder places every target at the same position, #120
pre-requisites	target: prereq	✅	✅
order-only pre-requisites	target: | prereq	✅	✅
order-only, unseparated	target: prereq|order-only	✅	✅		only the first | separates, a later one is text of the prerequisite holding it
pattern rules	%.o: %.c	✅	✅		% is ordinary text, the pattern itself is not modeled
suffix rules	.c.o:	✅	✅		ordinary target text
special targets	.PHONY: target	✅	✅		ordinary target text, see ast/target for the known names
wildcard pre-requisites	target: *.c	✅	✅		ordinary prerequisite text
static pattern rules	a.o: %.o: %.c				parse error on the second :
double-colon rules	target:: prereq				parse error on the second :
grouped targets	a b &: prereq	⚠️	⚠️		round-trips, but & is parsed as an additional target
escaped spaces in targets	a\\ b:	⚠️	⚠️		round-trips, but yields two targets rather than one
target-specific variables	target: VAR = value				parse error on =
pattern-specific variables	%.o: VAR = value				parse error on =
recipes
recipes	target:\n\trecipe text	✅	✅
multiple recipe lines	target:\n\tone\n\ttwo	✅	✅
recipe prefix characters	\t@echo, \t-rm x, \t+$(MAKE)	✅	✅		part of the recipe text
escaped $$ in recipes	\techo $$HOME	✅	✅		recipe bodies are captured verbatim
automatic variables in recipes	\tcp $< $@	⚠️	⚠️		verbatim text rather than VarRef nodes
variable references in recipes	\techo $(VAR)	⚠️	⚠️		verbatim text rather than VarRef nodes
blank line inside a recipe block	target:\n\tone\n\n\ttwo	⚠️	⚠️		ends the rule, following lines become BadObj nodes, #139
semicolon delimited recipes	target: ; recipe	✅	✅		a semicolon ends the prerequisite list wherever it is written, so target: ;recipe and target: prereq;recipe parse as well
custom .RECIPEPREFIX	.RECIPEPREFIX = >\n>recipe	✅	✅		the first character of the value introduces a recipe, an expanded value leaves the prefix unchanged
.RECIPEPREFIX of ;	.RECIPEPREFIX = ;\n;recipe	✅	✅		recorded as a TEXT prefix like any other custom one, so it stays apart from the ; of a target: ; recipe
variables
assignment operators	=, :=, ::=, :::=, ?=, !=	✅	✅
append assignment	VAR += x	✅	✅		the unspaced VAR+=x parses as well, see the row below
empty declarations	VAR :=	✅	✅
multi-word values	VAR := foo.c bar.c	✅	✅
unspaced :-prefixed operators	VAR:=x, VAR::=x, VAR:::=x	✅	✅		: delimits, so these scan correctly
unspaced =, ?=, !=, +=	VAR=x, VAR?=x, VAR!=x, VAR+=x	✅	✅		= and ! delimit, so the name, the operator, and the value are separate tokens
operators in a value	X = a = b, CFLAGS=-DFOO=1	✅	✅		make reads the first operator on a line, so a later =, !, or : is text of the value
assignment with no name	=x				parse error: variable name is empty, which is what make reports as well
definition blocks	define FOO\nbody\nendef	✅	✅		one DefineDir holding the body as the literal lines make records, a nested define is counted the way make counts it
definition block operators	define FOO :=, define FOO +=	✅	✅		every assignment operator an ordinary assignment accepts
undefine	undefine VAR	✅	✅		one UndefineDir, a directive naming several variables is a parse error and the line is kept as a BadObj
override	override VAR := x	⚠️	⚠️		no dedicated AST node, stored as a BadObj
export and unexport	export VAR, export VAR := x	⚠️	⚠️		no dedicated AST node, stored as a BadObj
private	private VAR := x	⚠️	⚠️		no dedicated AST node, stored as a BadObj
variable references
parentheses and braces	$(VAR), ${VAR}	✅	✅	✅
single character	$V	✅	✅
in targets	${VAR}:, $(FOO) $(BAR):	✅	✅	✅
in pre-requisites	target: ${FOO}	✅	✅
in variable values	A := $(B)	✅	✅
in conditions	ifeq ($(V),x)	✅	✅
escaped $$	target: $$V	⚠️	⚠️		round-trips, but is stored as an ast.Text holding $$ rather than a dedicated node
adjacent to text	target: prefix$(FOO), a$(F)b	✅	✅		$ delimits, so the text and the reference are one JuxtaposedExpr rather than one TEXT token
a $ that opens nothing	X := a$, X := $ b	✅	✅		a $ with no expansion after it is an ast.Text holding the character
joined expressions	target: $(FOO)bar, $$(notdir x)	✅	✅		expressions written with no blank between them are one JuxtaposedExpr
substitution references	$(VAR:.c=.o)				parse error on :
computed names	$($(VAR))				parse error on the inner $
names matching a builtin	$(dir), $(file), $(word)	⚠️	⚠️		round-trips, but is a FuncCall with no arguments, which is how make reads it
functions
text and filename functions	$(subst a,b,$(S)), $(wildcard *.c)	✅	✅	✅	a FuncCall, arguments split on the commas make splits on
$(shell ...)	A := $(shell echo x)	✅	✅	✅	the command is argument text, it is not parsed as shell syntax
$(call ...) and $(eval ...)	A := $(call f,x)	✅	✅	✅	the called name is the first argument, not a VarRef
control functions	$(foreach v,$(L),$(v)), $(if $(V),x,y)	✅	✅	✅	a comma inside a nested call belongs to the argument holding it
logging functions	$(info msg), $(warning msg), $(error msg)	✅	✅	✅	info is absent from the manual's list, so it is a call to a name make does not know
conditionals
equality directives	ifeq, ifneq	✅	✅
definition directives	ifdef, ifndef	✅	✅
parentheses syntax	ifeq (foo, bar)	✅	✅
double quotes	ifeq "foo" "bar"	✅	✅
single quotes	ifeq 'foo' 'bar'	✅	✅
mixed quotes	ifeq "foo" 'bar'	✅	✅
empty argument	ifeq ($(V),), ifeq "" ""	✅	✅
else	ifdef V\na:\nelse\nb:\nendif	✅	✅
else if chains	else ifdef W, else ifeq (a, b)	✅	✅
nested conditionals	ifdef V\nifdef W\nendif\nendif	✅	✅
wrapping variables or rules	ifdef V\nA := x\nendif	✅	✅
inside a recipe block	target:\nifdef V\n\techo x\nendif	✅	✅		the conditional joins the recipe list of the rule it follows
other directives
include directives	include a.mk, -include, sinclude	⚠️	⚠️		no dedicated AST node, stored as a BadObj
vpath directive	vpath %.c src	⚠️	⚠️		no dedicated AST node, stored as a BadObj
VPATH variable	VPATH := src	✅	✅		an ordinary variable assignment
load directive	load a.so	⚠️	⚠️		not tokenized, stored as a BadObj
many other things					please open an issue if there is anything missing you'd like to see!

Constructs marked unsupported either report a parse error or print differently than they were written. Syntax the parser has no node for is stored verbatim as an ast.BadObj and printed back unchanged, so the line survives a round trip.

Reference Coverage

The names make understands are enumerated by token, ast/target, and ast/variable. internal/conformance compares each enumeration against the summaries published in the GNU Make manual, so syntax added to make cannot go unnoticed.

Enumeration	go-make	Manual	Source
directives	17	17	Quick Reference
built-in functions	37	37	Quick Reference
special variables	14	14	Quick Reference
special targets	17	17	Special Built-in Target Names

The comparison runs against internal/conformance/testdata/quickref.json, a fixture extracted from those two pages, so the suite needs no network access. Run make sync-quickref to refresh it. The manual pages are pinned by content hash in flake.nix and fetched with fetchurl, so a refresh either reproduces the same input or fails with a hash mismatch reporting that the manual changed. A weekly workflow runs the refresh and fails if the fixture is out of date, so drift in the manual surfaces without anyone rerunning it by hand. It is not part of the per-commit checks, because it reaches gnu.org and an outage there is not a reason to block a merge.

Automatic variables ($@, $<, $(@D), and the rest) are summarized by the manual but are not enumerated here, so they are excluded from the comparison.

Will Not Support

Nothing, at this time

Workflow

Pre-Requisites

Go toolchain for the version listed in go.mod

Building

go-make is itself built using make.

Targets	Description
default goal	Runs the build target
build	Runs go build to verify the code compiles
test	Test changed packages
test_all	Test all packages
sync-quickref	Refresh the pinned GNU Make manual fixture used by internal/conformance (requires nix)
clean	Remove .make directory and coverage report
cover	Collect coverage for all tests and print report
tidy	Runs go mod tidy
dev	Setup the developer environment

Developer Environment

Apart from the Go toolchain, the only main dependency is the ginkgo cli to run tests. Targets will obtain dependencies automatically as needed.

Binaries are stored in a .gitignored bin/ directory at the root of the repository. An example .envrc file for direnv is provided in hack/example.envrc to add ./bin to your PATH automatically. To use it, run make .envrc or make dev. This will copy hack/example.envrc to .envrc at the root of the repository.

References

GNU Make Quick Reference: https://www.gnu.org/software/make/manual/html_node/Quick-Reference.html