Select unselectable in Bazel (or What Every Programmer Should Know About Bazel)

Caution. This article is intended for people who already use Bazel for their projects. I do not recommend to read it for beginners, underages, people having pregnancy, fragile mental state and pronounced neurotypical persons.

Let’s assume you want to write a macro. It will receive list of files, for which you want to generate set of rules which will produce some files. One can write it as follows:

def generate_some_cool_compilation_rules(data=[]):
    for input, output in data:
        native.genrule(
            name = "generate-%s" % (output),
            srcs = [input],
            outs = [output],
            cmd = "nasm -o $@ $<"
        )

Let me try to take this one step further. You want to conditionally generate outputs depending on some defines and use select() in arguments of the macro [5].

config_setting(name = "yes", define_values = { "type" : "YES" })
config_setting(name = "no", define_values = { "type" : "NO" })

generate_some_cool_compilation_rules(
    data = [("input.asm", "output.o")]
        + select({
            "//conditions:default" : [],
            "//:yes" : [("input_x64.asm", "special_output.o")]
        })
)

Will it blendwork? Actually, it won’t:

ERROR: BUILD:26:1: Traceback (most recent call last):
        File "BUILD", line 26
                generate_some_cool_compilation_rules(data = ([("input.asm", "output.o...")]})))
        File "rules.bzl", line 2, in generate_some_cool_compilation_rules
                for (input, output) in data: ...
type 'select' is not iterable

The thing is one can’t receive result of select expression at loading phase (and that is the only phase when macros work). You won’t be able to find out virtually anything about these expressions:

print(42 + select({ "//:yes" : 123 }))       # 42 + select({"//:yes": 123})
print(type(42 + select({ "//:yes" : 123 }))) # 'select'
print(dir(42 + select({ "//:yes" : 123 })))  # []

The only place where you would be able to see the result of evaluation of such expression from the code is implementation of some rule. Unfortunately, it is being evaluated in analysis phase which means it is too late to call other rules. If one is going to use Bazel like nice guys, he will be able to find out value of the only select(), and chaining of macros would be impossible [1].

Okay, Bazel. We can go chaotic way and write our own functions which will work. ^_^

def duct_tape(arg):
    return [arg]

It’s easy, isn’t it? Here is how it will be used:

generate_some_cool_compilation_rules(
    data = [("input.asm", "output.o")]
        + duct_tape({
            "//conditions:default" : [],
            "//:yes" : [("input_x64.asm", "output_x64.o")]
        })
)

And here is macro that started it all:

def generate_some_cool_compilation_rules(data=[]):
    for element in data:
        if type(element) == "tuple": # Badger, badger, badger, badger.
            input, output = element
            native.genrule(
                name = "generate-%s" % (output),
                srcs = [input],
                outs = [output],
                cmd = "nasm -o $@ $<"
            )
        elif type(element) == "dict": # Ah, snake, it's a snake.
            outputs = {}
            for selector, sublist in element.items():
                for input, output in sublist:
                    if not output in outputs:
                        outputs[output] = []
                    outputs[output].append((selector, input))
            for output, info in outputs.items():
                native.genrule(
                    name = "generate-%s" % (output),
                    srcs = select({ "//conditions:default" : [] }
                        + { selector : [input] for selector, input in info }),
                    outs = select({ "//conditions:default" : [] }
                        + { selector : [output] for selector, input in info }),
                    cmd = select({ "//conditions:default" : "true" }
                        + { selector : "nasm -o $@ $<" for selector, input in info })
                )
        else:
            fail("Mushroom, mushroom!")

Wait, Bazel tells us that:

ERROR: BUILD:37:1: //:generate-special_output.o: attribute "outs" is not configurable
ERROR: BUILD:37:1: //:generate-special_output.o: missing value for mandatory attribute 'outs' in 'genrule' rule

Unfortunately, we can’t just conditionally declare a rule in macro, because at this point result of select() is still unknown. And we can’t configure outs attribute [2].

How to configure unconfigurable (part 2)

It’s time to clarify the problem. Why would we need optional output? Actually, we can do the following with it:

  1. Fill it by other rules in alternative cases.
  2. Do not fill it, but reference it in other rule(s), in this case we will get an error.
  3. Do not fill it and do not use it.

Result of case #2 is as one expects and case #3 can be solved making custom rule specifying DefaultInfo(files = []) in output [3]. Let’s consider case #1:

generate_some_cool_compilation_rules(
    data = [("input.asm", "output.o")]
        + duct_tape({
            "//conditions:default" : [],
            "//:yes" : [("input_x64.asm", "special_output.o")]
        })
)

generate_some_cool_compilation_rules(
    data = [("input2.asm", "output2.o")]
        + duct_tape({
            "//conditions:default" : [],
            "//:no" : [("input_x64_alternative.asm", "special_output.o")]
        })
)

However we shall note, that we don’t need the output by itself, but we want to use it as an input for other rules. The example is as follows:

cc_binary(
    name = "test",
    srcs = [
        "output.o",        # Rule #1
        "output2.o",       # Rule #2
        "special_output.o" # Rules #1 and #2
    ]
)

How would that be implemented? The simplest way I can imagine (for the sake of clarity) is to generate files for all combinations of selectors and reality and use only a subset of them further:

def beautify_label(label):
    return label.replace('/', '_').replace(':', '_')

def generate_some_cool_compilation_rules(data=[]):
    results = None
    for element in data:
        if type(element) == "tuple": # Badger, badger, badger, badger.
            input, output = element
            native.genrule(
                name = "generate-%s" % (output),
                srcs = [input],
                outs = [output],
                cmd = "nasm -o $@ $<"
            )
        elif type(element) == "dict": # Ah, snake, it's a snake.
            outputs = {}
            for selector, sublist in element.items():
                for input, output in sublist:
                    if not output in outputs:
                        outputs[output] = []
                    outputs[output].append((selector, input))
            for output, info in outputs.items():
                for selector, input in info:
                    native.genrule(
                        name = "generate-%s" % (beautify_label(selector) + output),
                        srcs = select({ "//conditions:default" : [] } + { selector : [input] }),
                        outs = [beautify_label(selector) + output],
                        cmd = select({ "//conditions:default" : "true > $@" }
                            + { selector : "nasm -o $@ $<" })
                    )
                    result = select({ "//conditions:default" : [] }
                        + { selector : [beautify_label(selector) + output] })
                    if results == None:
                        results = result
                    else:
                        results += result
        else:
            fail("Mushroom, mushroom!")
    return results

Let’s change our BUILD file according the macro:

my_special_output = generate_some_cool_compilation_rules(
    data = [("input.asm", "output.o")]
        + duct_tape({
            "//conditions:default" : [],
            "//:yes" : [("input_x64.asm", "special_output.o")]
        })
)

my_special_output += generate_some_cool_compilation_rules(
    data = [("input2.asm", "output2.o")]
        + duct_tape({
            "//conditions:default" : [],
            "//:no" : [("input_x64_alternative.asm", "special_output.o")]
        })
)

cc_binary(
    name = "test",
    srcs = ["output.o", "output2.o"] + my_special_output
)

print(my_special_output) will give us:

select({"//:yes": ["___yesspecial_output.o"], "//conditions:default": []})
    + select({"//:no": ["___nospecial_output.o"], "//conditions:default": []})

Some overhead was introduced because we added extra outputs. If one run bazel build ... they will run redundant commands (true). That can be eliminated by custom rule with DefaultInfo(files = []).

Final things

Unfortunately, we can’t simply declare such outputs across different BUILD files. Here comes filegroup [4]:

filegroup(
    name = "special_output_part1",
    srcs = generate_some_cool_compilation_rules(
                data = [("input.asm", "output.o")]
                    + duct_tape({
                        "//conditions:default" : [],
                        "//:yes" : [("input_x64.asm", "special_output.o")]
                    }))
)

filegroup(
    name = "special_output_part2",
    srcs = generate_some_cool_compilation_rules(
                data = [("input2.asm", "output2.o")]
                    + duct_tape({
                        "//conditions:default" : [],
                        "//:no" : [("input_x64_alternative.asm", "special_output.o")]
                    }))
)

filegroup(
    name = "special_output",
    srcs = ["special_output_part1", "special_output_part2"]
)

cc_binary(
    name = "test",
    srcs = ["//:output.o", "//:output2.o", "//:special_output"]
)

References

  1. Extending Bazel -> Overview
  2. Build Encyclopedia -> Build Concepts
  3. Extending Bazel -> Concepts -> Rules
  4. Build Encyclopedia -> Rules -> General
  5. Build Encyclopedia -> Functions