proposal/L86-aspect-based-python-bazel-rules.md

Aspect-based Python Bazel Rules

• Author(s): Michael Beardsworth
• Approver: gnossen
• Status: Draft
• Implemented in: Bazel and Starlark
• Last updated: Sept. 8, 2021
• Discussion at: https://groups.google.com/g/grpc-io/c/NXdtW5xe9Js

Abstract

Proposes modifications to gRPC's Bazel rules for Python codegen to improve dependency tracking.

Background

• gRPC provides build rule logic written in Starlark to support Protobuf and gRPC Python codegen with Bazel.
• These rules (py_proto_library and py_grpc_library) each generate code from a single proto_library's protos.
• The produced Python libraries (or equivalently-functioning PyInfo providers) do not express dependencies on the generated Python code for the proto_library's dependencies.
• Users are surprised by this behavior. See #23769 for an example bug.
• This behavior is particularly surprising for Google-internal users. The internal versions of py_proto_library and py_grpc_library propagate Python dependencies.

Related Proposals:

py_proto_library and py_grpc_library are implemented in python_rules.bzl. There doesn't seem to be a proposal for that, though.

Proposal

• We propose rebuilding the py_proto_library and py_grpc_library rules to use aspects to perform code generation steps. This approach corresponds to the Google-internal design for these rules.
• The _gen_py_proto_aspect aspect visits proto_library rules to generate Python code for Protobuf. gRPC-owned code is still responsible for generating the Python code.
• The aspect produces a custom providers PyProtoInfo that wraps a PyInfo provider to avoid creating spurious dependencies for Python users that interface with the proto_library rules through some other means.
• The py_proto_library and py_grpc_library rules will only be responsible for collecting the PyInfo providers from their dependencies.
• The plugin attribute must be removed from py_proto_library. Aspects require the declaration of all possible parameter values up front, so it would not be possible for the new aspects to continue supporting arbitrary plugins. (Note that the plugin feature is not used in gRPC. It was introduced to support GAPIC, which no longer uses the feature.)
• In some use cases in gRPC e.g. grpcio_channelz, the py_proto_library rule is located in a different package than the corresponding proto_library rule. This rule layout is needed to generate Python code with import paths that match the Python package layout, rather than the directory structure containing the .proto files. Since aspect-based code generation associates the generated code with the Bazel package (i.e. repository path) of the proto_library rule rather than the py_proto_library rule, we need special handling for this case. When the py_proto_library is in a different Bazel package than the proto_library rule, we generate an additional set of Python files that import the generated Python files under the old convention. Additionally, an imports attribute is added, to allow the caller to add import paths (similar to the behavior of py_library. With these two changes, existing Python code can remain unmodified, with a minimal increase in BUILD file complexity.
• No behavior change should be observed by the user of py_proto_library or py_grpc_library unless they rely on the (removed) plugin attribute, or if they use the new imports attribute.

Rationale

The proposed approach addresses the open bug and corrects the dependency issues. However, it requires the removal of a (likely unused) feature of the build rules.

Alternatively a set of repository rules could be introduced that allow users to inject py_proto_library and py_grpc_library implementations into the gRPC Bazel build system. This alternative approach would allow users to work around the dependency issues by taking on additional burden. The new build logic could be moved into a separate repository, or potentially upstreamed to a Bazel-owned repository.

Implementation

The implementation is mostly complete by beardsworth in #27275.

Testing

Existing unit tests cover the current API surface. An additional test case will be added to the Bazel Python test repo that covers transitive dependency resolution.

Open issues (if applicable)

It is difficult to determine how many users rely on the plugin attribute. If many users rely on this behavior then it may not be feasible to adopt this proposal.