Harmonizing the build with Bazel: Part 2

At Fullstory, we are undergoing a long-term migration towards Bazel as our single standard build orchestrator. We want to share some about why we're doing this and what we expect from Bazel in a two-part blog series. Part One focused on describing the build architecture issues. Now, we’ll speak towards how Bazel supports our ongoing needs at Fullstory.

Harmony via orchestration

One of the primary benefits of an orchestrator, like Bazel, is to provide a single interface for interacting with the build. The tool organizes and conducts the underlying necessary actions, abstracting away compilers and interpreters into a single mental model for developers.

If you can consolidate to a single orchestrator, engineers can better understand how to build, run, and test various sections of the codebase without knowing the details of the codebase’s varied technology choices. The orchestrator generates a high-level view of the services available and promotes developers’ freedom without causing difficulty for downstream consumers. After all, providing orchestration allows everyone to play their parts!

Wide pre-existing support with extensibility

While Bazel was built to natively support C/C++ and Java, it pushed to provide extensibility that allowed strong open source contributions, supporting other languages and technologies. Included among those contributions are support for Go, JS/TS via Node, SCSS, and Docker, a serendipitous list for FullS

tory. As we extend Bazel to the rest of our stack, there are rulesets for other languages we will need to build: Python, Rust, Android, and iOS/Objective-C.

Meanwhile, its extensibility via the Starlark language allows us to add tooling for more specific needs within our codebase. For example, we've crafted an in-house rule for Webpack to ensure our monorepo's node packages are linked appropriately, as well as wrapped our Jest usage in a macro to make BUILD targets more readable. In the future, we will bring Node tools like typed-scss-modules and in-house tools like Tomato into the build tree and treat them like any other step in the process.

Build dependency trees and caching

One of Bazel's core features is build dependency management. Via the build's definitions, Bazel creates a directed acyclic graph (DAG) of dependencies among the defined actions, using it to ensure build hermeticity (build processes can only access explicitly defined dependencies). In short: Bazel always knows the exact set of necessary resources for any build. Bazel keeps cached action outputs at every step of the DAG, meaning that it can determine which actions to rebuild when files change. Bazel can then upload this cache to a remote location, such as a Google Cloud Storage bucket.

With Bazel's DAG and standardized description of the build, we can tie artifacts together across varying technologies, product areas, and teams. As such, we now have Docker images defined via Bazel, with dependencies being containerized in both the Go service and the frontend's static-asset artifacts. When we build the image, Bazel ensures that each dependency is either rebuilt or pulled from cache, performing the minimum work possible.

This works in CI, too! Our manual mappings get replaced directly by Bazel’s intelligence, building and testing only the pieces changed from the base branch. We’re also aiming for a future improvement via CI: building and caching a main build emulating our local machines. With that cached, developers will always have a cached version to pull down instead of rebuilding unchanged parts of the system.

Automated maintenance

Defining the builds in Bazel adds overhead, specifically in the form of its BUILD files. These files describe the dependency DAG and are the source of truth for what each build requires. Manual management creates a lot of tedious and painful management. Not very bionic, is it?

Thankfully, there are tools out there to help:

• Buildifier: Bazel's BUILD file linter and formatter

• Buildozer: CLI/API that can be used to programmatically change BUILD files, useful for adding your own auto-management tools

• Gazelle: an extensible BUILD file generator/manager, with support for Go and Protocol Buffers defined out-of-the-box

  • We extend Gazelle to add a myriad of frontend build generation/management as well, including rules for TS, SCSS, Webpack, Jest, and ESLint.

With these, the overhead of BUILD-file management becomes a reasonable trade-off for the benefits of Bazel's orchestration.

Straightforward CLI with powerful options

One glance at Bazel’s documentation could lead you to believe that Bazel’s CLI is overengineered and unclear. However, the knowledge most users need to be effective is relatively small. To start, there are only three core actions a user needs to know: build, test, and run. Each of these commands is as direct as it sounds: they build, test, or run the given target. Two other important actions are clean and query. The former cleans up Bazel's local builds while the latter allows developers to inspect and visualize portions of Bazel's DAG. With these five commands, any developer can be effective with Bazel.

Bazel supplies advanced options that can be configured and distributed via a shared .bazelrc. For instance, we share a configuration that sets up our shared remote build and a local-disk build cache, improving local build time when switching branches. We also replace some Bazel defaults to provide a nicer developer experience, such as outputting test errors into the terminal directly and providing quickly-usable debug build profiles.

A beautiful and shared symphony

With Bazel we expect to move into the future with fast, clear, stable builds for all developers at the company. The standardized developer interface, the pre-existing rulesets, and the extensible abstractions for new technologies ensure our developers can interact with the build without deep understanding of the separate portions of the stack. The benefits of the DAG and action cache allows developers to spend more time working and less time waiting, and it improves our ability to make intelligent build decisions in any environment. Best of all, our developers' build experiences will improve by sharing a single knowledge domain around the build itself.

If improving the digital experience excites you, whether for developers, customers, or users, then check out our open roles!