Harmonizing the build with Bazel: Part 1

At Fullstory, we are undergoing a long-term migration towards Bazel as our single standard build orchestrator. We want to share some about why we're doing this and what we expect from Bazel in a two-part blog series. Part One will specifically focus on describing the build architecture challenges we face at Fullstory as we grow.

An ensemble of soloists

Over the years, Fullstory's codebase has grown as one might expect; business needs evolve naturally and the engineering team grows accordingly. This growth triggers necessary shifts in development patterns; patterns which supported five engineers will not support ten, twenty, fifty engineers. Competing patterns naturally proliferate in the codebase (relevant xkcd). And as developers migrate between projects, swaths of stable code become either ambiguously owned or owned by teams lacking contextual knowledge.

In our production code, Fullstory has managed these competing patterns and contextual losses effectively. But to continue our growth, we explicitly chose to wrangle these issues within our architecture for build orchestration, test management, and CI configurations. To that end, we assembled our “Build & Productivity” (B&P) team, of which I am a proud member!

Build architecture discordance

The core of Fullstory’s stack depends on Go (our backend services), TypeScript/SCSS/Node (our frontend artifacts), and Docker (our deployment images). However the build processes are spread among various tools and technologies:

• The backend services are generated via make and internal Go tools called go-install and services.

• The frontend artifacts are crafted via a mixture of Gulp tasks or package.json scripts statements.

• Deployment artifacts are packaged and uploaded via dockerizer, another internal tool written in Go.

The backend and Docker build patterns have remained stable over time. Our internal Go tools support various configurations and allow the backend teams to work with little confusion. Building and executing remains clear, direct, and well-described due to the tooling’s flexibility.

Unfortunately, evolving standards have led to a diverging build landscape for the frontend. Among an attempt at standardizing with Gulp, a strong following of package.json scripts, and various bash scripts–all introduced through continued product growth and evolving technological needs–the build lacked cohesiveness and comprehensibility. If a new team member tried to detangle these scripts solo, they would certainly find themselves in a snarl of spaghetti.

A fun example of an build execution path for the frontend:

In addition, our build tooling is oriented towards “build the world” for everyone. As developers, we often wait for an unrelated part of the stack to rebuild, and after recurrent dealings with this, we inevitably create our own paths and proliferate new scripts with varying degrees of success. Worse, as humans we can unconsciously learn to accept this pain as a fixed part of the process, losing our ability to distinguish between necessary evils and opportunities for improvement. Fullstory remains acutely aware of this concern as it represents a failure requiring bionics.

Continuous integration cacophony

For CI, it’s common to start simple by building and testing everything. This does not scale as your team and product grows. You become motivated to build and test the minimum needed for the change in order to waste less time for developers waiting on results, reduce the impact of test flakiness, and save money.

But as Fullstory has grown, the build architecture struggled to meet these needs. Instead, our CI workflows had evolved their own architecture to support this build and test filtering. Using a manual mapping of code owners to test suites, the workflows decide which suites need execution based on the code changes being tested. This strategy has proven brittle. We would gain and lose coverage as the codebase changed, as we lacked appropriate processes to maintain the owners->suites mapping as well. We repeatedly found ourselves in one of two states: overaggressive, wasting time and resources running unaffected tests, or under-covered, not running necessary tests and allowing a breakage into the main branch.

Conducting ourselves accordingly

Organic growth as a product and necessary building-for-the-present have left us with a working but disorganized build architecture. Going forward, we want to bring clarity to building at Fullstory! Follow along in Part Two, which will speak towards Bazel helping solve those issues.