Bazel Blog

Thank you for BazelCon 2017

We are truly thankful to our community for making our first annual Bazel Conference a success! Check out all the videos of all the talks from BazelCon 2017 on YouTube.

Your feedback reflected high level of satisfaction, and there was something of interest for everyone:

  • Bazel usage and migration stories from TensorFlow, Kubernetes, SpaceX, Pinterest, Wix, Stripe, DataBricks, and Dropbox.
  • Talks about upcoming features and tools such as remote execution & caching, buildozer, robolectric tests, PodToBUILD, bazeltfc, rules_typescript presented by Uber, Two Sigma, Google, Pinterest, and Asana.
  • Office hours where you got your Bazel questions answered, met engineers, and debugged on the fly. One attendee used their session to configure remote execution with buildfarm!

BazelCon2017 by the Numbers:

  • 200+ attendees
  • 60 organizations
  • 30 speakers
  • 12 informative talks
  • 14 hours of hands-on debugging and Q&A during Office Hours

What we heard from you:

  • Bazel Query is great!
  • Python is used more widely than we realized, and needs better Bazel support.
  • Reproducible builds are important especially when you're flying rockets, building autonomous vehicles, delivering media, and calculating financial transactions, leading to wider Bazel adoption by these communities.
  • Documentation is often hard to find, and is either too basic or too advanced.
  • Bazel's parallelism is impressive: much faster than Maven.
  • IDE integration is important, particularly with Visual Studio, CLion, XCode.
  • Build Event Protocol enables many options for internal visualization of events.
  • Aspects are a powerful tool.

What we can do next:

  • Work on better documentation and training, particularly for intermediate-level topics.
  • Prioritize IDE integration.
  • Engage the community in building better Python support in Bazel.
  • Implement improved support for third-party dependencies.
  • Continue work on cross-platform improvements.
  • Engage the community in wider adoption and contribution to Bazel.

What you can do next:

  • Contribute to Bazel, particularly to the new community driven buildfarm effort.
  • Kick off local meet-ups (we will reach out to volunteers who responded to this in our survey).
  • Get more info on Bazel at bazel.build.
  • Join the discussion on bazel-discuss@googlegroups.com.

We look forward to working with you and growing our Bazel user community in 2018!

By Helen Altshuler and David Stanke

Bazel on Windows -- a year in retrospect

Bazel on Windows is no longer experimental. We think Bazel on Windows is now stable and usable enough to qualify for the Beta status we have given Bazel on other platforms.

Over the last year, we put a lot of work into improving Bazel's Windows support:

  • Bazel no longer depends on the MSYS runtime. This means Bazel is now a native Windows binary and we no longer link it to the MSYS DLLs. Bazel still needs Bash (MSYS or Git Bash) and the GNU binutils (binaries under /usr/bin) if your dependency graph includes genrule or sh_* rules (similarly to requiring python.exe to build py_* rules), but you can use any MSYS version and flavour you like, including Git Bash.
  • Bazel can now build Android applications. If you use native android_* rules, Bazel on Windows can now build and deploy Android applications.
  • Bazel is easier to set up. You now (typically) no longer need to set the following environment variables:
    • BAZEL_SH and BAZEL_PYTHON -- Bazel attempts to autodetect your Bash and Python installation paths.
    • JAVA_HOME -- we release Bazel with an embedded JDK. (We also release binaries without an embedded JDK if you want to use a different one.)
  • Visual C++ is the default C++ compiler. We no longer use GCC by default, though Bazel still supports it.
  • Bazel integrates better with the Visual C++ compiler. Bazel no longer dispatches to a helper script to run the compiler; instead Bazel now has a CROSSTOOL definition for Visual C++ and drives the compiler directly. This means Bazel creates fewer processes to run the compiler. By removing the script, we have eliminated one more point of failure.
  • Bazel creates native launchers. Bazel builds native Windows executables from java_binary, sh_binary, and py_binary rules. Unlike the .cmd files that Bazel used to build for these rules, the new .exe files no longer dispatch to a shell script to launch the xx_binaries, resulting in faster launch times. (If you see errors, you can use the --[no]windows_exe_launcher flag to fall back to the old behavior; if you do, please file a bug. We'd like to remove this flag and only support the new behavior.)

Coming soon

We are also working on bringing the following to Bazel on Windows:

  • Android Studio integration. We'll ensure Bazel works with Android Studio on Windows the same way it does on Linux and macOS. See issue #3888.
  • Dynamic C++ Linking. Bazel will support building and linking to DLLs on Windows. See issue #3311.
  • Skylark rule migration guide. We'll publish tutorials on writing Skylark rules that work not just on Linux and macOS, but also on Windows. See issue #3889.

Looking ahead, we aim to maintain feature parity between Windows and other platforms. We aim to maximize portability between host systems, so you get the same fast, correct builds on your developer OS of choice. If you run into any problems, please don't hesitate to file a bug.

By László Csomor

Bazel Conference 2017

Bazel team is pleased to announce our first annual Bazel Conference, focused on the needs of our community. The conference will feature user stories and feedback, migration talks, roadmap, hands-on and break-out tech sessions with Bazel engineers, contributors and users.

Dates: November 6 and 7, 2017 Location: Sunnyvale, California

We are inspired by our community engagement in the upcoming conference. We've received a large number of requests for talks and topics for discussion, and have scheduled tech talks by engineers at large and small companies on working with Bazel in wide variety of languages, including Scala, JavaScript and Go. We are looking forward to having 200 engineers from 35 companies across the world share their experience with Bazel.

We are humbled by the commitment to make Bazel even better, and are seeing engineers develop advanced features like Remote Execution, and sharing migration tips, tricks and tools. You will hear user stories and tips about iOS migration, and TensorFlow and Kubernetes experience with Bazel. We will also discuss as a community different tools out there that could be open sourced.

We are excited to see all of you at this first annual Bazel Conference in Sunnyvale, California on November 6 and 7, 2017!

Register by October 15, as seating is limited and we won't allow walk-ins. Detailed schedule will be published mid October, and location details will be sent out to registered attendees.

By Helen Altshuler

The New World of Bazel Releases

Bazel has been open-sourced exactly 2.5 years ago. It continues to be quite a journey, and we are very happy we have acquired some fellow travellers: many projects, organizations and companies that we all know and love rely on Bazel every day.

As our community grows, we owe to it a transparent and predictable release process. So we are taking some steps to bring more clarity and order to the world of Bazel releases:

  • We will cut (from master) a candidate for minor releases (0.7, 0.8 and so on) every month on approximately first business day of the month. The planned target dates for the cuts are published as GitHub issues with label 'release'
  • After the release candidate is cut, we will let it bake for two weeks before promoting it to the release. We will cherry-pick fixes for critical bugs, issuing new release candidates (0.7.0rc1, 0.7.0rc2 and so on), and release 0.minor.0 version at the end of baking period.
  • If critical bugs are discovered after the release, we will fix them and issue patch releases as needed (0.7.1, 0.7.2 and so on). Patch releases are always patches on top of existing minor releases - they are never cut from master.
  • No new features or backward incompatible changes ever appear in patch releases, and no new features or backward incompatible changes are added to release candidates after they have been cut.

Our website has more details on release policy.

As a result of this change, we now issue one minor release per month.

Our roadmap reflects our vision for Bazel 1.0 and beyond. We will annotate features on the roadmap with the release versions as those features get shipped.

By Dmitry Lomov

Bazel Conference 2017

Bazel team is pleased to announce our first annual Bazel Conference, focused on the needs of our community. The conference will feature user stories and feedback, migration talks, roadmap, hands-on and break-out tech sessions with Bazel engineers, contributors and users.

Dates: November 6 and 7, 2017 Location: Sunnyvale, California

We are inspired by our community engagement in the upcoming conference. We've received a large number of requests for talks and topics for discussion, and have scheduled tech talks by engineers at large and small companies on working with Bazel in wide variety of languages, including Scala, JavaScript and Go. We are looking forward to having 200 engineers from 35 companies across the world share their experience with Bazel.

We are humbled by the commitment to make Bazel even better, and are seeing engineers develop advanced features like Remote Execution, and sharing migration tips, tricks and tools. You will hear user stories and tips about iOS migration, and TensorFlow and Kubernetes experience with Bazel. We will also discuss as a community different tools out there that could be open sourced.

We are excited to see all of you at this first annual Bazel Conference in Sunnyvale, California on November 6 and 7, 2017!

Register by October 15, as seating is limited and we won't allow walk-ins. Detailed schedule will be published mid October, and location details will be sent out to registered attendees.

By Helen Altshuler

Introducing sandboxfs

sandboxfs is a new project to improve the way sandboxing works in Bazel by making it more efficient and correct. It's experimental and subject to change, but it's available now for you to check out! Read on for details.

Correct builds

As our motto claims, correctness is an integral part of Bazel. To achieve correctness, builds must be hermetic and reproducible, which means that all actions invoked by Bazel should be run in a well-defined environment. In other words: we want actions to run within a sandbox.

Let's consider an example. Think of a cc_library target that specifies foo.cc as a source. This cc_library rule contains a compilation action that, for this target, turns the foo.cc source file into a foo.o object file. This action will run clang, and it will need to:

  • read foo.cc,
  • read all the headers required by foo.cc,
  • write foo.o, and
  • possibly write some temporary files under /tmp/.

To be confident this build is correct, Bazel must ensure that the action has read-only access to the identified input files and write-only access to the output and temporary files we expect. Otherwise, Bazel cannot know if the compiler went astray and read random files from the system, making future builds inconsistent. One way to achieve these restrictions is by running each action within a sandbox.

Current sandboxing techniques

Today, Bazel uses different technologies to implement the sandboxing of actions. For example, on Linux, Bazel uses PID- and mount-namespaces, and on macOS, Bazel uses symlinks and the sandbox(7) facility. Note that sandboxing is disabled by default.

However, all these approaches have scalability and performance issues. A typical build action requires hundreds, if not thousands, of files and directories to be mapped into the sandbox. Setting these up takes time. On macOS, this approach is especially problematic because Bazel must create thousands of symlinks every time it invokes an action, and this is slow.

To make things worse, these approaches also have correctness issues. If symlinks are used, some tools (e.g. some compilers or linkers) will decide to extract the real path of such symlinks and work off that path. These tools may end up "discovering" and consuming undeclared files that are siblings of the symlink's target.

Enter sandboxfs

To resolve these issues, we came up with the idea of implementing a FUSE file system that allows us to define an arbitrary view of the host's file system under the mount point. We call this approach sandboxfs.

sandboxfs is efficient

The view sandboxfs offers is cheaply configured at mount time. With sandboxfs there is a single system call to configure the mount point versus thousands of symlink creation and deletion system calls.

The view can also be reconfigured cheaply across different actions, avoiding the need to remount the FUSE file system on each action, which would also be costly.

sandboxfs is correct

The view sandboxfs offers is fully virtual, so sandboxfs can enforce arbitrary read-only and read/write access permissions on any file or directory it exposes. Similarly, because the view is virtual, there are no symlinks involved. sandboxfs exposes real files and directories to the actions, so actions cannot reach into the original locations.

Isn't FUSE slow though?

Yes. As you may know, FUSE is slower than a real file system. This difference is because of the overhead of message-passing between the kernel and userspace, and because of all the context switches that are involved in serving a file system operation.

Our hypothesis is that, because builds are generally not I/O bound, the increased cost of I/O within the sandbox will pay for itself when compared to the cost of setting up and tearing down the sandbox for each action. No more creating thousands of symlinks or defining thousands of mount points.

But for now, that's all that it is: a hypothesis. We haven't finished stabilizing the sandboxfs code base, which means we haven't profiled nor tuned it. The integration points with Bazel are still being defined and implemented, which means it's not yet trivial to test-run Bazel's sandboxing with sandboxfs.

But it's here!

Though still in development, there is no reason to keep the code hostage any longer. We are very happy to announce that, as of today, it's now available as an open-source project under the Bazel umbrella! See:

https://github.com/bazelbuild/sandboxfs/

Keep in mind that this project is still an experiment and highly subject to change. In particular, be aware that the command line and the data formats are most certainly going to mutate (for simplicity if anything). But the current code is now sufficient to experiment and play with.

As sandboxfs is now an open-source project, please take a look and report any features you would like to see, any bugs you encounter, and... if you decide to delve into the code and address any of the many TODOs in it, feel free to send us your Pull Requests!

Enjoy!

Special credits go to Pallav Agarwal, whom we had the pleasure of hosting over a summer internship in the Google NYC office and who wrote the initial version of sandboxfs.

By Julio Merino

Backward compatibility

Bazel is in Beta and we are working hard towards Bazel 1.0 (see the roadmap). We are not there yet, and there are still many things we want to change, clean, and improve. Future releases of Bazel will not be 100% compatible with all previous Beta versions. We understand that breaking changes can be painful for users. That's why we want to make it as easy as we can to migrate to new Bazel versions.

During the Beta period, we may introduce breaking changes in each minor version (0.x). Starting with Bazel 1.0, breaking changes will cause a major version change (1.0, 2.0, etc.). This is known as Semantic Versioning. Major version changes should not be more frequent than once a year, and may be less.

For most breaking changes, we will add a flag, e.g. --incompatible_foo that is disabled by default. The flag --all_incompatible_changes will enable all of these flags at once, so you can see whether you're ready for the next major release.

The Bazel team will use the flags as follow:

  • To introduce a breaking change, we release the change along with a new flag that is unset by default, e.g. --incompatible_foo. This flag enables the new behavior, allowing you to test the future change. Flags for incompatible features are documented in the section Backward compatibility.

  • At a later release, the new flag is set by default. This change is effectively released, so it can happen only at a major version (or minor version during Beta). In the release notes, we tell you which incompatible changes are enabled. The flag for this change still exists, so if needed you can disable it.

  • Finally, the flag may be removed at any release in the future, and so you will no longer be able to disable the behaviour of the change. In the release notes, we list these removed flags.

When you migrate to a new release, we recommend the following workflow:

  • Use --all_incompatible_changes to ensure your code is forward-compatible with the next release. We are aware that you will not always be able to fix every issue, for example, there could incompatibilities with a repository you depend on.

  • When there is a new Bazel release, try your code again with the specific incompatible changes released in the new version. Check the release notes to know which flags to use. After verifying your project with the released changes, you can update to the new version. Verify your project again with the new version, as some small incompatible changes are not introduced behind flags.

  • If many users depend on your repository, please update it quickly after each Bazel release. This update will help your users test their code.

In all cases, the Bazel team will try to make version updates simple. We will try to document clearly the changes and provide good error messages. If anything is unclear, please contact us and we will help you.

To learn more about the changes we are doing, see the backward compatibility page.

By Laurent Le Brun

A new logo and homepage for Bazel

We are glad to unveil a new logo for Bazel:

Bazel logo

This new logo, which feels more modern and balanced, was designed by Sunkwan, creative lead at Google. It takes inspiration from assembled blocks, adds an emotional touch to it and keeps the green color that is now rooted into our identity. It was the top results of our internal and community votes.

Iteration on the Bazel logo

We also updated the homepage of the Bazel website: https://bazel.build

In addition to a visual refresh, this new website features clearer value statements, clearer call to actions and more relevant 'Get Started' links. We are also very proud to showcase a few famous users and their testimonials (feel free to add yours in the dedicated page of our wiki).

Screenshot of the Bazel homepage

Along the way, we moved the content of the main website and the blog in dedicated repositories (site repo, blog repo) and are using different subdomains: The documentation, which source is still hosted with our code in our main bazelbuild/bazel repo, is now served at docs.bazel.build. The blog is served at blog.bazel.build, which should streamline the blog publication process (contributions welcome!).

The work does not stop here, our next steps are to improve our documentation structure and 'Getting Started' content. By the way, feel free to suggest edits to our documentation, it is as easy as clicking the 'Edit' button at the top of documentation pages.

Edit button on Bazel docs

We hope you like this new visual identity.

By Steren Giannini

Strict Java Deps and `unused_deps`

This blog post describes how Bazel implements "strict deps" for Java compilations ("SJD"), and how it is leveraged in unused_deps, a tool to remove unused dependencies. It is my hope this knowledge will help write rules for similar JVM-based languages such as Scala and Kotlin.

What's "Strict Deps"?

By "strict deps", we loosely mean that all directly used classes are loaded from jars provided by a rule's direct dependencies. In other words, if a Java file mentions another class, then it must be reflected in the BUILD file.
(the concept is similar to Buck's "first-order dependencies")

class A {
  void foo(B b) { }  // <---- B is used, therefore we must depend on :B
}
java_library(
    name = "A",
    srcs = ["A.java"],
    deps = [":B"], # <--- this dependency is required
)

Note that any dependencies of B itself are not listed in the deps of A.

The initial motivation for SJD was the ability to remove unused dependencies.

Consider a dependency chain A -> B -> C without strict deps. It's impossible to know if C can be removed from B's deps just by looking at it - all transitive users of B must be considered to make this decision. Strict deps mandates that if A also uses C, it must depend on it directly, therefore making it safe to remove C from B's deps.

unused_deps

unused_deps is a tool to remove dependencies that aren't needed from a java_library (and other Java rules).

When Bazel builds a Java rule :Foo on the command line, it writes two files - Foo.jar-2.params and Foo.jdeps. The former contains the command-line arguments to the Java compiler and the latter contains a serialized src/main/protobuf/deps.proto which specifies which jars were loaded during compilation.

unused_deps loads the two files and figures out which rules aren't needed in a rule's deps attribute, and emits Buildozer commands to delete them.

Implementation

Bazel always passes the entire transitive classpath to javac, not only the direct classpath. This frees the user from having to aggregate their transitive dependencies manually. In other words, javac never fails because of a missing symbol, as long as every rule specifies its direct dependencies. This is done at JavaCompileAction.java#L729.

SJD is enforced by a compiler plugin implemented in StrictJavaDepsPlugin.java. When Bazel constructs the command-line to javac, it specifies which jars come from indirect dependencies using the --indirect_dependency flag. The plugin then walks the .java sources and reports any symbols that come from indirect jars.
(A sketch of how it works: The compiler stores the name of the jar from which a symbol was loaded. The plugin walks the AST after the type annotation phase, and stops at each 'type expression', then checks whether the originating jar is an --indirect_dependency. If it is the plugin generates an error message. The message includes the missing direct dependency to add.)

This approach has the advantage that violations are easy to fix - Bazel tells the user exactly what to do.

Summary

  • Bazel passes all jars from the transitive dependencies of a rule.
  • Bazel notifies the SJD compiler plugin which jars are indirect.
  • During compilation, the compiler plugin reports any symbol mentioned in the Java file that is loaded from an indirect jar.

By Carmi Grushko

Google Summer of Code 2017

Thank you very much to everyone who applied for Google Summer of Code with Bazel. We received many interesting proposals, and we are excited to see that so many of you are enthusiastic about Bazel. Since this is the first Google Summer of Code with Bazel, we decided to mentor only one student. Of course, you are all welcome to contribute to our projects, even if it is outside of Google Summer of Code.

Harmandeep is going to work with us on Bazel this summer, and will develop a tool to provide editor services (e.g. code completion) for BUILD and .bzl files, using the Microsoft Language Server Protocol. For more information, you can check the proposal and follow Harmandeep's blog.

A big thank you to everyone who applied!

By Laurent Le Brun