Bazel Blog

Exploring the IntelliJ Bazel Plugin's Sync Process

In this blogpost, we aim to understand IntelliJ with Bazel plugin's sync process by exploring the plugin's generated files and logs.

Introduction

The sync process is central to the user experience of the Bazel IntelliJ plugin. Its purpose is to query Bazel for information and build up IntelliJ's project structure to fit Bazel's model.

It runs automatically during a project import, and manually by either clicking on the sync icon in the menu bar or, partially syncing packages and individual files in contextual menus.

Running a sync generates a .ijwb directory in the project root. While users don’t typically need to know about the contents of this directory, we will explore these files to learn how the plugin works.

We will also explore the process logs, and connect the dots between them and the generated files to crystallize our understanding.

Let's dive in!

Structure of .ijwb

Using a Spring Boot project example, BlazeSyncManager generates this directory on a project sync.

$ tree -a .ijwb/
.ijwb/
├── .bazelproject
├── .blaze
│   ├── libraries
│   ├── modules
│   │   ├── .project-data-dir.iml
│   │   └── .workspace.iml
│   └── remoteOutputCache
└── .idea
    ├── .name
    ├── codeStyles
    │   └── codeStyleConfig.xml
    ├── externalDependencies.xml
    ├── libraries
    │   ├── Runner_deploy_ijar_d8363141.xml
    │   ├── hamcrest_library_1_3_c425095b.xml
    │   ├── spring_beans_5_1_5_RELEASE_c7c017d3.xml
    │   ├── spring_boot_2_1_3_RELEASE_f01f6f60.xml
    │   ├── spring_boot_autoconfigure_2_1_3_RELEASE_77fee7ea.xml
    │   ├── spring_boot_starter_web_2_1_3_RELEASE_968a1469.xml
    │   ├── spring_boot_test_2_1_3_RELEASE_1631a67.xml
    │   ├── spring_boot_test_autoconfigure_2_1_3_RELEASE_9cfb3ab1.xml
    │   ├── spring_context_5_1_5_RELEASE_69b9cfff.xml
    │   ├── spring_core_5_1_5_RELEASE_b1cbb181.xml
    │   ├── spring_test_5_1_5_RELEASE_be462134.xml
    │   └── spring_web_5_1_5_RELEASE_90feac64.xml
    ├── misc.xml
    ├── modules.xml
    ├── runConfigurations.xml
    ├── vcs.xml
    └── workspace.xml

.ijwb is known as the project directory. It contains metadata about the project that bridges Bazel and IntelliJ project models. For Android Studio and CLion, this directory is .aswb and .clwb respectively.

Let's investigate the components of this directory individually.

Project view file

$ tree -a
.
├── .bazelproject 

This is the IJwB project view file . It contains project-wide settings, like targets to sync, Bazel flags, and enabled languages. Check this file into your project's version control to share Bazel project settings.

The default project view file looks like this:

directories:
  # Add the directories you want added as source here
  # By default, we've added your entire workspace ('.')
  .

# Automatically includes all relevant targets under the 'directories' above
derive_targets_from_directories: true

targets:
  # If source code isn't resolving, add additional targets that compile it here

additional_languages:
  # Uncomment any additional languages you want supported
  # android
  # dart
  # kotlin
  # python
  # scala

derive_targets_from_directories collects all build targets recursively from the directories list. You can also specify targets manually in the targets list.

Bazel data subdirectory

.ijwb/.blaze is the Bazel data subdirectory, containing mostly IntelliJ module definitions.

Most of the persisted serialized data is in this directory. For the rest of the persisted data, find their OS-specific cache locations in the IntelliJ documentation.

├── .blaze 
│   ├── libraries

This is the location of the plugin's JAR cache. This helps provide a more robust code navigation experience, but with the possibility of missing changes made by Bazel outside of the IDE view.

│   ├── modules 

This directory contains IntelliJ module definition files.

│   │   ├── .project-data-dir.iml 

A module that includes just the user's data directory. This enables editing the project view without IntelliJ complaining it's outside the project.

│   │   └── .workspace.iml

Monolithic module for the Bazel workspace.

│   └── remoteOutputCache

A general-purpose local cache for output artifacts generated remotely. During a project sync, updated outputs of interest will be copied locally.

The filenames in this directory contain a checksum suffix to ensure a deterministic map from the cache entries sharing the same filename to the original artifact.

IntelliJ configuration subdirectory

.ijwb/.idea contains project-specific settings files managed by IntelliJ. IntelliJ reads XML files in this directory to set up the Project Structure: project, modules, libraries, SDKs, facets.

└── .idea
    ├── .name

Name of the project.

    ├── codeStyles
    │   └── codeStyleConfig.xml

Settings for code styles.

    ├── externalDependencies.xml

Settings for IntelliJ's external dependencies, which are typically plugins. Every Bazel IntelliJ project depends on the Bazel plugin:

<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
  <component name="ExternalDependencies">
    <plugin id="com.google.idea.bazel.ijwb" />
  </component>
</project>
    ├── libraries

Settings for external libraries. Note that this does not map directly to Bazel's concept of external repositories.

    │   ├── Runner_deploy_ijar_d8363141.xml
    │   ├── hamcrest_library_1_3_922bb0fc.xml
    │   ├── spring_beans_5_1_5_RELEASE_2e7c9dd2.xml
    │   ├── spring_boot_2_1_3_RELEASE_f91adddf.xml
    │   ├── spring_boot_autoconfigure_2_1_3_RELEASE_893b8c29.xml
    │   ├── spring_boot_starter_web_2_1_3_RELEASE_9ed0bc68.xml
    │   ├── spring_boot_test_2_1_3_RELEASE_eb13e348.xml
    │   ├── spring_boot_test_autoconfigure_2_1_3_RELEASE_378665d2.xml
    │   ├── spring_context_5_1_5_RELEASE_136cd23e.xml
    │   ├── spring_core_5_1_5_RELEASE_2076efa2.xml
    │   ├── spring_test_5_1_5_RELEASE_2cf15f55.xml
    │   └── spring_web_5_1_5_RELEASE_9cd2a623.xml

In this example, each XML file maps to a single JAR file. Most of them are downloaded through rules_jvm_external into an external repository named @maven. For example, .ijwb/.idea/libraries/hamcrest_library_1_3_922bb0fc.xml contains:

<component name="libraryTable">
  <library name="hamcrest-library-1.3_922bb0fc">
    <CLASSES>
      <root url="jar:///private/var/tmp/_bazel_jingwen/fb23c64ab599b03c55afa6d9c154aecf/execroot/__main__/bazel-out/darwin-fastbuild/bin/external/maven/v1/https/jcenter.bintray.com/org/hamcrest/hamcrest-library/1.3/hamcrest-library-1.3.jar!/" />
    </CLASSES>
    <JAVADOC />
    <SOURCES />
  </library>
</component>
    ├── misc.xml

Miscelleanous configuration.

    ├── modules.xml

Configuration for project modules. In the previous section, we saw that the plugin generated two modules for project data and the workspace itself. We see their .iml files referenced here:

<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
  <component name="ProjectModuleManager">
    <modules>
      <module fileurl="file://$PROJECT_DIR$/.blaze/modules/.project-data-dir.iml" filepath="$PROJECT_DIR$/.blaze/modules/.project-data-dir.iml" />
      <module fileurl="file://$PROJECT_DIR$/.blaze/modules/.workspace.iml" filepath="$PROJECT_DIR$/.blaze/modules/.workspace.iml" />
    </modules>
  </component>
</project>
    ├── runConfigurations.xml

Settings for run configurations.

    ├── vcs.xml

Settings for version control.

    └── workspace.xml

This file contains the last known state of the user's workflow in the IntelliJ application. Using these settings, IntelliJ can recreate the active tabs, settings, recently used run configurations, and VCS tab states after restarting the IDE.

Portability of the .ijwb directory

The .ijwb directory is not completely portable. Files like .bazelproject and codeStyleConfig.xml can be shared project-wide, but workspace.xml and .workspace.iml should be user-specific.

In general, extract .bazelproject file out of .ijwb/ to version control it, and follow JetBrains' recommendations on checking in specific files in the .idea directory.

Plugin cache

There are more plugin-specific cached state in the IDE installation directory. On macOS, this is the ~/Library/Caches/IdeaIC2019.2/blaze/projects directory.

$ tree
.
└── projects

This is the global project persistent data and configuration cache directory.

For every project opened using this IDE installation, there is an entry named after the project and the first 8 characters of a random UUID. The UUID is used to uniquely identify the project location.

    └── springboot-94bce85a
        ├── cache.dat.gz

This is the gzip of the serialized BlazeProjectData protocol buffer. This is written to disk on sync, and read when reopening projects.

message BlazeProjectData {
  reserved 1;
  TargetMap target_map = 2 [deprecated = true];
  BlazeInfo blaze_info = 3;
  BlazeVersionData blaze_version_data = 4;
  WorkspacePathResolver workspace_path_resolver = 5;
  WorkspaceLanguageSettings workspace_language_settings = 6;
  SyncState sync_state = 7;
  TargetData target_data = 8;
}

Most of the language specific data are in TargetData's TargetIdeInfo, defined in intellij_ide_info.proto. This is where the aspect-generated intellij-info.txt files come into play:

message TargetIdeInfo {
  string kind_string = 1;
  TargetKey key = 2;
  ArtifactLocation build_file_artifact_location = 3;
  repeated Dependency deps = 4;
  repeated string tags = 5;
  repeated string features = 6;
  TestInfo test_info = 7;

  // The time this target was most recently synced, in milliseconds since epoch.
  // Not provided by the aspect directly; instead filled in by the plugin during
  // sync.
  int64 sync_time_millis = 20;

  JavaIdeInfo java_ide_info = 100;
  JavaToolchainIdeInfo java_toolchain_ide_info = 101;

  AndroidIdeInfo android_ide_info = 110;
  AndroidAarIdeInfo android_aar_ide_info = 111;
  AndroidSdkIdeInfo android_sdk_ide_info = 112;
  AndroidInstrumentationInfo android_instrumentation_info = 113;

  CIdeInfo c_ide_info = 120;
  CToolchainIdeInfo c_toolchain_ide_info = 121;

  PyIdeInfo py_ide_info = 130;
  GoIdeInfo go_ide_info = 140;
  JsIdeInfo js_ide_info = 150;
  TsIdeInfo ts_ide_info = 160;
  DartIdeInfo dart_ide_info = 170;
  KotlinToolchainIdeInfo kt_toolchain_ide_info = 180;
}

TargetIdeInfo is heart of the plugin's multi-language support. During a sync, each Bazel target is associated with an intellij-info.txt file that contains a text representation of the TargetIdeInfo proto. Here's an example for the main Spring Boot java_binary target:

build_file_artifact_location {
  is_external: false
  is_new_external_version: true
  is_source: true
  relative_path: "src/main/java/hello/BUILD.bazel"
  root_execution_path_fragment: ""
}
deps {
  dependency_type: 0
  target {
    label: "@local_config_cc//:toolchain"
  }
}
deps {
  dependency_type: 0
  target {
    label: "@bazel_tools//tools/jdk:current_java_toolchain"
  }
}
deps {
  dependency_type: 1
  target {
    label: "//src/main/java/hello:lib"
  }
}
java_ide_info {
  jars {
    jar {
      is_external: false
      is_new_external_version: true
      is_source: false
      relative_path: "src/main/java/hello/app.jar"
      root_execution_path_fragment: "bazel-out/darwin-fastbuild/bin"
    }
    source_jar {
      is_external: false
      is_new_external_version: true
      is_source: false
      relative_path: "src/main/java/hello/app-src.jar"
      root_execution_path_fragment: "bazel-out/darwin-fastbuild/bin"
    }
    source_jars {
      is_external: false
      is_new_external_version: true
      is_source: false
      relative_path: "src/main/java/hello/app-src.jar"
      root_execution_path_fragment: "bazel-out/darwin-fastbuild/bin"
    }
  }
  main_class: "hello.Application"
}
key {
  label: "//src/main/java/hello:app"
}
kind_string: "java_binary"

This file contains all language-specific information IntelliJ needs to know about the target, which can be used to integrate with language plugins directly. It enables features such as semantic code browsing, autocomplete, refactoring, reference finding and go-to-definition.

        └── project.view.dat

Finally, this is the serialized form of the Bazel project view. This prevents the need for parsing the .bazelproject project view file every time we open the project.

Connecting the dots with the logs

Now that we understand what the generated files are and what they're for, we can explore the sync process' timeline through logs. Here's a simplified form of what you'd see in the Bazel Console during a sync:

Syncing project: Sync (incremental)...

< ... >

Building Bazel targets...
Command: bazel build \
  --aspects=@intellij_aspect//:intellij_info_bundled.bzl%intellij_info_aspect \
  --output_groups=intellij-info-generic,intellij-info-java,intellij-resolve-java \
  <other flags> \
  //src/main/java/hello:app

Here, the plugin invokes Bazel to apply the plugin aspect over the transitive closure of the specified Bazel targets in the project view file.

The output groups determine the set of requested files. In this Java project, the plugin requests for the intellij-info-java and intellij-resolve-java output groups by default, which instructs Bazel to run the actions that produces the respective intellij-info.txt and JAR files for the project view's targets. This builds up the TargetData and BlazeProjectData structures, which the plugin serializes onto the disk for persistence.

After the build completes, the plugin processes the earlier outputs to generate the internal project model, writes the necessary IDE metadata files, and commits the project structure:

Loading: 0 packages loaded
Analyzing: 4 targets (2 packages loaded, 123 targets configured)

<...>

INFO: Build completed successfully, 49 total actions

<...>

Parsing build outputs...
Total rules: 48, new/changed: 42, removed: 0
Reading IDE info result...

Updating target map
Loaded 42 aspect files, total size 77kB
Target map size: 48

<...>

Reading package manifests...

Updating Jar Cache...
Prefetching files...
Refreshing files
Computing directory structure...
Committing project structure...
Workspace has 12 libraries
Workspace has 2 modules
Updating in-memory state...

Sync finished

Language-specific sync processes

After the plugin processes TargetData into a TargetMap of configured targets and aspects, it notifies language sync plugins to do further language-specific processing. See a list of BlazeSyncPlugins here.

For example, the Java sync plugin further processes generated jdeps files from the intellij-resolve-java output group for dependency analysis, and also sets up the right JDK and source roots for the IntelliJ project structure.

When all sync plugins complete, the main sync process finishes and the project is ready for development in IntelliJ.