Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bazel: numerous fixes, get nearly all tests runnable #29188

Merged
merged 9 commits into from
Aug 28, 2024

Conversation

ParkMyCar
Copy link
Member

@ParkMyCar ParkMyCar commented Aug 22, 2024

This PR makes a number of tweaks to our Bazel setup to get nearly all Rust tests runnable with Bazel. There is a second PR stacked on top of this one that gets all tests working.

All of the changes are logically grouped by commit, they're all relatively small and shouldn't require thorough review, they are:

  1. Update cargo-gazelle to not generate rust_library targets for crates that aren't a library, e.g. catalog-debug or clusterd.
  2. Fix integration tests by updating cargo-gazelle to specify the crate_name attribute. This specifies the name that should be used for the generated binary so it matches what Cargo does.
  3. Fix tests that use the insta crate for snapshotting by specifying the data that they need and an env var so insta can find the snapshots.
  4. Upgrades our version of rules_rust to the latest.
    1. This isn't strictly needed but something that I thought was nice to do
  5. Updates our .bazelrc to specify some Mac only options so doc tests work, also ideally it should improve performance of the Bazel Sandbox.
  6. Updates our usage of the cxx crate with protobuf-native to fix an "ambiguous library" linker error. We were previously depending on two different versions of cxx.
  7. Fix tests that use an external instance of Postgres by specifying COCKROACH_URL as an environment variable that gets passed through to tests.

Motivation

Progress towards https://github.com/MaterializeInc/database-issues/issues/7923

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

@ParkMyCar ParkMyCar force-pushed the bazel/testing branch 3 times, most recently from a6522a1 to 7c9de99 Compare August 22, 2024 23:02
* changes cargo-gazelle to skip generating rust_library targets for crates that don't have one
* update cargo-gazelle to specify the crate_name for integration tests. This
  fixes some crates like `insta` that rely on the test name for path generation.
Fix tests that use `insta` for snapshotting

* Update a few crates Cargo.toml so their generated BUILD.bazel files contain the right
  environment variables
* upgrade the version of rules_rust we use to get on the latest
* run bin/bazel gen
* specify --experimental_inprocess_symlink_creation for macOS which is required for Rust Doc tests
* specify --experimental_inmemory_sandbox_stashes for macOS to improve sandbox perf

add some new settings to .bazelrc for sandbox perf
* with cargo-gazelle, specify the same set of `crate_features` for lib tests as the library target
* refine how we use cxx and thus link the protobuf-native crate. Fixes an "ambiguous library"
  error we were getting when linking.
* Define COCKROACH_URL as an environment variable that gets passed to the test runner
@ParkMyCar ParkMyCar changed the title bazel: Various fixes, add Bazel-ified Cargo Test to CI bazel: numerous fixes, get nearly all tests runnable Aug 28, 2024
@ParkMyCar ParkMyCar marked this pull request as ready for review August 28, 2024 14:36
Comment on lines +21 to +27
const PROTO_DIRECTORY: &str = {
if mz_build_tools::is_bazel_build() {
"src/catalog/protos"
} else {
"protos"
}
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not great, but possible to live with I think!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah FWIW this is my least favorite part, I'll follow up with this!

@ParkMyCar ParkMyCar merged commit 3fbc271 into MaterializeInc:main Aug 28, 2024
82 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Aug 28, 2024
@ParkMyCar ParkMyCar deleted the bazel/testing branch January 13, 2025 00:41
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants