Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow Bazel project to load Rust crates selectively #335

Open
lolski opened this issue Dec 10, 2021 · 1 comment
Open

Allow Bazel project to load Rust crates selectively #335

lolski opened this issue Dec 10, 2021 · 1 comment
Assignees
@alexjpwalker
Labels
priority: high type: feature
Milestone
TypeDB: Rust Rewrite

Comments

@lolski
Copy link
Member

lolski commented Dec 10, 2021 •
edited
Loading

Currently, a Bazel project loads the crates by using the raze_fetch_remote_crates() macro which will load all declared crates without being able to select the ones that are actually needed.

That isn't ideal and we should create a more sophisticated tool for loading the crates selectively:

crates(
  "antlr-rust",
  "rust-rocksdb"
)
@lolski lolski added this to the TypeDB Rust Rewrite milestone Dec 10, 2021
@lolski lolski changed the title Allow Rust repositories to select crates they want to load Allow loading Rust crate selectively Dec 13, 2021
@lolski lolski changed the title Allow loading Rust crate selectively Allow Bazel projeect to load Rust crates selectively Dec 13, 2021
@lolski
Copy link
Member Author

lolski commented Dec 13, 2021

According to Max, it is not possible to provide a list of crates that you want to load in the raze_fetch_remote_crates() macro. However, let's still double-check if that's indeed the case.

@lolski lolski changed the title Allow Bazel projeect to load Rust crates selectively Allow Bazel project to load Rust crates selectively Dec 13, 2021
lolski pushed a commit to lolski/dependencies that referenced this issue Oct 7, 2022
@sugarmanz
Better support for multiplatform Maven publications w/ local JARs (ty…
8eb8a0e 
…pedb#336)

## What is the goal of this PR?

Support bundling local `java_import` deps that may or may not conflict when bundled (typedb#335).

The use case here is around bundling JARs for different platforms and potentially having a multiplatform JAR as well. 

## What are the changes implemented in this PR?

1. Support for bundling `java_import` targets

Add a bit more graceful handling for dependencies that don't have a `maven_coordinates` tag. Originally, the aggregation would attempt to just pull directly from `target[OutputGroupInfo].compilation_outputs` without checking to see if it actually existed, resulting in the error from typedb#335. Now, that logic is guarded in checking that `compilation_outputs` is truthy. Additionally, the source JAR is optional in `java_import`, so I added some safety against adding the source JAR when it doesn't exist.

Just that alone didn't fix `java_import` targets though, as `java_import` doesn't treat the input JAR as `compilation_outputs`. Locally, I had wrapped the `java_import` rule to copy over `target[JavaInfo].runtime_output_jars` into `target[OutputGroupInfo].compilation_outputs`, but requiring consumers to have to do that isn't great. So, I added another branch to the aggregation method, that checks for `target[JavaInfo].runtime_output_jars` and will create a collection of dependencies if truthy. Because `java_import` can accept multiple JARs, we need to ensure that a dependency struct for each will be created, potentially mapping to a source jar, if also defined for the index.

There are technically two different fields from `JavaInfo` that would've worked for `java_import`:
[runtime_output_jars](https://bazel.build/rules/lib/JavaInfo?hl=en#runtime_output_jars)
[full_compile_jars](https://bazel.build/rules/lib/JavaInfo?hl=en#full_compile_jars)

I used `runtime_output_jars` only because that made a _little_ more sense to me? To bundle the runtime JAR vs the compile JAR. I'm not really sure, but there is no difference between the two for `java_import`s. Open to input here.

<details>
<summary>Provider info for a `java_import` target</summary>

```
    "JavaInfo": struct(
        annotation_processing = None,
        api_generating_plugins = <unknown object com.google.devtools.build.lib.rules.java.AutoValue_JavaPluginInfo_JavaPluginData>,
        compilation_info = <unknown object com.google.devtools.build.lib.rules.java.JavaCompilationInfoProvider>,
        compile_jars = depset([
            <generated file j2v8/libs/_ijar/j2v8_empty/j2v8/libs/j2v8_empty-6.1.0-ijar.jar>,
            <generated file j2v8/libs/_ijar/j2v8_empty/j2v8/libs/j2v8_macos_x86_64-6.1.0-ijar.jar>
        ], order = "preorder"),
        full_compile_jars = depset([
            <source file j2v8/libs/j2v8_empty-6.1.0.jar>,
            <source file j2v8/libs/j2v8_macos_x86_64-6.1.0.jar>
        ], order = "preorder"),
        java_outputs = [
            <unknown object com.google.devtools.build.lib.rules.java.AutoValue_JavaRuleOutputJarsProvider_JavaOutput>,
            <unknown object com.google.devtools.build.lib.rules.java.AutoValue_JavaRuleOutputJarsProvider_JavaOutput>
        ],
        outputs = <unknown object com.google.devtools.build.lib.rules.java.JavaRuleOutputJarsProvider>,
        plugins = <unknown object com.google.devtools.build.lib.rules.java.AutoValue_JavaPluginInfo_JavaPluginData>,
        runtime_output_jars = [
            <source file j2v8/libs/j2v8_empty-6.1.0.jar>,
            <source file j2v8/libs/j2v8_macos_x86_64-6.1.0.jar>
        ],
        source_jars = [<source file j2v8/libs/srcs.jar>],
        transitive_compile_time_jars = depset([
            <generated file j2v8/libs/_ijar/j2v8_empty/j2v8/libs/j2v8_empty-6.1.0-ijar.jar>,
            <generated file j2v8/libs/_ijar/j2v8_empty/j2v8/libs/j2v8_macos_x86_64-6.1.0-ijar.jar>
        ], order = "preorder"),
        transitive_deps = depset([
            <generated file j2v8/libs/_ijar/j2v8_empty/j2v8/libs/j2v8_empty-6.1.0-ijar.jar>,
            <generated file j2v8/libs/_ijar/j2v8_empty/j2v8/libs/j2v8_macos_x86_64-6.1.0-ijar.jar>
        ], order = "preorder"),
        transitive_native_libraries = depset([], order = "topological"),
        transitive_runtime_deps = depset([
            <source file j2v8/libs/j2v8_empty-6.1.0.jar>,
            <source file j2v8/libs/j2v8_macos_x86_64-6.1.0.jar>
        ], order = "preorder"),
        transitive_runtime_jars = depset([
            <source file j2v8/libs/j2v8_empty-6.1.0.jar>,
            <source file j2v8/libs/j2v8_macos_x86_64-6.1.0.jar>
        ], order = "preorder"),
        transitive_source_jars = depset([<source file j2v8/libs/srcs.jar>])
    ),
    "OutputGroupInfo": struct(
        _direct_source_jars = depset([<source file j2v8/libs/srcs.jar>]),
        _hidden_top_level_INTERNAL_ = depset([<source file j2v8/libs/j2v8_empty-6.1.0.jar>, <source file j2v8/libs/j2v8_macos_x86_64-6.1.0.jar>]),
        _source_jars = depset([<source file j2v8/libs/srcs.jar>]),
        compilation_outputs = depset([])
    )
```
</details>

Finally, the last thing required for local JARs was ensuring that multiple targets aren't bundling the local JAR multiple times. The strategy here was to filter out transitive JAR dependencies from dependencies that have Maven coordinates, since the local JAR would have already been bundled within that dependency and can be pulled in from a Maven repository.

https://github.com/sugarmanz/bazel-distribution/blob/412162f901f6650e67e014156945ee06c6619b1b/maven/rules.bzl#L223-L229

2. ~~Add flag for allowing conflicts during JAR assembly~~

~~In the use case above, I mentioned publishing platform specific JARs, of which each bundle their own copy of a platform specific local JAR that includes the classes and platform binary for that lib. I also need to publish an "uber" JAR that contains all the binaries for each platform. When attempting to assemble the JAR for this, the JAR assembler would throw an error for duplicate entries. This makes sense for the primary use case, but in this case, I know that the classes are the same and would like to ignore the duplicates, as is currently done when I build my project with Gradle. Thus, I added a flag to not throw when duplicates are detected. This defaults to `True`, ensuring the same behavior unless specifically overridden by the consumer.~~

Shelved because this issue was inherently fixed with:
https://github.com/sugarmanz/bazel-distribution/blob/412162f901f6650e67e014156945ee06c6619b1b/maven/rules.bzl#L223-L229

3. Scope generated `deploy.py`

This essentially enables the `deploy_maven` rule to be used multiple times in the same build file, which is desirable for my mulitplatform project, as I want to have all my deploy targets for that module in the same build file.

Closes typedb#335
@alexjpwalker alexjpwalker mentioned this issue Oct 10, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexjpwalker alexjpwalker

Labels
priority: high type: feature
Projects
None yet
Milestone
TypeDB: Rust Rewrite
Development

No branches or pull requests
2 participants
@lolski @alexjpwalker