Turn off sbt delta lake tests in CI #903
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@pengyu-hou / @caiocamatta-stripe / @mickjermsurawong-openai - could either of you folks review? |
pengyu-hou
approved these changes
Jan 16, 2025
There was a problem hiding this comment.
Thanks. Once we have the bazel PR merged, we should have spark 3.5. Let's revisit this later.
pengyu-hou
reviewed
Jan 16, 2025
| @@ -177,9 +147,6 @@ workflows: | |||
| - "Scala 13 -- Tests": | |||
| requires: | |||
| - "Pull Docker Image" | |||
| - "Scala 13 -- Delta Lake Format Tests": | |||
There was a problem hiding this comment.
On a second thought, is it possible to make it optional? So it won't be a blocker to merge PRs.
There was a problem hiding this comment.
let me check if circle ci supports this - if possible I agree, it would be nice
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Turn off delta lake tests in CI - We are seeing these tests flake out fairly often and they result in holding up the rest of our dev loops. We are exercising testing of the table utils path already (the Hive flow). We can explore enabling these on Spark 3.5 and when we're running tests on Bazel there. Isolation and Spark 3.5 are the only differences with our internal fork's setup that come to my mind and these tests have not been flaky on our end.
Why / Goal
Test Plan
Checklist
Reviewers