GH-45185: [C++][Parquet] Raise an error for invalid repetition levels when delimiting records #45186
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
Component: Parquet
Component: C++
awaiting review
|
Note for others taking a look, tests won't be successful until we merge the parquet-testing PR |
adamreeve
force-pushed
the
validate-repetition
branch
from
be347da to
22794e2
Compare
|
The failing tests all look to be caused by #45305 rather than this change. |
mapleFU
reviewed
Jan 20, 2025
github-actions
bot
added
awaiting committer review
mapleFU
reviewed
Jan 21, 2025
| TEST(TestArrowReaderAdHoc, InvalidRepetitionLevels) { | ||
| // GH-45185 - Repetition levels start with 1 instead of 0 | ||
| auto path = test::get_data_file("ARROW-GH-45185.parquet", /*is_good=*/false); | ||
| TryReadDataFile(path, ::arrow::StatusCode::IOError); |
There was a problem hiding this comment.
Would you mind also check the status is "The repetition level at the start of a record must be 0 but got ..."
mapleFU
reviewed
Jan 21, 2025
cpp/src/parquet/column_reader.cc
Outdated
| @@ -1611,7 +1611,12 @@ class TypedRecordReader : public TypedColumnReaderImpl<DType>, | |||
| // another record start or exhausting the ColumnChunk | |||
| int64_t level = levels_position_; | |||
| if (at_record_start_) { | |||
| ARROW_DCHECK_EQ(0, rep_levels[levels_position_]); | |||
| if (rep_levels[levels_position_] != 0) { | |||
There was a problem hiding this comment.
Can we use ARROW_PREDICT_FALSE to check it?
mapleFU
approved these changes
Jan 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
See #45185. Invalid repetition levels would previously only cause a fatal error in debug builds.
What changes are included in this PR?
Converts an existing
ARROW_DCHECK_EQof the repetition level with a check that will raise an exception in release builds too.Are these changes tested?
Yes, using a new example file (apache/parquet-testing#67)
Are there any user-facing changes?
Yes, reading columns with invalid repetition levels as Arrow arrays will now raise an exception.