Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🚧 Compute and use the str_offset when reading nested large string cols with chunked Parquet reader. #17702

Draft
wants to merge 7 commits into
base: branch-25.02
Choose a base branch
from
Draft

🚧 Compute and use the str_offset when reading nested large string cols with chunked Parquet reader. #17702

wants to merge 7 commits into from

Conversation

mhaseeb123
Copy link
Member

@mhaseeb123 mhaseeb123 commented Jan 9, 2025 •
edited
Loading

Description

Note: Under 🚧

This PR enables computing the str_offset required to correctly compute the offsets columns for nested large strings columns with chunked Parquet reader when chunk_read_limit is small resulting in multiple output table chunks per subpass.

Fixes #17692. Blocked until #17693 is merged and changes are incorporated.

Checklist

  • Figure out the root cause (the why) of the problem being fixed.
  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Jan 9, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Jan 9, 2025
@mhaseeb123 mhaseeb123 added 5 - DO NOT MERGE Hold off on merging; see PR for details 2 - In Progress Currently a work in progress bug Something isn't working cuIO cuIO issue non-breaking Non-breaking change labels Jan 9, 2025
@mhaseeb123 mhaseeb123 requested a review from nvdbaranec January 10, 2025 03:07
@mhaseeb123 mhaseeb123 removed the 5 - DO NOT MERGE Hold off on merging; see PR for details label Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nvdbaranec nvdbaranec Awaiting requested review from nvdbaranec

At least 1 approving review is required to merge this pull request.

Assignees

@mhaseeb123 mhaseeb123

Labels
2 - In Progress Currently a work in progress bug Something isn't working cuIO cuIO issue libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
libcudf
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] Chunked parquet reader with small chunk_read_limit does not correctly read nested large string columns.
1 participant
@mhaseeb123