Skip decoding of pages marked as pruned in PQ reader #18347
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
github-actions
bot
added
libcudf
mhaseeb123
added
feature request
|
Here are the four Parquet test files used in this PR. |
| @@ -641,6 +666,20 @@ CUDF_KERNEL void __launch_bounds__(decode_block_size) | |||
|
|
|||
| bool const has_repetition = s->col.max_level[level_type::REPETITION] > 0; | |||
|
|
|||
| // Exit early if the page is invalid | |||
| // MH: What to do if `has_repetition` is true? | |||
There was a problem hiding this comment.
Hi @nvdbaranec,
Do you think we need to any special handling for lists (has_repetiion == true) in this kernel. I only see it being used to set the dst address. Same question for a couple other kernels.
| @@ -236,6 +249,16 @@ CUDF_KERNEL void __launch_bounds__(decode_block_size) | |||
| int page_idx = blockIdx.x; | |||
| int t = threadIdx.x; | |||
| int out_thread0; | |||
|
|
|||
| // Exit early if the page is invalid | |||
| // MH: How to handle all types in this decoder? Also what to do if `has_repetition` is true? | |||
There was a problem hiding this comment.
A couple questions about this one:
- Same question as below about lists.
- Do we need any special handling for any other types as well?
- When is this generic decoder called?
| @@ -473,6 +483,20 @@ CUDF_KERNEL void __launch_bounds__(decode_block_size) | |||
|
|
|||
| bool const has_repetition = s->col.max_level[level_type::REPETITION] > 0; | |||
|
|
|||
| // Exit early if the page is invalid | |||
| // MH: What to do if `has_repetition` is true? | |||
There was a problem hiding this comment.
Same question about lists here
Description
This PR implements the necessary mechanism to skip decoding of data pages marked as pruned in the Parquet reader.
Closes #18316
Part of #17896
Checklist