Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ScalarValue::try_from_array holds onto input array for Struct, Lists types #15205

Open
Dandandan opened this issue Mar 13, 2025 · 0 comments
Open
Labels
bug Something isn't working performance Make DataFusion faster

Comments

@Dandandan
Copy link
Contributor

Dandandan commented Mar 13, 2025

Describe the bug

ScalarValue::try_from_array are used in areas such as accumulators.
Currently arrowslice is used for Struct/Union/List types, causing accumulators holding onto the original arrays (rather than a single value), and thus higher memory usage.

To Reproduce

Create an (struct / list type) array of size n and use ScalarValue::try_from_array to convert it into a ScalarValue.

Expected behavior

Value is copied into a new Array or there would be needing efficient way of representing the scalar value.
Memory usage should be corresponding to a single value rather than the full input array.

Additional context

No response

@Dandandan Dandandan added the bug Something isn't working label Mar 13, 2025
@Dandandan Dandandan changed the title ScalarValue::try_from_array holds onto input array for Struct, Union, Lists types ScalarValue::try_from_array holds onto input array for Struct, Lists types Mar 13, 2025
@alamb alamb added the performance Make DataFusion faster label Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working performance Make DataFusion faster
Projects
None yet
Development

No branches or pull requests

2 participants