Skip to content
This repository has been archived by the owner on Jun 14, 2024. It is now read-only.

[FEATURE REQUEST]: Create helper function to check whether index is actually used in the plan #492

Open
4 tasks
paryoja opened this issue Sep 2, 2021 · 1 comment
Labels
enhancement New feature or request untriaged This is the default tag for a newly created issue

Comments

@paryoja
Copy link
Contributor

paryoja commented Sep 2, 2021

Feature requested

I need a function to check whether Spark actually used an index I intended to use.
It could be helpful to understand the behavior of hyperspace indexes and optimize the behavior of it.

Acceptance criteria

Describe what you consider (e.g., functionality requirement) has to be true to mark this feature as complete

  • Check whether a LogicalPlan has any use of any hyperspace indexes.
  • Given hyperspace index name, check whether a LogicalPlan has a use of the index.
  • Given hyperspace index name and version value, check whether a LogicalPlan has a use of the index with the version value.

Success criteria

Define any measurements that need to be performed

  • Functionality implemented

Additional context

In the DeltaLakeIntegrationTest.scala, it has isIndexVersionUsed and isIndexUsed for checking the test results. It could be better to expose the functions for the general use.

@paryoja paryoja added enhancement New feature or request untriaged This is the default tag for a newly created issue labels Sep 2, 2021
@sezruby
Copy link
Collaborator

sezruby commented Sep 2, 2021

hs.explain(df) returns used indexes, but I think we need to improve the output result so that it shows the used log version & the number of times each index applied.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request untriaged This is the default tag for a newly created issue
Projects
None yet
Development

No branches or pull requests

2 participants