Skip to content

[CI][Benchmarks] add contributing guide #17995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: sycl
Choose a base branch
from

Conversation

pbalcer
Copy link
Contributor

@pbalcer pbalcer commented Apr 14, 2025

No description provided.

@pbalcer pbalcer requested a review from a team as a code owner April 14, 2025 08:54
@pbalcer
Copy link
Contributor Author

pbalcer commented Apr 14, 2025

There are a few things I noticed when writing this that I think we should fix:


* `passed`: Boolean indicating if verification passed (default: `True`).

We should just throw exceptions when a benchmark fails, not produce an invalid result.


* `name`: Set to `label` by the framework.

The name field in Result is inconsistently used throughout the framework. We should clean this up and always use label where appropriate.


* **Use unique names:** Ensure `benchmark.name()` and `result.label` are descriptive and unique.

What I was contemplating writing is that benchmark names should ideally never change because they are unique keys the visualization scripts use to collect benchmark data into series.
But this is a very annoying limitation. We should introduce a (maybe autogenerated?) key to groups/benchmarks that is unique and not used for any sort of visualization (or maybe treat the existing name/label as a key, and add a new "display name"?)

@PatKamin
Copy link
Contributor

What I was contemplating writing is that benchmark names should ideally never change because they are unique keys the visualization scripts use to collect benchmark data into series. But this is a very annoying limitation. We should introduce a (maybe autogenerated?) key to groups/benchmarks that is unique and not used for any sort of visualization (or maybe treat the existing name/label as a key, and add a new "display name"?)

I think that new "display_name" and "display_label" variables would work best in this situation. They could by default take the current value of name/label and be overridden, if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants