[Perf] Evaluate API: Support parallelized evaluator batch run through pf.run #3380
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
promptflow SDK CLI Azure E2E Test Result users/ninhu/perf_improve 4 files 4 suites 4m 18s ⏱️ Results for commit 5ddefa3. ♻️ This comment has been updated with latest results. |
SDK CLI Global Config Test Result users/ninhu/perf_improve6 tests 6 ✅ 1m 24s ⏱️ Results for commit 5ddefa3. ♻️ This comment has been updated with latest results. |
Executor Unit Test Result users/ninhu/perf_improve796 tests 796 ✅ 3m 48s ⏱️ Results for commit 5ddefa3. ♻️ This comment has been updated with latest results. |
Executor E2E Test Result users/ninhu/perf_improve246 tests 238 ✅ 5m 4s ⏱️ For more details on these failures, see this check. Results for commit 5ddefa3. ♻️ This comment has been updated with latest results. |
SDK CLI Test Result users/ninhu/perf_improve 4 files 4 suites 1h 7m 21s ⏱️ Results for commit 5ddefa3. ♻️ This comment has been updated with latest results. |
ninghu
changed the title
ninghu
changed the title
nick863
reviewed
Jun 11, 2024
src/promptflow-evals/promptflow/evals/evaluate/_code_client/proxy_client.py
Outdated
Show resolved
Hide resolved
src/promptflow-evals/promptflow/evals/evaluate/_code_client/proxy_client.py
Outdated
Show resolved
Hide resolved
nick863
previously approved these changes
Jun 11, 2024
singankit
reviewed
Jun 11, 2024
src/promptflow-devkit/promptflow/_sdk/_utilities/general_utils.py
Outdated
Show resolved
Hide resolved
thy09
approved these changes
Jun 15, 2024
wangchao1230
approved these changes
Jun 17, 2024
ninghu
added a commit
that referenced
this pull request
Jun 18, 2024
… runs instead of processes. (#3412) # Description This change is related to #3380 # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Clement Wang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The change in this PR addresses the performance issues we were seeing with the Evaluate API. This is the first step in optimizing performance. The improvements include:
Improvements from this change:
Windows OS, remote tracking disabled
4 evaluators, 100 rows:
1 evaluator, 1 row:
Investigation details can be found here
All Promptflow Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines