[AutoBuild] Support model selection when building agents #2413
Conversation
There was a problem hiding this comment.
Thank you very much, @whiskyboy. LGTM!
I've only 2 tiny questions, but it's OK to ignore them.
notebook/autobuild_basic.ipynb
Outdated
| " # You need to provide a short description of your model here.\n", | ||
| " \"profile\": \"A pre-trained model that is good at sovling mathematical reasoning problems.\",\n", | ||
| " }\n", |
There was a problem hiding this comment.
A description alone is not enough. Context length, cost, required GPU resources, and benchmark ranking are also necessary for model selection.
There was a problem hiding this comment.
I agree. But I think a description is a good start point as we can always add these extra infomation into the description field. We can split these fields out once we firgure out how to use these information for model selection. WDYT?
There was a problem hiding this comment.
Yes, but context length and required GPU resources are crucial for successfully solving a task. We should add some restrictions or guidance for users to adopt this feature correctly, like splitting these fields out.
There was a problem hiding this comment.
Yeah, it makes sense to me. How do you prefer to use these crucial informations? Like longer context model has a higher priority, and check the gpu's avaliablity in runtime?
There was a problem hiding this comment.
longer context model has a higher priority, and check the gpu's availability in runtime
LGTM!
There was a problem hiding this comment.
@LinxinS97 Mind reviewing again?
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| - | Generic High Entropy Secret | fa66435 | test/oai/test_utils.py | View secret | |
| - | Generic High Entropy Secret | fa66435 | test/oai/test_utils.py | View secret | |
| - | Generic High Entropy Secret | fa66435 | test/oai/test_utils.py | View secret | |
| - | Generic High Entropy Secret | fa66435 | test/oai/test_utils.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Our GitHub checks need improvements? Share your feedbacks!
|
@whiskyboy Have you joined the discord? Could we have a meeting? I would like to share with you some latest information on autobuild and hear about your next plan. |
@LinxinS97 Sure, would love to. My discord name is whiskyboy0077. BTW, are you working at Microsoft? If yes, can we talk via Teams? |
Why are these changes needed?
In some cases we need to set different agent with different backbone models, such as when we want to group a set of small LLMs with different capacities to complete a complex task.
This PR will allow
AgentBuilder()accept a list of models as parameter toagent_model, and thebuilder_managerwill select the most suitable model for each agent based on the agent/model profile and the task description.Checks