Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Support Gemini via Vertex AI #2030

Merged
merged 7 commits into from
Oct 22, 2024
Merged

feat: Support Gemini via Vertex AI #2030

merged 7 commits into from
Oct 22, 2024

Conversation

dividedmind
Copy link
Collaborator

Note to access the API you have to create a service account at https://console.cloud.google.com/apis/credentials — after creation you'll get a JSON file with credentials. This JSON has to be put into GOOGLE_WEB_CREDENTIALS environment variable.

Example configuration:

GOOGLE_WEB_CREDENTIALS='{   "type": "service_account",   "project_id": "gen-lang-client-0551265112",   "private_key_id": "1fd731b64e9543d54e2921d7cfbe40c5f6f2845a",   "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",   "client_email": "divide-s-navie-experiments@gen-lang-client-0551265112.iam.gserviceaccount.com",   "client_id": "112105442388554579010",   "auth_uri": "https://accounts.google.com/o/oauth2/auth",   "token_uri": "https://oauth2.googleapis.com/token",   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/divide-s-navie-experiments%40gen-lang-client-0551265112.iam.gserviceaccount.com",   "universe_domain": "googleapis.com" }'
APPMAP_NAVIE_MODEL=gemini-1.5-pro-002

@dividedmind dividedmind self-assigned this Oct 3, 2024
@kgilpin kgilpin marked this pull request as draft October 3, 2024 12:16
kgilpin
kgilpin reviewed Oct 3, 2024
View reviewed changes
Copy link
Contributor

@kgilpin kgilpin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! There are some build errors so I flipped this to Draft, temporarily.

@dustinbyrne dustinbyrne marked this pull request as ready for review October 4, 2024 14:10
@kgilpin
Copy link
Contributor

kgilpin commented Oct 6, 2024

Tests still failing.

@dividedmind dividedmind force-pushed the feat/support-vertex-ai branch from f8a16c3 to aa92192 Compare October 13, 2024 11:09
@dividedmind
Copy link
Collaborator Author

I forked the library that was ESM-only and added a CommonJS build (kriasoft/web-auth-library#32) — hopefully this will make it work without fuss.

@kgilpin
Copy link
Contributor

kgilpin commented Oct 13, 2024

Thanks @dividedmind

Do you have thoughts on the error we were seeing in the benchmark? Would really like to get a clean run though with Gemini.

@dividedmind dividedmind force-pushed the feat/support-vertex-ai branch from aa92192 to a31dfaa Compare October 13, 2024 11:38
@dividedmind
Copy link
Collaborator Author

Thanks @dividedmind

Do you have thoughts on the error we were seeing in the benchmark? Would really like to get a clean run though with Gemini.

You mean the recitation problem? I'm looking at it next.

@dividedmind dividedmind force-pushed the feat/support-vertex-ai branch 2 times, most recently from 99e51f5 to f798f3e Compare October 18, 2024 15:08
dustinbyrne
dustinbyrne approved these changes Oct 18, 2024
View reviewed changes
packages/navie/src/services/google-vertexai-completion-service.ts
Comment on lines +112 to +119
if (attempt < maxAttempts - 1 && tokens.length === 0) {
const nextAttempt = CompletionRetryDelay * 2 ** attempt;
warn(`Received ${JSON.stringify(cause)}, retrying in ${nextAttempt}ms`);
await new Promise<void>((resolve) => {
setTimeout(resolve, nextAttempt);
});
continue;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this doesn't attempt to prune input tokens

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I figured it's unlikely to be necessary with 2M token input limit...

packages/navie/src/services/vector-terms-service.ts
Comment on lines +98 to +100
context: z.string(),
instructions: z.string(),
terms: z.array(z.string()),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description of these properties could move out of the prompt and in to the schema

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, although I tried to change it as little as possible as not to break anything.

@dividedmind dividedmind force-pushed the feat/support-vertex-ai branch from f798f3e to f1e6372 Compare October 21, 2024 18:47
@dividedmind dividedmind merged commit f37e4a5 into main Oct 22, 2024
21 of 23 checks passed
@dividedmind dividedmind deleted the feat/support-vertex-ai branch October 22, 2024 12:17
@appland-release
Copy link
Contributor

🎉 This PR is included in version @appland/navie-v1.34.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dustinbyrne dustinbyrne dustinbyrne approved these changes

@kgilpin kgilpin Awaiting requested review from kgilpin

Assignees

@dividedmind dividedmind

Labels
released
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dividedmind @kgilpin @appland-release @dustinbyrne