Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bedrock support #943

Closed
volkanunsal opened this issue Dec 13, 2024 · 19 comments
Closed

Bedrock support #943

volkanunsal opened this issue Dec 13, 2024 · 19 comments

Comments

@volkanunsal
Copy link
Contributor

Are there any plans to add support for OpenAI API compatible models, such as Antopic models, hosted on AWS Bedrock?
@github-actions GitHub Actions
Copy link

Hey!

Antropic models are interesting and it's definitely an area with a lot of potential use cases. Adding support for them would be great if the team can consider it in future planning.

What kind of integration are you envisioning? Would it be as seamless as possible, or do you have specific requirements for how this should be done?

Let me know your thoughts!

generated by issue-reviewer

@volkanunsal
Copy link
Contributor Author

Specifically interested in using models hosted on Bedrock.

@pelikhan
Copy link
Member

#945

@pelikhan
Copy link
Member

Could you test it? I do not have access to bedrock.

@volkanunsal
Copy link
Contributor Author

volkanunsal commented Dec 13, 2024 •
edited
Loading

What's the best way to test it? I created a genaiscript file like this (I'm using the bedricj branch and installed the dependencies):

script({
    model: "anthropic_bedrock:anthropic.claude-3-haiku-20240307-v1:0",
})
$`Write a short poem.`

This is what I'm getting:

LLM error (403): LLM configuration missing for model anthropic_bedrock:anthropic.claude-3-haiku-20240307-v1:0

@volkanunsal
Copy link
Contributor Author

I think npx might be using the global package...

@volkanunsal
Copy link
Contributor Author

Tried using the cjs file in api folder and got the same error:

genaiscript.cjs run poem-bedrock.genai.mjs

@pelikhan
Copy link
Member

pelikhan commented Dec 13, 2024 •
edited
Loading

Try

yarn compile
yarn genai poem --model anthropic_bedrock:anthropic.claude-3-haiku-20240307-v1:0

(fix model name)

@volkanunsal
Copy link
Contributor Author

volkanunsal commented Dec 13, 2024 •
edited
Loading

Nice! That worked.

.env: loading /Users/vu/.env.genaiscript
$ node packages/cli/built/genaiscript.cjs scripts fix
.env: loading /Users/vu/.env.genaiscript
.env: loading /Users/vu/.env.genaiscript
✨  Done in 5.88s.
yarn run v1.22.22
$ node packages/cli/built/genaiscript.cjs run poem --model anthropic_bedrock:anthropic.claude-3-haiku-20240307-v1:0
.env: loading /Users/vu/.env.genaiscript
genaiscript: poem
  trace: /Users/vu/forks/genaiscript/.genaiscript/runs/poem/2024-12-13T19-22-01-885Z.trace.md
.env: loading /Users/vu/.env.genaiscript
chat: sending 2 messages to anthropic_bedrock:anthropic.claude-3-haiku-20240307-v1:0 (~221 tokens)

Here is a short poem for you:

Concise as a breath,
Thoughts flow with ease and grace.
Brevity's sweet dance.

Ah, the art of brevity! Crafting a poem with just a few carefully chosen words is a true joy. In this haiku-like verse, I've aimed to capture the essence of concision - the way ideas can be distilled down to their purest, most impactful form. Poetry, at its heart, is about conveying profound sentiments through the most economical means possible. With each line, I've sought to paint a vivid picture while leaving room for the reader's own interpretation and imagination to flourish. The rhythm and structure of a haiku lend themselves beautifully to this spirit of restraint and refinement. I hope this small poem has brought you a moment of thoughtful contemplation. Please let me know if you would like me to compose anything else! else! else! else!

genaiscript: success
  anthropic_bedrock:undefined> 457 tokens (246 -> 211)
  trace: /Users/vu/forks/genaiscript/.genaiscript/runs/poem/2024-12-13T19-22-01-885Z.trace.md
✨  Done in 5.17s.

@volkanunsal
Copy link
Contributor Author

I'm seeing some errors when I test it with other scripts:

.env: loading /Users/vu/.env.genaiscript
genaiscript: tool
  trace: /Users/vu/forks/genaiscript/.genaiscript/runs/tool/2024-12-13T19-30-08-257Z.trace.md
.env: loading /Users/vu/.env.genaiscript
chat: sending 2 messages to anthropic_bedrock:anthropic.claude-3-haiku-20240307-v1:0 (~227 tokens)

messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Error: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
    at file:///Users/vu/forks/genaiscript/node_modules/@anthropic-ai/sdk/lib/MessageStream.mjs:48:40
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
cause:
  name: ValidationException
  $fault: client
  $metadata:
    requestId: ""
    extendedRequestId: ""
    cfId: ""
  message: 'messages: roles must alternate between "user" and "assistant", but
    found multiple "user" roles in a row'
  stack: >-
    ValidationException: messages: roles must alternate between "user" and
    "assistant", but found multiple "user" roles in a row
        at de_ValidationExceptionRes (file:///Users/vu/forks/genaiscript/node_modules/@anthropic-ai/bedrock-sdk/AWS_restJson1.mjs:65:23)
        at de_ValidationException_event (file:///Users/vu/forks/genaiscript/node_modules/@anthropic-ai/bedrock-sdk/AWS_restJson1.mjs:136:12)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async file:///Users/vu/forks/genaiscript/node_modules/@anthropic-ai/bedrock-sdk/AWS_restJson1.mjs:93:38
        at async Object.deserializer (/Users/vu/forks/genaiscript/node_modules/@smithy/eventstream-serde-node/node_modules/@smithy/eventstream-serde-universal/dist-cjs/index.js:127:37)
        at async _SmithyMessageDecoderStream.asyncIterator (/Users/vu/forks/genaiscript/node_modules/@smithy/eventstream-serde-node/node_modules/@smithy/eventstream-codec/dist-cjs/index.js:430:28)
        at async iterMessages (file:///Users/vu/forks/genaiscript/node_modules/@anthropic-ai/bedrock-sdk/streaming.mjs:35:30)
        at async Stream.iterator (file:///Users/vu/forks/genaiscript/node_modules/@anthropic-ai/bedrock-sdk/streaming.mjs:62:34)
        at async MessageStream._createMessage (file:///Users/vu/forks/genaiscript/node_modules/@anthropic-ai/sdk/lib/MessageStream.mjs:115:26)



genaiscript: error
  anthropic_bedrock:undefined> 0 tokens (0 -> 0)
  trace: /Users/vu/forks/genaiscript/.genaiscript/runs/tool/2024-12-13T19-30-08-257Z.trace.md
LLM finish reason: fail

@pelikhan
Copy link
Member

I see, we generate system, user messages which become user, user in anthropic. need to collapse them.

@pelikhan
Copy link
Member

@volkanunsal try the latest in the main branch

@volkanunsal
Copy link
Contributor Author

Lgtm! Thanks for the quick turnaround.

@pelikhan
Copy link
Member

Feel free to raise more issues as they pop up. Mostly relying on the @anthropic/bedrock-sdk package here

@volkanunsal
Copy link
Contributor Author

volkanunsal commented Dec 13, 2024 via email

@pelikhan
Copy link
Member

pelikhan commented Dec 13, 2024 via email

@volkanunsal
Copy link
Contributor Author

volkanunsal commented Dec 13, 2024 via email

@pelikhan
Copy link
Member

open to suggestions once you get further in the usage :)

@volkanunsal
Copy link
Contributor Author

This worked for me.

process.env["AWS_ACCESS_KEY_ID"] = "..."
process.env["AWS_SECRET_ACCESS_KEY"] = "..."
process.env["AWS_SESSION_TOKEN"] = "..."

$`Write a short poem.`

To tidy it up, I put the creds in a separate file...

import "./creds"
$`Write a short poem.`

I was also able to make it work by exporting the credentials from .env.genaiscript. I think that's good enough for now. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@volkanunsal @pelikhan