Add multi-language system prompts and BedrockChatAdapter implementation #576

michel-heon · 2024-09-24T14:14:12Z

Pull Request: Centralize and Internationalize System Prompts

This pull request addresses the issue of scattered system prompts across the codebase and the lack of support for internationalization, as described in the corresponding Git issue #571.

Changes:

Commit: 256279db811d17f6c558ccf469bfbec0e0d93583
Objective: Centralize system prompt management and enable internationalization (i18n).
Affected Files:
- system_prompts.py: A new module created to centralize all system prompts and support multiple languages (English and Canadian French).
- base.py: Refactored methods (get_prompt, get_condense_question_prompt, get_qa_prompt) to retrieve prompts from system_prompts.py.
- __init__.py: Updated to import system prompts for simplified access.

Key Improvements:

Centralization: All system prompts are now stored in one place (system_prompts.py), improving manageability and scalability.
Internationalization: The system can now support multiple languages, with English and Canadian French prompts available as an initial setup. This framework allows for future expansion into other languages.
Adapter Compatibility: Adapters such as azure-openai, mistral, claude, titan, and llama are updated to use the new prompt management system, ensuring consistency and reducing code duplication.

Testing Instructions:

Deploy the Application: Follow the deployment instructions provided in the project documentation to deploy the application.
Target Language: The language used by the system prompts is now managed by the mechanism in system_prompts.py.
Modify Language: You can modify the target language at runtime by editing the language selection in the GenAIChatBotStack-LangchainInterfaceReques Lambda function.
Prompt Tracing: The system prompts are logged and can be traced in CloudWatch. Additionally, they are available in the prompt field of the metadata variable in the AWS GenAI Chatbot console for further analysis.

Expected Outcome:

Simplified prompt management across adapters.
Enhanced scalability and flexibility in adding new languages.

- Implement system prompts in English and Canadian French for AI interactions in `system_prompts.py`. - Enhance `BedrockChatAdapter` with prompt templates for QA, conversation, and follow-up questions in `base.py`. - Update `__init__.py` to include system prompt imports for easy access. - Configure logger in `base.py` to trace key operations for QA prompts and conversational prompts.

charles-marion

Thank you for creation this PR! I think it's a good addition to the project (And sorry for the delay)

The build process failed. I recommend to run npm run-vetall

lib/model-interfaces/langchain/functions/request-handler/adapters/bedrock/base.py

lib/model-interfaces/langchain/functions/request-handler/adapters/base/base.py

lib/model-interfaces/langchain/functions/request-handler/adapters/bedrock/base.py

lib/model-interfaces/langchain/functions/request-handler/adapters/base/base.py

...del-interfaces/langchain/functions/request-handler/adapters/shared/prompts/system_prompts.py

charles-marion · 2024-09-25T15:05:24Z

...del-interfaces/langchain/functions/request-handler/adapters/shared/prompts/system_prompts.py

+                     "Si vous ne trouvez pas la réponse dans les documents, informez l'utilisateur que l'information n'est pas disponible. "
+                     "Si possible, dressez la liste des documents référencés.",
+        # Prompt for conversational interaction between a human and AI (French-Canadian)
+        'conversation_prompt': "Vous êtes un assistant IA utilisant la Génération Augmentée par Récupération (RAG). "


This prompt is used when RAG is not used. I think this prompt should be changed.

it is modified according to the specification

@michel-heon Can you clarify what specification you are referring to?

My point is this prompt is a copy/similar to the one above qa_prompt used when a workspace is selected suggesting to use documents and RAG (but this prompt is used when no workspace is set, so no documents)

For reference this is the english version: The following is a friendly conversation between a human and an AI. " "If the AI does not know the answer to a question, it truthfully says it does not know.

It is used here

aws-genai-llm-chatbot/lib/model-interfaces/langchain/functions/request-handler/adapters/base/base.py

Line 217 in cbe2635

chain = self.get_prompt() | self.llm

Sorry, I misunderstood. I've changed the prompt to reflect the comment.

charles-marion · 2024-09-25T15:07:12Z

...del-interfaces/langchain/functions/request-handler/adapters/shared/prompts/system_prompts.py

+        'conversation_prompt': "The following is a friendly conversation between a human and an AI. "
+                               "If the AI does not know the answer to a question, it truthfully says it does not know.",
+        # Prompt for rephrasing a follow-up question to be a standalone question
+        'condense_question_prompt': "Given the conversation inside the tags <conv></conv>, rephrase the follow up question inside <followup></followup> to be a standalone question.",


Maybe we could merge condense_question_prompt and contextualize_q_system_prompt since they have the same goal. (The later is only used by bedrock)

charles-marion · 2024-09-25T15:14:55Z

...del-interfaces/langchain/functions/request-handler/adapters/shared/prompts/system_prompts.py

+    # Add other languages here if needed
+
+# Set default language (English)
+lang = Language.ENGLISH.value  # Default language is set to English


I would recommend adding this as a selection option of the cli and pass it as an env variable

Could be added later and maybe also a documentation page explaining how to add languages.

I'm all for thinking things through, as both solutions offer their own list of advantages and disadvantages. I propose to create a new issue for this feature after this PR is closed.

template updates; - improve prompt system for multilingual support; - expand test coverage for Bedrock adapters with guardrail integration.

charles-marion · 2024-10-15T21:47:10Z

@michel-heon please tag me or click the re-request review button if you'd like me to have a look.
(Note I noticed the file base.py_new which is probably not relevant?)

michel-heon · 2024-10-16T10:13:47Z

@charles-marion In fact, the base.py_new file is an error that I've deleted. And indeed, the code is ready for revision.

charles-marion

Thank you for the update.
Note that I appreciate the help with this change. I can address my comments and complete the PR if you'd like (I don't want to take too much of your time)

charles-marion · 2024-10-16T22:30:31Z

lib/model-interfaces/langchain/functions/request-handler/adapters/base/base.py

-                model=self.model_id,
-                metric_type="token_usage",
-                value=self.callback_handler.usage.get("total_tokens"),
+                extra={


Changing the JSON format here would break the metric in the dashboard. Please undo
https://github.com/aws-samples/aws-genai-llm-chatbot/blob/main/lib/monitoring/index.ts#L289

charles-marion · 2024-10-16T22:31:19Z

lib/model-interfaces/langchain/functions/request-handler/adapters/base/base.py


 class Mode(Enum):
    CHAIN = "chain"


+def get_guardrails() -> dict:


This is only applicable for bedrock. Why did you add it here?

charles-marion · 2024-10-16T22:32:45Z

lib/model-interfaces/langchain/functions/request-handler/adapters/base/base.py

@@ -342,3 +362,245 @@ def run(self, prompt, workspace_id=None, *args, **kwargs):
                return self.run_with_chain(prompt, workspace_id)

        raise ValueError(f"unknown mode {self._mode}")
+
+
+class BedrockChatAdapter(ModelAdapter):


I think this is a copy of https://github.com/aws-samples/aws-genai-llm-chatbot/blob/main/lib/model-interfaces/langchain/functions/request-handler/adapters/bedrock/base.py

It would revert this change in this file.

charles-marion · 2024-10-16T22:33:32Z

lib/model-interfaces/langchain/functions/request-handler/adapters/bedrock/base.py

        return {
            "guardrailIdentifier": os.environ["BEDROCK_GUARDRAILS_ID"],
            "guardrailVersion": os.environ.get("BEDROCK_GUARDRAILS_VERSION", "DRAFT"),
        }
+    logger.info("No guardrails ID found.")


Suggested change

logger.info("No guardrails ID found.")

logger.debug("No guardrails ID found.")

Otherwise it will be logged on every llm call.

charles-marion · 2024-10-16T22:36:51Z

lib/model-interfaces/langchain/functions/request-handler/adapters/bedrock/base.py

+        top_p = model_kwargs.get("topP")
+        max_tokens = model_kwargs.get("maxTokens")
+
+        if temperature:


It would not set the value if temperature is 0

Suggested change

if temperature:

if temperature is not None:

charles-marion · 2024-10-16T22:41:08Z

lib/model-interfaces/langchain/functions/request-handler/adapters/bedrock/base.py

-Standalone question:"""  # noqa: E501
-        return PromptTemplateWithHistory(
-            template=template, input_variables=["input", "chat_history"]
+        # Change le niveau global à DEBUG


Suggested change

# Change le niveau global à DEBUG

charles-marion · 2024-10-16T22:42:08Z

lib/model-interfaces/langchain/functions/request-handler/adapters/bedrock/base.py

+        # Change le niveau global à DEBUG
+        # Fetch the prompt and translated words based on the current language
+        condense_question_prompt = prompts[locale]["condense_question_prompt"]
+        logger.info(f"condense_question_prompt: {condense_question_prompt}")


Suggested change

logger.info(f"condense_question_prompt: {condense_question_prompt}")

logger.debug(f"condense_question_prompt: {condense_question_prompt}")

I would recommend to mark them all as debug to reduce cloudwatch usage. (nitpick sorry)

charles-marion · 2024-10-16T22:44:53Z

lib/model-interfaces/langchain/functions/request-handler/adapters/base/base.py

+# Setting programmatic log level
+# logger.setLevel("DEBUG")


Suggested change

# Setting programmatic log level

# logger.setLevel("DEBUG")

I would remove this because there is already a global log level setting here
https://github.com/aws-samples/aws-genai-llm-chatbot/blob/main/lib/shared/index.ts#L52

Is it possible to include this information in the developer's guide documentation?

enhanced logging for BedrockChatAdapter initialization, and streamlined QA prompts. Removed redundant base.py_new and ensured BedrockChatAdapter configuration is aligned with main branch consistency.

michel-heon · 2024-10-17T08:36:26Z

I've just finished making the corrections and pushed the updated changes. No worries about the time involved—it's genuinely a pleasure to contribute to this effort.

charles-marion

Thank you for your help with this change!.

I will merge it later this week.

...del-interfaces/langchain/functions/request-handler/adapters/shared/prompts/system_prompts.py

French

michel-heon · 2024-10-29T13:24:03Z

The i18n mechanism works for Bedrock, but not for azureopenai. This fix could be part of a future PR?

charles-marion · 2024-10-29T15:45:48Z

The i18n mechanism works for Bedrock, but not for azureopenai. This fix could be part of a future PR?

Do you mean it breaks the azureopenai flow or just always use english prompts?
Yes it could be a future PR if it does not break capabilities.

Happy to merge today/tomorrow if that's not the case.

michel-heon · 2024-10-30T08:23:33Z

In fact, the use of azureopenai refers to the default system-prompt in langchain, which must be overloaded. The same applies to Mistral. I agree with deferring this development to another PR and merging the current work.

charles-marion · 2024-10-30T22:38:11Z

Build is blocked until the following is merged. #598
I will merge this PR when the above is merged.

charles-marion

I ran the integration tests and fixed the formatting.

LGTM. Thank you for you contribution!

michel-heon mentioned this pull request Sep 24, 2024

Centralize and Internationalize System Prompts Across Adapter #571

Open

charles-marion reviewed Oct 8, 2024

View reviewed changes

Merge remote-tracking branch 'origin/main' into issue-571

a6cb4e0

charles-marion mentioned this pull request Oct 9, 2024

Indexing Q-A in workspace for RAG #582

Open

- Enhance BedrockChatAdapter initialization with logging and QA prompt

a101eb8

template updates; - improve prompt system for multilingual support; - expand test coverage for Bedrock adapters with guardrail integration.

michel-heon force-pushed the issue-571 branch from cbe2635 to a101eb8 Compare October 11, 2024 19:56

michel-heon added 2 commits October 11, 2024 16:41

Merge remote-tracking branch 'origin/main' into issue-571

121c7df

Merge with main id 77ec531

dd91d3f

michel-heon force-pushed the issue-571 branch from f12accb to dd91d3f Compare October 11, 2024 20:57

michel-heon requested a review from charles-marion October 16, 2024 10:13

Suppression de base.py_new

2a56637

charles-marion requested changes Oct 16, 2024

View reviewed changes

Applied requested corrections, including multi-language system prompts,

75204eb

enhanced logging for BedrockChatAdapter initialization, and streamlined QA prompts. Removed redundant base.py_new and ensured BedrockChatAdapter configuration is aligned with main branch consistency.

michel-heon requested a review from charles-marion October 17, 2024 08:36

charles-marion approved these changes Oct 21, 2024

View reviewed changes

charles-marion added 3 commits October 21, 2024 14:47

format: Update python format

f227999

format: removed un-used imports.

48a4d41

Merge branch 'main' into issue-571

2f9ef99

charles-marion reviewed Oct 24, 2024

View reviewed changes

...del-interfaces/langchain/functions/request-handler/adapters/shared/prompts/system_prompts.py Show resolved Hide resolved

Fix system prompt associated to condense_question_prompt for English and

6369e8d

French

michel-heon requested a review from charles-marion October 29, 2024 13:24

Merge branch 'main' into issue-571

3572f1e

format: Update formatting and test

562e1e5

charles-marion approved these changes Oct 31, 2024

View reviewed changes

charles-marion merged commit 6a18d87 into aws-samples:main Oct 31, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add multi-language system prompts and BedrockChatAdapter implementation #576

Add multi-language system prompts and BedrockChatAdapter implementation #576

michel-heon commented Sep 24, 2024

charles-marion left a comment •

edited

Loading

charles-marion Sep 25, 2024

michel-heon Oct 9, 2024

charles-marion Oct 9, 2024

michel-heon Oct 10, 2024

charles-marion Sep 25, 2024

michel-heon Oct 10, 2024

charles-marion Sep 25, 2024

michel-heon Oct 10, 2024

charles-marion commented Oct 15, 2024

michel-heon commented Oct 16, 2024

charles-marion left a comment

charles-marion Oct 16, 2024

charles-marion Oct 16, 2024

charles-marion Oct 16, 2024

charles-marion Oct 16, 2024

charles-marion Oct 16, 2024

charles-marion Oct 16, 2024

charles-marion Oct 16, 2024

charles-marion Oct 16, 2024

michel-heon Oct 19, 2024

michel-heon commented Oct 17, 2024

charles-marion left a comment

michel-heon commented Oct 29, 2024

charles-marion commented Oct 29, 2024 •

edited

Loading

michel-heon commented Oct 30, 2024

charles-marion commented Oct 30, 2024 •

edited

Loading

charles-marion left a comment

	logger.info("No guardrails ID found.")
	logger.debug("No guardrails ID found.")

	logger.info(f"condense_question_prompt: {condense_question_prompt}")
	logger.debug(f"condense_question_prompt: {condense_question_prompt}")

Add multi-language system prompts and BedrockChatAdapter implementation #576

Add multi-language system prompts and BedrockChatAdapter implementation #576

Conversation

michel-heon commented Sep 24, 2024

Pull Request: Centralize and Internationalize System Prompts

Changes:

Key Improvements:

Testing Instructions:

Expected Outcome:

charles-marion left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charles-marion commented Oct 15, 2024

michel-heon commented Oct 16, 2024

charles-marion left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michel-heon commented Oct 17, 2024

charles-marion left a comment

Choose a reason for hiding this comment

michel-heon commented Oct 29, 2024

charles-marion commented Oct 29, 2024 • edited Loading

michel-heon commented Oct 30, 2024

charles-marion commented Oct 30, 2024 • edited Loading

charles-marion left a comment

Choose a reason for hiding this comment

charles-marion left a comment •

edited

Loading

charles-marion commented Oct 29, 2024 •

edited

Loading

charles-marion commented Oct 30, 2024 •

edited

Loading