From ceb703639d5a8e04ba7390f93c422c6888f779ed Mon Sep 17 00:00:00 2001 From: Han Wang Date: Mon, 5 Feb 2024 17:58:34 +0800 Subject: [PATCH] [SDK] Fix flow as function connection override when node has default variant. (#1971) # Description This pull request primarily concerns the handling of flow context configurations and their potential interactions in the PromptFlow package. The changes include additional documentation on potential issues with using multiple overrides, a bug fix for flow function connection override when a node has a default variant, and adjustments to the flow context resolver and test cases to accommodate these changes. Documentation Updates: * [`docs/how-to-guides/execute-flow-as-a-function.md`](diffhunk://#diff-b01a987e6744c2de5b19ecd68893c154271b0350749ad6512dc7de20981e7a4fR87-R99): Added a section on flow with multiple overrides, warning users about potential undefined behavior when using `connection` and `overrides` to override the same node. * [`examples/tutorials/get-started/flow-as-function.ipynb`](diffhunk://#diff-6344b82d7eea09b34e46391271f05c9efaa24c09c16d6bd2867196bb32d9a55cL19-R23): Added a note about the potential issues with flow context configurations affecting each other. Bug Fixes: * [`src/promptflow/CHANGELOG.md`](diffhunk://#diff-41ec3f7c4b5d4c0e670407d3c00a03a6966d7ebf617b1536473e33a12e2bc765R17): Documented a bug fix for flow as function connection override when a node has a default variant. Code Adjustments: * [`src/promptflow/promptflow/_sdk/operations/_flow_context_resolver.py`](diffhunk://#diff-fa095e2bc6aacfd65f8ce4272436ad57636b92a08c143a8b411393ee6a52320bR55-L61): Modified the `_resolve_variant` method to import `overwrite_variant` only when necessary and adjusted the handling of the `flow_context.variant` attribute. * [`src/promptflow/tests/sdk_cli_test/e2etests/test_flow_as_func.py`](diffhunk://#diff-164e59c72526e78234995ba0af35845687c275bbd556145539746197d6067ef7R242-R251): Added a test case for a flow with a default variant, testing that the function can run successfully with a connection override. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --- README.md | 2 +- .../execute-flow-as-a-function.md | 13 +++ .../e2e-development/chat-with-pdf.md | 4 +- .../get-started/flow-as-function.ipynb | 6 +- src/promptflow/CHANGELOG.md | 1 + .../_sdk/operations/_flow_context_resolver.py | 6 +- .../e2etests/test_flow_as_func.py | 10 ++ .../classify_with_llm.jinja2 | 21 ++++ .../convert_to_dict.py | 12 ++ .../data.jsonl | 3 + .../fetch_text_content_from_url.py | 28 +++++ .../fetch_text_content_from_url_input.jsonl | 1 + .../flow.dag.yaml | 110 ++++++++++++++++++ .../prepare_examples.py | 31 +++++ .../samples.json | 8 ++ .../summarize_text_content.jinja2 | 7 ++ .../summarize_text_content__variant_1.jinja2 | 7 ++ .../webClassification20.csv | 21 ++++ 18 files changed, 284 insertions(+), 7 deletions(-) create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/classify_with_llm.jinja2 create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/convert_to_dict.py create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/data.jsonl create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url.py create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url_input.jsonl create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/flow.dag.yaml create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/prepare_examples.py create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/samples.json create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content.jinja2 create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content__variant_1.jinja2 create mode 100644 src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/webClassification20.csv diff --git a/README.md b/README.md index 4fa416b71ff..b8a2b9b0adf 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ You can install it from the "FlowContextResolver": """Resolve variant of the flow and store in-memory.""" # TODO: put all varint string parser here + from promptflow._sdk._submitter import overwrite_variant + if not flow_context.variant: - return self + tuning_node, variant = None, None else: tuning_node, variant = parse_variant(flow_context.variant) - from promptflow._sdk._submitter import overwrite_variant - overwrite_variant( flow_dag=self.flow_dag, tuning_node=tuning_node, diff --git a/src/promptflow/tests/sdk_cli_test/e2etests/test_flow_as_func.py b/src/promptflow/tests/sdk_cli_test/e2etests/test_flow_as_func.py index b828b50771b..def2f0f0443 100644 --- a/src/promptflow/tests/sdk_cli_test/e2etests/test_flow_as_func.py +++ b/src/promptflow/tests/sdk_cli_test/e2etests/test_flow_as_func.py @@ -239,3 +239,13 @@ def test_flow_as_func_perf_test(self): f = load_flow(f"{FLOWS_DIR}/print_env_var") for i in range(100): f(key="key") + + def test_flow_with_default_variant(self, azure_open_ai_connection): + f = load_flow(f"{FLOWS_DIR}/web_classification_default_variant_no_llm_type") + f.context = FlowContext( + connections={ + "summarize_text_content": {"connection": azure_open_ai_connection}, + } + ) + # function can successfully run with connection override + f(url="https://www.youtube.com/watch?v=o5ZQyXaAv1g") diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/classify_with_llm.jinja2 b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/classify_with_llm.jinja2 new file mode 100644 index 00000000000..6d7c3da4005 --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/classify_with_llm.jinja2 @@ -0,0 +1,21 @@ +system: +Your task is to classify a given url into one of the following categories: +Movie, App, Academic, Channel, Profile, PDF or None based on the text content information. +The classification will be based on the url, the webpage text content summary, or both. + +user: +The selection range of the value of "category" must be within "Movie", "App", "Academic", "Channel", "Profile", "PDF" and "None". +The selection range of the value of "evidence" must be within "Url", "Text content", and "Both". +Here are a few examples: +{% for ex in examples %} +URL: {{ex.url}} +Text content: {{ex.text_content}} +OUTPUT: +{"category": "{{ex.category}}", "evidence": "{{ex.evidence}}"} + +{% endfor %} + +For a given URL and text content, classify the url to complete the category and indicate evidence: +URL: {{url}} +Text content: {{text_content}}. +OUTPUT: \ No newline at end of file diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/convert_to_dict.py b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/convert_to_dict.py new file mode 100644 index 00000000000..3b287df5c65 --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/convert_to_dict.py @@ -0,0 +1,12 @@ +import json + +from promptflow import tool + + +@tool +def convert_to_dict(input_str: str): + try: + return json.loads(input_str) + except Exception as e: + print("input is not valid, error: {}".format(e)) + return {"category": "None", "evidence": "None"} diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/data.jsonl b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/data.jsonl new file mode 100644 index 00000000000..248b61c6e95 --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/data.jsonl @@ -0,0 +1,3 @@ +{"url": "https://www.youtube.com/watch?v=kYqRtjDBci8", "answer": "Channel", "evidence": "Both"} +{"url": "https://arxiv.org/abs/2307.04767", "answer": "Academic", "evidence": "Both"} +{"url": "https://play.google.com/store/apps/details?id=com.twitter.android", "answer": "App", "evidence": "Both"} diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url.py b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url.py new file mode 100644 index 00000000000..5eb5ea345d4 --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url.py @@ -0,0 +1,28 @@ +import bs4 +import requests + +from promptflow import tool + + +@tool +def fetch_text_content_from_url(fetch_url: str): + # Send a request to the URL + try: + headers = { + "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.35" + } + response = requests.get(fetch_url, headers=headers) + if response.status_code == 200: + # Parse the HTML content using BeautifulSoup + soup = bs4.BeautifulSoup(response.text, "html.parser") + soup.prettify() + return soup.get_text()[:2000] + else: + msg = ( + f"Get url failed with status code {response.status_code}.\nURL: {fetch_url}\nResponse: {response.text[:100]}" + ) + print(msg) + return "No available content" + except Exception as e: + print("Get url failed with error: {}".format(e)) + return "No available content" diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url_input.jsonl b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url_input.jsonl new file mode 100644 index 00000000000..bd6303cc04a --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url_input.jsonl @@ -0,0 +1 @@ +{"inputs.url":"https://www.microsoft.com/en-us/d/xbox-wireless-controller-stellar-shift-special-edition/94fbjc7h0h6h"} diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/flow.dag.yaml b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/flow.dag.yaml new file mode 100644 index 00000000000..f57858a34e1 --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/flow.dag.yaml @@ -0,0 +1,110 @@ +$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json +inputs: + url: + type: string + default: https://www.microsoft.com/en-us/d/xbox-wireless-controller-stellar-shift-special-edition/94fbjc7h0h6h +outputs: + category: + type: string + reference: ${convert_to_dict.output.category} + evidence: + type: string + reference: ${convert_to_dict.output.evidence} +nodes: +- name: fetch_text_content_from_url + type: python + source: + type: code + path: fetch_text_content_from_url.py + inputs: + fetch_url: ${inputs.url} +- name: summarize_text_content + use_variants: true +- name: prepare_examples + type: python + source: + type: code + path: prepare_examples.py + inputs: {} +- name: classify_with_llm + type: llm + source: + type: code + path: classify_with_llm.jinja2 + inputs: + deployment_name: gpt-35-turbo + suffix: '' + max_tokens: '128' + temperature: '0.1' + top_p: '1.0' + logprobs: '' + echo: 'False' + stop: '' + presence_penalty: '0' + frequency_penalty: '0' + best_of: '1' + logit_bias: '' + url: ${inputs.url} + examples: ${prepare_examples.output} + text_content: ${summarize_text_content.output} + provider: AzureOpenAI + connection: azure_open_ai_connection + api: chat + module: promptflow.tools.aoai +- name: convert_to_dict + type: python + source: + type: code + path: convert_to_dict.py + inputs: + input_str: ${classify_with_llm.output} +node_variants: + summarize_text_content: + default_variant_id: variant_1 + variants: + variant_0: + node: + type: llm + source: + type: code + path: summarize_text_content.jinja2 + inputs: + deployment_name: gpt-35-turbo + suffix: '' + max_tokens: '128' + temperature: '0.2' + top_p: '1.0' + logprobs: '' + echo: 'False' + stop: '' + presence_penalty: '0' + frequency_penalty: '0' + best_of: '1' + logit_bias: '' + text: ${fetch_text_content_from_url.output} + provider: AzureOpenAI + connection: azure_open_ai_connection + api: chat + module: promptflow.tools.aoai + variant_1: + node: + type: llm + source: + type: code + path: summarize_text_content__variant_1.jinja2 + inputs: + deployment_name: gpt-35-turbo + suffix: '' + max_tokens: '256' + temperature: '0.3' + top_p: '1.0' + logprobs: '' + echo: 'False' + stop: '' + presence_penalty: '0' + frequency_penalty: '0' + best_of: '1' + logit_bias: '' + text: ${fetch_text_content_from_url.output} + connection: azure_open_ai_connection + api: chat diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/prepare_examples.py b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/prepare_examples.py new file mode 100644 index 00000000000..40d007ad4a2 --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/prepare_examples.py @@ -0,0 +1,31 @@ +from promptflow import tool + + +@tool +def prepare_examples(): + return [ + { + "url": "https://play.google.com/store/apps/details?id=com.spotify.music", + "text_content": "Spotify is a free music and podcast streaming app with millions of songs, albums, and original podcasts. It also offers audiobooks, so users can enjoy thousands of stories. It has a variety of features such as creating and sharing music playlists, discovering new music, and listening to popular and exclusive podcasts. It also has a Premium subscription option which allows users to download and listen offline, and access ad-free music. It is available on all devices and has a variety of genres and artists to choose from.", + "category": "App", + "evidence": "Both", + }, + { + "url": "https://www.youtube.com/channel/UC_x5XG1OV2P6uZZ5FSM9Ttw", + "text_content": "NFL Sunday Ticket is a service offered by Google LLC that allows users to watch NFL games on YouTube. It is available in 2023 and is subject to the terms and privacy policy of Google LLC. It is also subject to YouTube's terms of use and any applicable laws.", + "category": "Channel", + "evidence": "URL", + }, + { + "url": "https://arxiv.org/abs/2303.04671", + "text_content": "Visual ChatGPT is a system that enables users to interact with ChatGPT by sending and receiving not only languages but also images, providing complex visual questions or visual editing instructions, and providing feedback and asking for corrected results. It incorporates different Visual Foundation Models and is publicly available. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models.", + "category": "Academic", + "evidence": "Text content", + }, + { + "url": "https://ab.politiaromana.ro/", + "text_content": "There is no content available for this text.", + "category": "None", + "evidence": "None", + }, + ] diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/samples.json b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/samples.json new file mode 100644 index 00000000000..e757982fd10 --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/samples.json @@ -0,0 +1,8 @@ +[ + { + "url": "https://www.microsoft.com/en-us/d/xbox-wireless-controller-stellar-shift-special-edition/94fbjc7h0h6h" + }, + { + "url": "https://www.microsoft.com/en-us/windows/" + } +] diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content.jinja2 b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content.jinja2 new file mode 100644 index 00000000000..f27cb1c42e2 --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content.jinja2 @@ -0,0 +1,7 @@ +system: +Please summarize the following text in one paragraph. 100 words. +Do not add any information that is not in the text. + +user: +Text: {{text}} +Summary: diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content__variant_1.jinja2 b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content__variant_1.jinja2 new file mode 100644 index 00000000000..d340923ca19 --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content__variant_1.jinja2 @@ -0,0 +1,7 @@ +system: +Please summarize some keywords of this paragraph and have some details of each keywords. +Do not add any information that is not in the text. + +user: +Text: {{text}} +Summary: diff --git a/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/webClassification20.csv b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/webClassification20.csv new file mode 100644 index 00000000000..ab695088ecb --- /dev/null +++ b/src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/webClassification20.csv @@ -0,0 +1,21 @@ +url,answer,evidence +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Channel,Url +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Channel,Url +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Channel,Url +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Channel,Url +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Url +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Url +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,None,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,None,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Url +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Url +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,POI,Url +https://www.youtube.com/watch?v=o5ZQyXaAv1g,None,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content +https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content