[SDK] Fix flow as function connection override when node has default …

…variant. (#1971)

# Description

This pull request primarily concerns the handling of flow context
configurations and their potential interactions in the PromptFlow
package. The changes include additional documentation on potential
issues with using multiple overrides, a bug fix for flow function
connection override when a node has a default variant, and adjustments
to the flow context resolver and test cases to accommodate these
changes.

Documentation Updates:

*
[`docs/how-to-guides/execute-flow-as-a-function.md`](diffhunk://#diff-b01a987e6744c2de5b19ecd68893c154271b0350749ad6512dc7de20981e7a4fR87-R99):
Added a section on flow with multiple overrides, warning users about
potential undefined behavior when using `connection` and `overrides` to
override the same node.
*
[`examples/tutorials/get-started/flow-as-function.ipynb`](diffhunk://#diff-6344b82d7eea09b34e46391271f05c9efaa24c09c16d6bd2867196bb32d9a55cL19-R23):
Added a note about the potential issues with flow context configurations
affecting each other.

Bug Fixes:

*
[`src/promptflow/CHANGELOG.md`](diffhunk://#diff-41ec3f7c4b5d4c0e670407d3c00a03a6966d7ebf617b1536473e33a12e2bc765R17):
Documented a bug fix for flow as function connection override when a
node has a default variant.

Code Adjustments:

*
[`src/promptflow/promptflow/_sdk/operations/_flow_context_resolver.py`](diffhunk://#diff-fa095e2bc6aacfd65f8ce4272436ad57636b92a08c143a8b411393ee6a52320bR55-L61):
Modified the `_resolve_variant` method to import `overwrite_variant`
only when necessary and adjusted the handling of the
`flow_context.variant` attribute.
*
[`src/promptflow/tests/sdk_cli_test/e2etests/test_flow_as_func.py`](diffhunk://#diff-164e59c72526e78234995ba0af35845687c275bbd556145539746197d6067ef7R242-R251):
Added a test case for a flow with a default variant, testing that the
function can run successfully with a connection override.
# All Promptflow Contribution checklist:
- [ ] **The pull request does not introduce [breaking changes].**
- [ ] **CHANGELOG is updated for new features, bug fixes or other
significant changes.**
- [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).**
- [ ] **Create an issue and link to the pull request to get dedicated
review from promptflow team. Learn more: [suggested
workflow](../CONTRIBUTING.md#suggested-workflow).**

## General Guidelines and Best Practices
- [ ] Title of the pull request is clear and informative.
- [ ] There are a small number of commits, each of which have an
informative message. This means that previously merged commits do not
appear in the history of the PR. For more information on cleaning up the
commits in your PR, [see this
page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).

### Testing Guidelines
- [ ] Pull request includes test coverage for the included changes.

README.md

@@ -105,7 +105,7 @@ You can install it from the <a href="https://marketplace.visualstudio.com/items?
 
 #### Deep delve into flow development
 
-[Getting started with prompt flow](https://microsoft.github.io/promptflow/how-to-guides/quick-start.html): A step by step guidance to invoke your first flow run.
+[Getting started with prompt flow](./docs/cloud/azureai/quick-start/index.md): A step by step guidance to invoke your first flow run.
 
 ### Learn from use cases
 

docs/how-to-guides/execute-flow-as-a-function.md

@@ -84,6 +84,19 @@ for r in result["answer"]:
 
 Reference our [sample](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/get-started/flow-as-function.ipynb) for usage.
 
+## Flow with multiple overrides
+
+**Note**: the flow context configs may affect each other in some cases. For example, using `connection` & `overrides` to override same node.
+The behavior is undefined for those scenarios. Pleas avoid such usage.
+
+```python
+# overriding `classify_with_llm`'s connection and inputs in the same time will lead to undefined behavior.
+f.context = FlowContext(
+    connections={"classify_with_llm": {"connection": connection_obj}},
+    overrides={"nodes.classify_with_llm.inputs.url": sample_url}
+)
+```
+
 ## Next steps
 
 Learn more about:

examples/tutorials/e2e-development/chat-with-pdf.md

@@ -28,9 +28,9 @@ To go through this tutorial you should:
    pip install -r requirements.txt
 ```
 
-2. Install and configure [Prompt flow for VS Code extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) follow [Quick Start Guide](https://microsoft.github.io/promptflow/how-to-guides/quick-start.html). (_This extension is optional but highly recommended for flow development and debugging._)
+1. Install and configure [Prompt flow for VS Code extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) follow [Quick Start Guide](https://microsoft.github.io/promptflow/how-to-guides/quick-start.html). (_This extension is optional but highly recommended for flow development and debugging._)
 
-3. Deploy an OpenAI or Azure OpenAI chat model (e.g. gpt4 or gpt-35-turbo-16k), and an Embedding model (text-embedding-ada-002).  Follow the [how-to](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal) for an Azure OpenAI example.
+2. Deploy an OpenAI or Azure OpenAI chat model (e.g. gpt4 or gpt-35-turbo-16k), and an Embedding model (text-embedding-ada-002).  Follow the [how-to](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal) for an Azure OpenAI example.
 
 ## Console chatbot chat_with_pdf
 A typical RAG process consists of two steps:

examples/tutorials/get-started/flow-as-function.ipynb

@@ -16,7 +16,11 @@
     "- Execute a flow function with fields override\n",
     "- Execute a flow function with streaming output\n",
     "\n",
-    "**Motivations** - This guide will walk you through the main scenarios of executing flow as a function. You will learn how to consume flow as a function in different scenarios for more pythonnic usage."
+    "**Motivations** - This guide will walk you through the main scenarios of executing flow as a function. You will learn how to consume flow as a function in different scenarios for more pythonnic usage.\n",
+    "\n",
+    "\n",
+    "**Note**: the flow context configs may affect each other in some cases. For example, using `connection` & `overrides` to override same node. \n",
+    "The behavior is undefined for those scenarios. Pleas avoid such usage."
    ]
   },
   {

src/promptflow/CHANGELOG.md

@@ -14,6 +14,7 @@
 - [SDK/CLI][azure] Fixed bug for cloud batch run referencing registry flow with automatic runtime.
 - [SDK/CLI] Fix "Without Import Data" in run visualize page when invalid JSON value exists in metrics.
 - [SDK/CLI][azure] Fix azureml serving get UAI(user assigned identity) token failure bug.
+- [SDK/CLI] Fix flow as function connection override when node has default variant.
 
 ### Improvements
 

src/promptflow/promptflow/_sdk/operations/_flow_context_resolver.py

@@ -52,13 +52,13 @@ def _resolve(self, flow_context: FlowContext):
     def _resolve_variant(self, flow_context: FlowContext) -> "FlowContextResolver":
         """Resolve variant of the flow and store in-memory."""
         # TODO: put all varint string parser here
+        from promptflow._sdk._submitter import overwrite_variant
+
         if not flow_context.variant:
-            return self
+            tuning_node, variant = None, None
         else:
             tuning_node, variant = parse_variant(flow_context.variant)
 
-        from promptflow._sdk._submitter import overwrite_variant
-
         overwrite_variant(
             flow_dag=self.flow_dag,
             tuning_node=tuning_node,

src/promptflow/tests/sdk_cli_test/e2etests/test_flow_as_func.py

@@ -239,3 +239,13 @@ def test_flow_as_func_perf_test(self):
         f = load_flow(f"{FLOWS_DIR}/print_env_var")
         for i in range(100):
             f(key="key")
+
+    def test_flow_with_default_variant(self, azure_open_ai_connection):
+        f = load_flow(f"{FLOWS_DIR}/web_classification_default_variant_no_llm_type")
+        f.context = FlowContext(
+            connections={
+                "summarize_text_content": {"connection": azure_open_ai_connection},
+            }
+        )
+        # function can successfully run with connection override
+        f(url="https://www.youtube.com/watch?v=o5ZQyXaAv1g")

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/classify_with_llm.jinja2

@@ -0,0 +1,21 @@
+system:
+Your task is to classify a given url into one of the following categories:
+Movie, App, Academic, Channel, Profile, PDF or None based on the text content information.
+The classification will be based on the url, the webpage text content summary, or both.
+
+user:
+The selection range of the value of "category" must be within "Movie", "App", "Academic", "Channel", "Profile", "PDF" and "None".
+The selection range of the value of "evidence" must be within "Url", "Text content", and "Both".
+Here are a few examples:
+{% for ex in examples %}
+URL: {{ex.url}}
+Text content: {{ex.text_content}}
+OUTPUT:
+{"category": "{{ex.category}}", "evidence": "{{ex.evidence}}"}
+
+{% endfor %}
+
+For a given URL and text content, classify the url to complete the category and indicate evidence:
+URL: {{url}}
+Text content: {{text_content}}.
+OUTPUT:

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/convert_to_dict.py

@@ -0,0 +1,12 @@
+import json
+
+from promptflow import tool
+
+
+@tool
+def convert_to_dict(input_str: str):
+    try:
+        return json.loads(input_str)
+    except Exception as e:
+        print("input is not valid, error: {}".format(e))
+        return {"category": "None", "evidence": "None"}

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/data.jsonl

@@ -0,0 +1,3 @@
+{"url": "https://www.youtube.com/watch?v=kYqRtjDBci8", "answer": "Channel", "evidence": "Both"}
+{"url": "https://arxiv.org/abs/2307.04767", "answer": "Academic", "evidence": "Both"}
+{"url": "https://play.google.com/store/apps/details?id=com.twitter.android", "answer": "App", "evidence": "Both"}

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url.py

@@ -0,0 +1,28 @@
+import bs4
+import requests
+
+from promptflow import tool
+
+
+@tool
+def fetch_text_content_from_url(fetch_url: str):
+    # Send a request to the URL
+    try:
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.35"
+        }
+        response = requests.get(fetch_url, headers=headers)
+        if response.status_code == 200:
+            # Parse the HTML content using BeautifulSoup
+            soup = bs4.BeautifulSoup(response.text, "html.parser")
+            soup.prettify()
+            return soup.get_text()[:2000]
+        else:
+            msg = (
+                f"Get url failed with status code {response.status_code}.\nURL: {fetch_url}\nResponse: {response.text[:100]}"
+            )
+            print(msg)
+            return "No available content"
+    except Exception as e:
+        print("Get url failed with error: {}".format(e))
+        return "No available content"

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/fetch_text_content_from_url_input.jsonl

@@ -0,0 +1 @@
+{"inputs.url":"https://www.microsoft.com/en-us/d/xbox-wireless-controller-stellar-shift-special-edition/94fbjc7h0h6h"}

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/flow.dag.yaml

@@ -0,0 +1,110 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+inputs:
+  url:
+    type: string
+    default: https://www.microsoft.com/en-us/d/xbox-wireless-controller-stellar-shift-special-edition/94fbjc7h0h6h
+outputs:
+  category:
+    type: string
+    reference: ${convert_to_dict.output.category}
+  evidence:
+    type: string
+    reference: ${convert_to_dict.output.evidence}
+nodes:
+- name: fetch_text_content_from_url
+  type: python
+  source:
+    type: code
+    path: fetch_text_content_from_url.py
+  inputs:
+    fetch_url: ${inputs.url}
+- name: summarize_text_content
+  use_variants: true
+- name: prepare_examples
+  type: python
+  source:
+    type: code
+    path: prepare_examples.py
+  inputs: {}
+- name: classify_with_llm
+  type: llm
+  source:
+    type: code
+    path: classify_with_llm.jinja2
+  inputs:
+    deployment_name: gpt-35-turbo
+    suffix: ''
+    max_tokens: '128'
+    temperature: '0.1'
+    top_p: '1.0'
+    logprobs: ''
+    echo: 'False'
+    stop: ''
+    presence_penalty: '0'
+    frequency_penalty: '0'
+    best_of: '1'
+    logit_bias: ''
+    url: ${inputs.url}
+    examples: ${prepare_examples.output}
+    text_content: ${summarize_text_content.output}
+  provider: AzureOpenAI
+  connection: azure_open_ai_connection
+  api: chat
+  module: promptflow.tools.aoai
+- name: convert_to_dict
+  type: python
+  source:
+    type: code
+    path: convert_to_dict.py
+  inputs:
+    input_str: ${classify_with_llm.output}
+node_variants:
+  summarize_text_content:
+    default_variant_id: variant_1
+    variants:
+      variant_0:
+        node:
+          type: llm
+          source:
+            type: code
+            path: summarize_text_content.jinja2
+          inputs:
+            deployment_name: gpt-35-turbo
+            suffix: ''
+            max_tokens: '128'
+            temperature: '0.2'
+            top_p: '1.0'
+            logprobs: ''
+            echo: 'False'
+            stop: ''
+            presence_penalty: '0'
+            frequency_penalty: '0'
+            best_of: '1'
+            logit_bias: ''
+            text: ${fetch_text_content_from_url.output}
+          provider: AzureOpenAI
+          connection: azure_open_ai_connection
+          api: chat
+          module: promptflow.tools.aoai
+      variant_1:
+        node:
+          type: llm
+          source:
+            type: code
+            path: summarize_text_content__variant_1.jinja2
+          inputs:
+            deployment_name: gpt-35-turbo
+            suffix: ''
+            max_tokens: '256'
+            temperature: '0.3'
+            top_p: '1.0'
+            logprobs: ''
+            echo: 'False'
+            stop: ''
+            presence_penalty: '0'
+            frequency_penalty: '0'
+            best_of: '1'
+            logit_bias: ''
+            text: ${fetch_text_content_from_url.output}
+          connection: azure_open_ai_connection
+          api: chat

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/prepare_examples.py

@@ -0,0 +1,31 @@
+from promptflow import tool
+
+
+@tool
+def prepare_examples():
+    return [
+        {
+            "url": "https://play.google.com/store/apps/details?id=com.spotify.music",
+            "text_content": "Spotify is a free music and podcast streaming app with millions of songs, albums, and original podcasts. It also offers audiobooks, so users can enjoy thousands of stories. It has a variety of features such as creating and sharing music playlists, discovering new music, and listening to popular and exclusive podcasts. It also has a Premium subscription option which allows users to download and listen offline, and access ad-free music. It is available on all devices and has a variety of genres and artists to choose from.",
+            "category": "App",
+            "evidence": "Both",
+        },
+        {
+            "url": "https://www.youtube.com/channel/UC_x5XG1OV2P6uZZ5FSM9Ttw",
+            "text_content": "NFL Sunday Ticket is a service offered by Google LLC that allows users to watch NFL games on YouTube. It is available in 2023 and is subject to the terms and privacy policy of Google LLC. It is also subject to YouTube's terms of use and any applicable laws.",
+            "category": "Channel",
+            "evidence": "URL",
+        },
+        {
+            "url": "https://arxiv.org/abs/2303.04671",
+            "text_content": "Visual ChatGPT is a system that enables users to interact with ChatGPT by sending and receiving not only languages but also images, providing complex visual questions or visual editing instructions, and providing feedback and asking for corrected results. It incorporates different Visual Foundation Models and is publicly available. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models.",
+            "category": "Academic",
+            "evidence": "Text content",
+        },
+        {
+            "url": "https://ab.politiaromana.ro/",
+            "text_content": "There is no content available for this text.",
+            "category": "None",
+            "evidence": "None",
+        },
+    ]

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/samples.json

@@ -0,0 +1,8 @@
+[
+  {
+    "url": "https://www.microsoft.com/en-us/d/xbox-wireless-controller-stellar-shift-special-edition/94fbjc7h0h6h"
+  },
+  {
+    "url": "https://www.microsoft.com/en-us/windows/"
+  }
+]

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content.jinja2

@@ -0,0 +1,7 @@
+system:
+Please summarize the following text in one paragraph. 100 words.
+Do not add any information that is not in the text.
+
+user:
+Text: {{text}}
+Summary:

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/summarize_text_content__variant_1.jinja2

@@ -0,0 +1,7 @@
+system:
+Please summarize some keywords of this paragraph and have some details of each keywords.
+Do not add any information that is not in the text.
+
+user:
+Text: {{text}}
+Summary:

src/promptflow/tests/test_configs/flows/web_classification_default_variant_no_llm_type/webClassification20.csv

@@ -0,0 +1,21 @@
+url,answer,evidence
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Channel,Url
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Channel,Url
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Channel,Url
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Channel,Url
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Url
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Url
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,None,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,None,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Url
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Url
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Profile,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,POI,Url
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,None,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content
+https://www.youtube.com/watch?v=o5ZQyXaAv1g,Academic,Text content