Preview samples of pf code first experience (microsoft#1)

Signed-off-by: Brynn Yin <[email protected]> Co-authored-by: Clement Wang <[email protected]> Co-authored-by: Korin <[email protected]> Co-authored-by: Brynn Yin <[email protected]> Co-authored-by: Han Wang <[email protected]> Co-authored-by: Brynn Yin <[email protected]> Co-authored-by: zhangxingzhi <[email protected]> Co-authored-by: Xingzhi Zhang <[email protected]> Co-authored-by: Zhengfei Wang <[email protected]> Co-authored-by: Daniel Schneider <[email protected]> Co-authored-by: zhen <[email protected]> Co-authored-by: Philip Gao (from Dev Box) <[email protected]> Co-authored-by: Ying Chen <[email protected]> Co-authored-by: Ying Chen <[email protected]>
ttthree · Aug 3, 2023 · b9cb0e1 · b9cb0e1
1 parent 500c4c3
commit b9cb0e1
Show file tree

Hide file tree

Showing 141 changed files with 5,961 additions and 5 deletions.
diff --git a/.github/workflows/configuration.yml b/.github/workflows/configuration.yml
@@ -0,0 +1,44 @@
+# This code is autogenerated.
+# Code is generated by running custom script: python3 readme.py
+# Any manual changes to this file may cause incorrect behavior.
+# Any manual changes will be overwritten if the code is regenerated.
+
+name: configuration
+on:
+  pull_request:
+    branches: [ main,preview/code-first ]
+  workflow_dispatch:
+
+jobs:
+  configuration:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Generate config.json
+        run: echo ${{ secrets.TEST_WORKSPACE_CONFIG_JSON }} > ${{ github.workspace }}/examples/config.json
+      - name: Azure Login
+        uses: azure/login@v1
+        with:
+          creds: ${{ secrets.AZURE_CREDENTIALS }}
+      - name: Setup Python 3.9 environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: Prepare requirements
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ${{ github.workspace }}/examples/requirements.txt
+          pip install -r ${{ github.workspace }}/examples/dev_requirements.txt
+      - name: Create Aoai Connection
+        run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY }}" api_base="${{ secrets.AOAI_API_ENDPOINT }}"
+      - name: Test Notebook
+        working-directory: examples
+        run: |
+          papermill -k python configuration.ipynb configuration.output.ipynb
+      - name: Upload artifact
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: artifact
+          path: examples
diff --git a/.github/workflows/connections_connection.yml b/.github/workflows/connections_connection.yml
@@ -0,0 +1,44 @@
+# This code is autogenerated.
+# Code is generated by running custom script: python3 readme.py
+# Any manual changes to this file may cause incorrect behavior.
+# Any manual changes will be overwritten if the code is regenerated.
+
+name: connections_connection
+on:
+  pull_request:
+    branches: [ main,preview/code-first ]
+  workflow_dispatch:
+
+jobs:
+  connection:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Generate config.json
+        run: echo ${{ secrets.TEST_WORKSPACE_CONFIG_JSON }} > ${{ github.workspace }}/examples/config.json
+      - name: Azure Login
+        uses: azure/login@v1
+        with:
+          creds: ${{ secrets.AZURE_CREDENTIALS }}
+      - name: Setup Python 3.9 environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: Prepare requirements
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ${{ github.workspace }}/examples/requirements.txt
+          pip install -r ${{ github.workspace }}/examples/dev_requirements.txt
+      - name: Create Aoai Connection
+        run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY }}" api_base="${{ secrets.AOAI_API_ENDPOINT }}"
+      - name: Test Notebook
+        working-directory: examples/connections
+        run: |
+          papermill -k python connection.ipynb connection.output.ipynb
+      - name: Upload artifact
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: artifact
+          path: examples/connections
diff --git a/.github/workflows/tutorials_flowinpipeline_pipeline.yml b/.github/workflows/tutorials_flowinpipeline_pipeline.yml
@@ -0,0 +1,44 @@
+# This code is autogenerated.
+# Code is generated by running custom script: python3 readme.py
+# Any manual changes to this file may cause incorrect behavior.
+# Any manual changes will be overwritten if the code is regenerated.
+
+name: tutorials_flowinpipeline_pipeline
+on:
+  pull_request:
+    branches: [ main,preview/code-first ]
+  workflow_dispatch:
+
+jobs:
+  pipeline:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Generate config.json
+        run: echo ${{ secrets.TEST_WORKSPACE_CONFIG_JSON }} > ${{ github.workspace }}/examples/config.json
+      - name: Azure Login
+        uses: azure/login@v1
+        with:
+          creds: ${{ secrets.AZURE_CREDENTIALS }}
+      - name: Setup Python 3.9 environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: Prepare requirements
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ${{ github.workspace }}/examples/requirements.txt
+          pip install -r ${{ github.workspace }}/examples/dev_requirements.txt
+      - name: Create Aoai Connection
+        run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY }}" api_base="${{ secrets.AOAI_API_ENDPOINT }}"
+      - name: Test Notebook
+        working-directory: examples/tutorials/flow-in-pipeline
+        run: |
+          papermill -k python pipeline.ipynb pipeline.output.ipynb
+      - name: Upload artifact
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: artifact
+          path: examples/tutorials/flow-in-pipeline
diff --git a/.github/workflows/tutorials_getstarted_quickstart.yml b/.github/workflows/tutorials_getstarted_quickstart.yml
@@ -0,0 +1,44 @@
+# This code is autogenerated.
+# Code is generated by running custom script: python3 readme.py
+# Any manual changes to this file may cause incorrect behavior.
+# Any manual changes will be overwritten if the code is regenerated.
+
+name: tutorials_getstarted_quickstart
+on:
+  pull_request:
+    branches: [ main,preview/code-first ]
+  workflow_dispatch:
+
+jobs:
+  quickstart:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Generate config.json
+        run: echo ${{ secrets.TEST_WORKSPACE_CONFIG_JSON }} > ${{ github.workspace }}/examples/config.json
+      - name: Azure Login
+        uses: azure/login@v1
+        with:
+          creds: ${{ secrets.AZURE_CREDENTIALS }}
+      - name: Setup Python 3.9 environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: Prepare requirements
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ${{ github.workspace }}/examples/requirements.txt
+          pip install -r ${{ github.workspace }}/examples/dev_requirements.txt
+      - name: Create Aoai Connection
+        run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY }}" api_base="${{ secrets.AOAI_API_ENDPOINT }}"
+      - name: Test Notebook
+        working-directory: examples/tutorials/get-started
+        run: |
+          papermill -k python quickstart.ipynb quickstart.output.ipynb
+      - name: Upload artifact
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: artifact
+          path: examples/tutorials/get-started
diff --git a/.github/workflows/tutorials_getstarted_quickstartazure.yml b/.github/workflows/tutorials_getstarted_quickstartazure.yml
@@ -0,0 +1,44 @@
+# This code is autogenerated.
+# Code is generated by running custom script: python3 readme.py
+# Any manual changes to this file may cause incorrect behavior.
+# Any manual changes will be overwritten if the code is regenerated.
+
+name: tutorials_getstarted_quickstartazure
+on:
+  pull_request:
+    branches: [ main,preview/code-first ]
+  workflow_dispatch:
+
+jobs:
+  quickstart-azure:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Generate config.json
+        run: echo ${{ secrets.TEST_WORKSPACE_CONFIG_JSON }} > ${{ github.workspace }}/examples/config.json
+      - name: Azure Login
+        uses: azure/login@v1
+        with:
+          creds: ${{ secrets.AZURE_CREDENTIALS }}
+      - name: Setup Python 3.9 environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: Prepare requirements
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ${{ github.workspace }}/examples/requirements.txt
+          pip install -r ${{ github.workspace }}/examples/dev_requirements.txt
+      - name: Create Aoai Connection
+        run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY }}" api_base="${{ secrets.AOAI_API_ENDPOINT }}"
+      - name: Test Notebook
+        working-directory: examples/tutorials/get-started
+        run: |
+          papermill -k python quickstart-azure.ipynb quickstart-azure.output.ipynb
+      - name: Upload artifact
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: artifact
+          path: examples/tutorials/get-started
diff --git a/.github/workflows/tutorials_runmanagement_cloudrunmanagement.yml b/.github/workflows/tutorials_runmanagement_cloudrunmanagement.yml
@@ -0,0 +1,44 @@
+# This code is autogenerated.
+# Code is generated by running custom script: python3 readme.py
+# Any manual changes to this file may cause incorrect behavior.
+# Any manual changes will be overwritten if the code is regenerated.
+
+name: tutorials_runmanagement_cloudrunmanagement
+on:
+  pull_request:
+    branches: [ main,preview/code-first ]
+  workflow_dispatch:
+
+jobs:
+  cloud-run-management:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+      - name: Generate config.json
+        run: echo ${{ secrets.TEST_WORKSPACE_CONFIG_JSON }} > ${{ github.workspace }}/examples/config.json
+      - name: Azure Login
+        uses: azure/login@v1
+        with:
+          creds: ${{ secrets.AZURE_CREDENTIALS }}
+      - name: Setup Python 3.9 environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: Prepare requirements
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ${{ github.workspace }}/examples/requirements.txt
+          pip install -r ${{ github.workspace }}/examples/dev_requirements.txt
+      - name: Create Aoai Connection
+        run: pf connection create -f ${{ github.workspace }}/examples/connections/azure_openai.yml --set api_key="${{ secrets.AOAI_API_KEY }}" api_base="${{ secrets.AOAI_API_ENDPOINT }}"
+      - name: Test Notebook
+        working-directory: examples/tutorials/run-management
+        run: |
+          papermill -k python cloud-run-management.ipynb cloud-run-management.output.ipynb
+      - name: Upload artifact
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v3
+        with:
+          name: artifact
+          path: examples/tutorials/run-management
diff --git a/.gitignore b/.gitignore
@@ -82,6 +82,8 @@ target/
 profile_default/
 ipython_config.py
 
+# Vscode
+.vscode/
 # pyenv
 #   For a library or package, you might want to ignore these files since the code is
 #   intended to run in multiple environments; otherwise, check them in:
@@ -159,5 +161,12 @@ cython_debug/
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
 
+# ignore 
+# TODO: ignore when PFS supports submit without tools.json
+.promptflow
+.runs
+connection.json
+.env
+.azureml
 # secrets
 **/connections.json
diff --git a/README.md b/README.md
@@ -21,13 +21,13 @@ With prompt flow, you will be able to:
 
 You can develope your flow locally and seamlessly move the experience to azure cloud, learn more: [Azure Machine Learning prompt flow](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/overview-what-is-prompt-flow?view=azureml-api-2).
 
-By joining the Prompt Flow community, you can build AI-first apps faster and have a front-row
-peek at how the SDK is being built. Prompt Flow has been released as open-source so that more
-pioneering developers can join us in crafting the future of how people building AI applications.
+Prompt Flow has been released as open-source so that more pioneering developers can join us in crafting the future of how people building AI applications.
 
 ## Get Started with Prompt flow ⚡
 
-WIP
+For users: please start with our [examples](./examples/README.md) & [docs](./docs/README.md).
+
+For developers: please start with our dev setup guide: [dev_setup.md](./docs/dev/dev_setup.md).
 
 ## Contributing
 

diff --git a/docs/README.md b/docs/README.md
@@ -0,0 +1,3 @@
+# Promptflow Docs
+
+WIP.
diff --git a/docs/dev/dev_setup.md b/docs/dev/dev_setup.md
@@ -1 +1,2 @@
+# Dev Steup
 WIP
diff --git a/examples/README.md b/examples/README.md
@@ -0,0 +1,64 @@
+# Promptflow examples
+
+[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![license: MIT](https://img.shields.io/badge/License-MIT-purple.svg)](../LICENSE)
+
+## Prerequisites
+
+- Bootstrap your python env. 
+  - e.g: create a new [conda](https://conda.io/projects/conda/en/latest/user-guide/getting-started.html) environment. `conda create -n pf-examples python=3.9`.
+  - install required packages in python environment : `pip install -r requirements.txt`
+    - show installed sdk: `pip show promptflow`
+
+
+## Examples available
+
+NOTE: the status in below table is a fake one. WIP.
+
+**Tutorials** ([tutorials](tutorials))
+path|status|description
+-|-|-
+[quickstart.ipynb](tutorials/get-started/quickstart.ipynb)|[![tutorials_getstarted_quickstart](https://github.com/microsoft/promptflow/actions/workflows/tutorials_getstarted_quickstart.yml/badge.svg)](https://github.com/microsoft/promptflow/actions/workflows/tutorials_getstarted_quickstart.yml)| get started
+[deploy.md](tutorials/flow-deploy/deploy.md)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| deploy flow as endpoint
+[run.ipynb](tutorials/advanced-run-management/run.ipynb)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| advanced flow run management
+[quickstart-azure.ipynb](tutorials/get-started/quickstart-azure.ipynb)|[![tutorials_getstarted_quickstartazure](https://github.com/microsoft/promptflow/actions/workflows/tutorials_getstarted_quickstartazure.yml/badge.svg)](https://github.com/microsoft/promptflow/actions/workflows/tutorials_getstarted_quickstartazure.yml)| get started - local to cloud
+[pipeline.ipynb](tutorials/flow-in-pipeline/pipeline.ipynb)|[![tutorials_flowinpipeline_pipeline](https://github.com/microsoft/promptflow/actions/workflows/tutorials_flowinpipeline_pipeline.yml/badge.svg)](https://github.com/microsoft/promptflow/actions/workflows/tutorials_flowinpipeline_pipeline.yml)| flow as component in pipeline
+[cloud-run-management.ipynb](tutorials/run-management/cloud-run-management.ipynb)|[![tutorials_runmanagement_cloudrunmanagement](https://github.com/microsoft/promptflow/actions/workflows/tutorials_runmanagement_cloudrunmanagement.yml/badge.svg)](https://github.com/microsoft/promptflow/actions/workflows/tutorials_runmanagement_cloudrunmanagement.yml)| cloud run management
+
+
+**Flows** ([flows](flows))
+
+[Standard](flows/standard/) flows
+
+path|status|description
+-|-|-
+[basic](flows/standard/basic/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a basic flow with prompt and python tool.
+[basic-with-connection](flows/standard/basic-with-connection/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a basic flow using custom connection with prompt and python tool
+[basic-with-builtin-llm](flows/standard/basic-with-builtin-llm/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a basic flow using builtin llm tool
+[intent-copilot](flows/standard/intent-copilot/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a flow created from existing langchain python code
+[flow-with-symlinks](flows/standard/flow-with-symlinks/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a flow created with external code reference
+[web-classification](flows/standard/web-classification/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a flow demonstrating multi-class classification with LLM. Given an url, it will classify the url into one web category with just a few shots, simple summarization and classification prompts.
+
+
+[Evaluation](flows/evaluation/) flows
+
+path|status|description
+-|-|-
+[basic-eval](flows/standard/basic-eval/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a basic evaluation flow.
+[classification-accuracy-eval](flows/standard/classification-accuracy-eval/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a flow illustrating how to evaluate the performance of a classification system.
+
+[Chat](flows/chat/) flows
+path|status|description
+-|-|-
+[basic-chat](flows/standard/basic-chat/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a basic chat flow.
+[chat-with-wikipedia](flows/standard/chat-with-wikipedia/flow.dag.yaml)|[![batch-score-rest](https://github.com/Azure/azureml-examples/workflows/cli-scripts-batch-score-rest/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/cli-scripts-batch-score-rest.yml)| a flow demonstrating Q&A with GPT3.5 using information from Wikipedia to make the answer more grounded. 
+
+**Connections** ([connections](connections))
+path|status|description
+-|-|-
+[connection.ipynb](connections/connection.ipynb)|[![connections_connection](https://github.com/microsoft/promptflow/actions/workflows/connections_connection.yml/badge.svg)](https://github.com/microsoft/promptflow/actions/workflows/connections_connection.yml)| connections sdk experience
+
+## Reference
+
+* [Promptflow public documentation](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/overview-what-is-prompt-flow?view=azureml-api-2)
+* [Promptflow internal documentation](https://promptflow.azurewebsites.net/)