Extend process_notebooks for testing (#1789)

* Extend process_notebooks for testing

* add command

* spelling and lint

* update docs

* Update contributing.md

* add shebang

* Update contributing.md

* lint

.github/workflows/deploy-website.yml

@@ -52,7 +52,7 @@ jobs:
           quarto render .
       - name: Process notebooks
         run: |
-          python process_notebooks.py
+          python process_notebooks.py render
       - name: Test Build
         run: |
           if [ -e yarn.lock ]; then
@@ -98,7 +98,7 @@ jobs:
           quarto render .
       - name: Process notebooks
         run: |
-          python process_notebooks.py
+          python process_notebooks.py render
       - name: Build website
         run: |
           if [ -e yarn.lock ]; then

notebook/agentchat_RetrieveChat.ipynb

@@ -3036,7 +3036,8 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.10.12"
-  }
+},
+"test_skip": "Requires interactive usage"
  },
  "nbformat": 4,
  "nbformat_minor": 4

notebook/contributing.md

@@ -74,3 +74,37 @@ Learn more about configuring LLMs for agents [here](/docs/llm_configuration).
 :::
 ````
 ``````
+
+## Testing
+
+Notebooks can be tested by running:
+
+```sh
+python website/process_notebooks.py test
+```
+
+This will automatically scan for all notebooks in the notebook/ and website/ dirs.
+
+To test a specific notebook pass its path:
+
+```sh
+python website/process_notebooks.py test notebook/agentchat_logging.ipynb
+```
+
+Options:
+- `--timeout` - timeout for a single notebook
+- `--exit-on-first-fail` - stop executing further notebooks after the first one fails
+
+### Skip tests
+
+If a notebook needs to be skipped then add to the notebook metadata:
+```json
+{
+    "...": "...",
+    "metadata": {
+        "test_skip": "REASON"
+    }
+}
+```
+
+Note: Notebook metadata can be edited by opening the notebook in a text editor (Or "Open With..." -> "Text Editor" in VSCode)

website/README.md

@@ -33,8 +33,7 @@ Navigate to the `website` folder and run:
 
 ```console
 pydoc-markdown
-quarto render ./docs
-python ./process_notebooks.py
+python ./process_notebooks.py render
 yarn start
 ```
 

website/build_website.sh

@@ -28,11 +28,8 @@ fi
 # Generate documentation using pydoc-markdown
 pydoc-markdown
 
-# Render the website using Quarto
-quarto render ./docs
-
 # Process notebooks using a Python script
-python ./process_notebooks.py
+python ./process_notebooks.py render
 
 # Start the website using yarn
 yarn start

website/docs/Contribute.md

@@ -175,6 +175,8 @@ Tests for the `autogen.agentchat.contrib` module may be skipped automatically if
 required dependencies are not installed. Please consult the documentation for
 each contrib module to see what dependencies are required.
 
+See [here](https://github.com/microsoft/autogen/blob/main/notebook/contributing.md#testing) for how to run notebook tests.
+
 #### Skip flags for tests
 
 - `--skip-openai` for skipping tests that require access to OpenAI services.
@@ -216,11 +218,11 @@ Then:
 
 ```console
 npm install --global yarn  # skip if you use the dev container we provided
-pip install pydoc-markdown  # skip if you use the dev container we provided
+pip install pydoc-markdown pyyaml termcolor # skip if you use the dev container we provided
 cd website
 yarn install --frozen-lockfile --ignore-engines
 pydoc-markdown
-quarto render ./docs
+python process_notebooks.py render
 yarn start
 ```
 
@@ -245,7 +247,7 @@ Once at the CLI in Docker run the following commands:
 cd website
 yarn install --frozen-lockfile --ignore-engines
 pydoc-markdown
-quarto render ./docs
+python process_notebooks.py render
 yarn start --host 0.0.0.0 --port 3000
 ```