Merge pull request #26 from agriyakhetarpal/update/sympy-workflow

Ensure that the SymPy version always remains up to date

.github/workflows/deploy.yml

@@ -6,7 +6,13 @@ on:
       - main
   pull_request:
     branches:
-      - '*'
+      - "*"
+  # To ensure we always fetch the latest SymPy wheel
+  # and build the site daily
+  schedule:
+    - cron: "0 3 * * *"
+  # Allow manual trigger to rebuild if something failed
+  workflow_dispatch:
 
 jobs:
   build:
@@ -17,19 +23,32 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v5
         with:
-          python-version: '3.11'
-      - name: Install the dependencies
+          python-version: "3.11"
+
+      - name: Install JupyterLite dependencies
         run: |
           python -m pip install -r requirements.txt
-      - name: Build the JupyterLite site
+
+      - name: Download latest stable SymPy wheel
         run: |
-          jupyter lite build
+          mkdir -p custom_wheels
+          python -m pip download sympy \
+          --only-binary=:all: \
+          --no-deps \
+          --dest custom_wheels
+
+      - name: Reduce the size of the wheel
+        run: pipx run unvendor_tests_from_wheel.py custom_wheels/
+
+      - name: Build the JupyterLite site
+        run: jupyter lite build
+
       - name: Add custom index.html (live.sympy.org landing page) and static/ files
-        run: |
-          ./generateindex.py
+        run: pipx run generate_index.py
+
       - name: Add CNAME file containing live.sympy.org
-        run: |
-          echo "live.sympy.org" > ./_output/CNAME
+        run: echo "live.sympy.org" > ./_output/CNAME
+
       - name: Upload (dist)
         uses: actions/upload-artifact@v4
         with:
@@ -49,6 +68,7 @@ jobs:
           name: jupyterlite-demo-dist-${{ github.run_number }}
           path: ./dist
           merge-multiple: true
+
       - name: Deploy
         uses: JamesIves/[email protected]
         with:

.gitignore

@@ -114,3 +114,6 @@ dmypy.json
 # jupyterlite
 *.doit.db
 _output
+
+# Downloaded wheels for SymPy
+*.whl

README.md

@@ -27,27 +27,23 @@ and provide only minimal customization: a custom landing page that loads the REP
 
 The operation of JupyterLite is configured in several places:
 - `requirements.txt`: specifies the version `jupyterlab`, `jupyterlite-core` and `jupyterlite-pyodide-kernel`.
-- `jupyter_lite_config.json`: general configs for the Jupyter lite build system
+- `jupyter_lite_config.json`: general configs for the JupyterLite build system and custom wheels to make available to the Pyodide kernel for installation.
 - `repl/jupyter-lite.json`: configs specific to the REPL app. See [example here](https://github.com/ivanistheone/live/blob/357e60a228b43ac28ef835953d00f4495a429d78/repl/jupyter-lite.json).
 - `overrides.json`: customizations of the JupyterLite UI
 
 In addition to the above, the SymPy Live shell depends on the following customizations:
 - Custom landing page [index.html](https://github.com/sympy/live/blob/main/templates/index.html) that embeds the JupyterLite REPL.
-- Automatic import `from sympy import *` and init commands in [index.html template](https://github.com/sympy/live/blob/main/templates/index.html#L6-L11).
+- Automatic install (`%pip install sympy`), import (`from sympy import *`) and init commands in [index.html template](https://github.com/sympy/live/blob/main/templates/index.html#L6-L11).
 - JavaScript code to decode evaluate query string, see [here](https://github.com/sympy/live/blob/main/templates/index.html#L49-L56).
 
 > [!NOTE]
 > The versions of `jupyterlab` and `jupyterlite-core` determine which version of JupyterLite we are running.
 > Use the [upstream requirements file](https://github.com/jupyterlite/demo/blob/main/requirements.txt) as reference.
 
 > [!NOTE]
-> The version of SymPy running on [live.sympy.org/](https://live.sympy.org/) depends on the package version included in Pyodide.
-> See [here](https://pyodide.org/en/stable/usage/packages-in-pyodide.html) for the current package versions.
-> Updating the SymPy version requires first updating the [SymPy package in Pyodide](https://github.com/pyodide/pyodide/tree/main/packages/sympy).
-> See this [PR](https://github.com/pyodide/pyodide/pull/5098) for example. 
-> Once the new version of `jupyterlite-pyodide-kernel` is released (which can take months),
-> we can update the version in `requirements.txt` and deploy the updated SymPy Live Shell following the instructions below.
-
+> Pyodide might not have the latest version of SymPy available in PyPI. The [version of SymPy available in Pyodide](https://github.com/pyodide/pyodide/tree/main/packages/sympy) is updated only when a new version of Pyodide is released. This can take a considerable amount of time, which means that it can get outdated.
+> Therefore, we ensure that the version of SymPy used in the SymPy Live Shell is up-to-date by running the `pip download sympy --no-deps` command before deployment, which downloads the latest version of SymPy from PyPI and stores it in `custom_wheels/`. This directory is indexed by JupyterLite via the `jupyter_lite_config.json` file to make the wheel available to the Pyodide kernel for the REPL app for installation.
+> It is still recommended to keep updating the SymPy version in Pyodide to the latest version to ensure that the SymPy Live Shell is always up-to-date, so that we don't have to run the `pip download` command before every deployment.
 
 
 ### Deploying in production
@@ -57,12 +53,15 @@ Deployment to production is automated by the Github actions workflow [deploy.yml
 which performs the following steps:
 
  1. Installs Python and dependencies listed in `requirements.txt`
- 2. Runs the command `jupyter lite build` to build the JupyterLab static site, placing the results in `_output/`.
- 3. Runs `./generateindex.py` to overwrite the index file `_output/index.html` with the custom SymPy Live Shell landing page.
- 4. Adds the file `_output/CNAME` containing `live.sympy.org`.
- 5. Deploys the contents of `_output` to GitHub pages (`gh-pages` branch hosted at https://live.sympy.org).
+ 2. Runs the `pip download sympy --no-deps --dest custom_wheels` command and stores the wheel in `custom_wheels/`. This directory is indexed by JupyterLite via the [`jupyter_lite_config.json`](jupyter_lite_config.json) file to make the wheel available to the Pyodide kernel
+    for the REPL app.
+ 3. Runs `unvendor_tests_from_wheel.py custom_wheels` to remove the tests from the downloaded wheel, reducing its size.
+ 4. Runs the command `jupyter lite build` to build the JupyterLab static site, placing the results in `_output/`.
+ 5. Runs `generate_index.py` to overwrite the index file `_output/index.html` with the custom SymPy Live Shell landing page.
+ 6. Adds the file `_output/CNAME` containing `live.sympy.org`.
+ 7. Deploys the contents of `_output` to GitHub pages (`gh-pages` branch hosted at https://live.sympy.org).
 
-The last step (deploy to `gh-pages`) runs only on push to the `main` branch.
+The last step (deploy to `gh-pages`) runs only on pushes to the `main` branch.
 
 
 ### Running locally
@@ -72,12 +71,21 @@ but you'll run local web server to view the static files in `_output`,
 and modify one line in `templates/index.html` to load the iframe from localhost (127.0.0.1):
 
  1. Install Python and dependencies listed in `requirements.txt`
- 2. Runs the command `jupyter lite build` to build the JupyterLab static site, placing the results in `_output/`.
- 3. Edit [`templates/index.html`](https://github.com/sympy/live/blob/main/templates/index.html#L3)
+ 2. Download latest stable SymPy wheel:
+
+   ```bash
+    mkdir -p custom_wheels
+    python -m pip download sympy --only-binary=:all: --no-deps --dest custom_wheels
+   ```
+
+   This wheel will be indexed by JupyterLite via the `jupyter_lite_config.json` file and made available to the Pyodide kernel for the REPL app for installation.
+ 3. Optionally, run `unvendor_tests_from_wheel.py custom_wheels` with a PEP 723 compatible script runner such as `pipx`, `uv`, `hatch`, etc. to remove the tests from the downloaded wheel, reducing its size.
+ 3. Run the command `jupyter lite build` to build the JupyterLab static site, placing the results in `_output/`.
+ 4. Edit [`templates/index.html`](https://github.com/sympy/live/blob/main/templates/index.html#L3)
     to change the value of `host` from `https://www.sympy.org/live` to `http://127.0.0.1:8000`.
- 4. Run `./generateindex.py` to overwrite the index file `_output/index.html` with the custom SymPy Live Shell landing page.
- 5. Run a local web server in the directory `_output/`. For example, you can run `cd _output/` followed by `python3 -m http.server 8000`.
- 6. Open `http://127.0.0.1:8000/index.html` in your browser.
+ 5. Run `generate_index.py` with a PEP 723 compatible script runner such as `pipx`, `uv`, `hatch`, etc. to overwrite the index file `_output/index.html` with the custom SymPy Live Shell landing page.
+ 6. Run a local web server in the directory `_output/`. For example, you can run `cd _output/` followed by `python3 -m http.server 8000`.
+ 7. Open `http://127.0.0.1:8000/index.html` in your browser.
 
 
 

generateindex.py → generate_index.py

@@ -1,34 +1,50 @@
 #!/usr/bin/env python3
 
+# /// script
+# requires-python = ">=3.9"
+# dependencies = [
+#   "jinja2",
+#   "babel",
+# ]
+# ///
+
 import os
 import babel
 import babel.dates
 import hashlib
 import datetime
 import shutil
-
-
+import re
+from pathlib import Path
 
 # Setup jinja2
 ################################################################################
 from jinja2 import Environment, FileSystemLoader
 
+
 def md5_filter(value):
     return hashlib.md5(value).hexdigest()
 
+
 def format_datetime(year, month, day):
-    return babel.dates.format_date(datetime.date(year, month, day),
-                                   locale=env.globals["locale"])
+    return babel.dates.format_date(
+        datetime.date(year, month, day), locale=env.globals["locale"]
+    )
 
-env = Environment(loader=FileSystemLoader('templates'),
-                  extensions=['jinja2.ext.i18n'])
 
-env.filters["md5"] = md5_filter
-env.globals["datetime"] = format_datetime
-env.install_null_translations() # Currently only template is in English; no i18n
+def get_sympy_version():
+    """Get SymPy version from wheel file downloaded in the custom_wheels/ directory."""
+    wheel = next(Path("custom_wheels").glob("sympy-*-py3-none-any.whl"))
+    version = re.match(r"sympy-(.+)-py3-none-any\.whl", wheel.name).group(1)
+    return version
 
 
+env = Environment(loader=FileSystemLoader("templates"), extensions=["jinja2.ext.i18n"])
 
+env.filters["md5"] = md5_filter
+env.globals["datetime"] = format_datetime
+env.globals["sympy_version"] = get_sympy_version()
+env.install_null_translations()  # Currently only template is in English; no i18n
 
 # Generate index.html add required static/ files
 ################################################################################
@@ -45,7 +61,7 @@ def format_datetime(year, month, day):
     s = t.render()
     # os.makedirs(language, exist_ok=True)
     destpath = os.path.join(destdir, template)
-    shutil.copyfile(destpath, destpath+".bak")  # backup jupyterlite index.html
+    shutil.copyfile(destpath, destpath + ".bak")  # backup jupyterlite index.html
     with open(destpath, "wb") as f:
         f.write(s.encode("utf-8"))
         f.write(b"\n")
@@ -56,4 +72,3 @@ def format_datetime(year, month, day):
 if os.path.exists(staticdestdir):
     shutil.rmtree(staticdestdir)
 shutil.copytree(staticsrcdir, staticdestdir)
-

jupyter_lite_config.json

@@ -3,5 +3,8 @@
     "apps": ["repl"],
     "no_unused_shared_packages": true,
     "no_sourcemaps": true
+  },
+  "PipliteAddon": {
+    "piplite_urls": ["custom_wheels/"]
   }
 }

requirements.txt

@@ -1,12 +1,8 @@
 # Core modules and Pyodide kernel (mandatory)
-# Pyodide kernel version 0.5.1 comes with Pyodide 0.27.1
+# Pyodide kernel version 0.5.2 comes with Pyodide 0.27.1
 # which in-turn comes with SymPy 1.13.3:
 # https://jupyterlite-pyodide-kernel.readthedocs.io/en/stable/#compatibility
 # https://pyodide.org/en/stable/usage/packages-in-pyodide.html
 # We constrain/pin them to avoid breaking changes on rebuilds.
 jupyterlite-core>=0.5.0,<0.6.0
-jupyterlite-pyodide-kernel==0.5.1
-
-# Specific to generating SymPy Live index page
-jinja2==3.1.5
-babel==2.16.0
+jupyterlite-pyodide-kernel==0.5.2

templates/index.html

@@ -3,12 +3,16 @@
 {% set host = "https://www.sympy.org/live" %}
 {% set path = "/repl/" %}
 {% set query = "?toolbar=1&kernel=python" %}
-{% set code1 = "from sympy import *" %}
-{% set code2 = "init_printing()" %}
-{% set code3 = "x, y, z, t = symbols('x y z t')" %}
-{% set code4 = "k, m, n = symbols('k m n', integer=True)" %}
-{% set code5 = "f, g, h = symbols('f g h', cls=Function)" %}
-{% set initcode = "\n".join([code1, code2, code3, code4, code5]) %}
+{% set preinitcode = "\n".join([
+    "%pip install sympy==" ~ sympy_version
+]) %}
+{% set initcode = "\n".join([
+    "from sympy import *",
+    "init_printing()",
+    "x, y, z, t = symbols('x y z t')",
+    "k, m, n = symbols('k m n', integer=True)",
+    "f, g, h = symbols('f g h', cls=Function)"
+]) %}
 
 
 {% block title %}SymPy Live Shell{% endblock %}
@@ -46,7 +50,7 @@
   $(document).ready(function() {
     // default iframe src with SymPy initcode
     // see https://jupyterlite.readthedocs.io/en/stable/quickstart/embed-repl.html#configuration for URL customisation
-    var iframe_src = "{{ host }}{{ path }}{{ query|safe }}&code={{ initcode|urlencode }}";
+    var iframe_src = "{{ host }}{{ path }}{{ query|safe }}&code={{ preinitcode|urlencode }}&code={{ initcode|urlencode }}";
     // check if we have additional statements to evaluate in the query string
     var statements = getURLParameter('evaluate');
     if (statements) {